Understanding memory usage on Linux hosts just got easier

While looking at the features that were added to the Linxu 2.6.20.X kernels, I came across this gem on the kernelnewbies website:

“Measuring how much memory processes are using is more difficult than it looks, specially when processes are sharing the memory used. Features like /proc/$PID/smaps (added in 2.6.14) help, but it has not been enough. 2.6.25 adds new statistics to make this task easier. A new /proc/$PID/pagemaps file is added for each process. In this file the kernel exports (in binary format) the physical page localization for each page used by the process. Comparing this file with the files of other processes allows to know what pages they are sharing. Another file, /proc/kpagemaps, exposes another kind of statistics about the pages of the system. The author of the patch, Matt Mackall, proposes two new statistic metrics: “proportional set size” (PSS) – divide each shared page by the number of processes sharing it; and “unique set size” (USS) (counting of pages not shared). The first statistic, PSS, has also been added to each file in /proc/$PID/smaps. In this HG repository you can find some sample command line and graphic tools that exploits all those statistics.”

This is awesome, and I thinking having USS and PSS statistics will greatly help admins understand how memory is being used on their systems. If you want to read more about this, check out the following LWN article.

x86 / linux boot process

There is quite a bit of documentation around the internet on the linux boot process, but Gustavo Duarte I think did an excellent job describing this in a clear and concise way.  He also has several links to the Linux  kernel source code and describes what is occurring step-by-step through the bootstrap phase all the way to the execution of /sbin/init.

His first entry lays the foundation of the basis of the x86 Intel chipset, memory map, and logical motherboard layout.   This provides a basic understanding about the traditional hardware motherboard implementations.

Next, he describes BIOS initialization, and loading of the MBR.  This briefly touches on the boot loader which starts the Linux bootstrap phase.

Finally, the kernel boot process is detailed with links to C and Assembly source code, with a brief narrative of exactly what is happening.

This was an awesome description of the early-on start up and initialization phases of hardware and bootstrapping of the O/S.  Gustavo provides a great description of real-mode and protected-mode CPU states.

Thanks Gustavo!

Remotely mounting directories through SSH

I manage a fair number of Linux hosts, and have recently been looking for ways to securely mount remote directories on my servers for administrative purposes. NFS and Samba don’t have a terribly good security track record, so I don’t like to use either of these solutions unless truly warranted. Rsync over SSH is pretty sweet, but it’s not quite as transparent as I would like it to be. Since all of my hosts support SSH, I started to wonder if someone had developed a solution to transparently move files between two systems using SSH. After a bit of digging, I came across the super cool sshfs fuse module, which does just that!

Sshfs allows you to “mount” a remote directory over the SSH protocol, and it provides transparent access to files stored on a remote server. To use this nifty module with Fedora, you first need to install the fuse-sshfs package:

$ yum install fuse-sshfs

Once the fuse kernel modules and userland utilities are installed, the sshfs utility can be used to mount a remote directory on a local mount point. In the following example, the sshfs utility is used to mount the directory $HOME/backup on the server giddieup on the local directory /home/matty/backup:

$ sshfs -C -o reconnect,idmap=user giddieup:backup/ /home/matty/backup

Once the sshfs command completes, you can add and remove files to the locally mounted directory (/home/matty/backup in this case), and these changes will be automatically propogated to the remote server. The first time I ran sshfs I received the error “Operation not permitted.” After digging into this further, I noticed that the fusermount and sshfs utilities were not setuid root out of the box. To address this problem, I changed the group ownership of both utilities to fuse, put myself in the fuse group, added the setuid bit and changed the mode of both executables to 0750. The opensolaris community is currently porting FUSE to Solaris, and I am looking forward to eventually being able to use SSHFS on my Solaris hosts!

Linux udev humor

The udev device management framework is one of the new features that was added to the Linux 2.6 kernel, and allows the /dev namespace to be populated based on hotplug events sent from the kernel to the userspace udevd daemon. While reading through the udev FAQ, I found a good explanation of udev:

Q: How is udev related to devfs?
A: udev works entirely in userspace, using hotplug events the kernel sends
   whenever a device is added or removed from the kernel. Details about
   the devices are exported by the kernel to the sysfs filesystem at /sys
   All device naming policy permission control and event handling is done in
   userspace. devfs is operated from within the kernel.

As well as some comedic writing:

Q: But udev will not automatically load a driver if a /dev node is opened
   when it is not present like devfs will do.
A: Right, but Linux is supposed to load a module when a device is discovered
   not to load a module when it's accessed.

Q: Oh come on, pretty please.  It can't be that hard to do.
A: Such a functionality isn't needed on a properly configured system. All
   devices present on the system should generate hotplug events, loading
   the appropriate driver, and udev will notice and create the
   appropriate device node.  If you don't want to keep all drivers for your
   hardware in memory, then use something else to manage your modules
   (scripts, modules.conf, etc.)  This is not a task for udev.

Q: But I love that feature of devfs, please?
A: The devfs approach caused a lot of spurious modprobe attempts as
   programs probed to see if devices were present or not.  Every probe
   attempt created a process to run modprobe, almost all of which were

This made me laugh silly, and I wish more FAQ maintainers used this style of writing. Whoever wrote this, I commend you!