Blog O’ Matty

Adjusting how often the Linux kernel checks for MCEs

I wrote about Linux mcelog utility a few weeks back, and described how it can be used to monitor the /dev/mcelog device for machine check exception (MCEs). By default, the Linux kernel will check for MCEs every five minutes. The polling interval is defined in the sysfs check_interval entry, which you can view with cat:

$ cat /sys/devices/system/machinecheck/machinecheck0/check_interval
12c

$ python
>>> print “%d” % 0×12c
300

To configure the host to use a shorter check interval, you can echo the desired value to the sysfs entry for processor 0:

$ echo 60 > /sys/devices/system/machinecheck/machinecheck0/check_interval

$ cat /sys/devices/system/machinecheck/machinecheck0/check_interval
3c

$ cat /sys/devices/system/machinecheck/machinecheck1/check_interval
3c

If you want to get additional information on check_interval, check out the machinecheck text file in the kernel documentation directory. If you are curious how the code actually detects a MCE, you can look through the source code in <KERNEL_SOURCE_ROOT>/arch/x86/kernel/cpu/mcheck.

Understanding the Linux /boot directory

When I first began using Linux quite some time ago, I remember thinking to myself WTF is all this stuff in /boot. There were files related to grub, a file called vmlinuz, and several ASCII text files with cool sounding names. After reading through the Linux kernel HOWTO, the /boot directory layout all came together, and understanding the purpose of each file has helped me better understand how things work, and allowed me to solve numerous issues in a more expedient manner. Given a typical CentOS or Fedora host, you will probably see something similar to the following in /boot:

$ cd /boot

$ tree

.
|-- System.map-2.6.29.5-191.fc11.x86_64
|-- System.map-2.6.30
|-- config-2.6.29.5-191.fc11.x86_64
|-- config-2.6.30
|-- efi
|   `-- EFI
|       `-- redhat
|           `-- grub.efi
|-- grub
|   |-- device.map
|   |-- e2fs_stage1_5
|   |-- fat_stage1_5
|   |-- ffs_stage1_5
|   |-- grub.conf
|   |-- iso9660_stage1_5
|   |-- jfs_stage1_5
|   |-- menu.lst -> ./grub.conf
|   |-- minix_stage1_5
|   |-- reiserfs_stage1_5
|   |-- splash.xpm.gz
|   |-- stage1
|   |-- stage2
|   |-- ufs2_stage1_5
|   |-- vstafs_stage1_5
|   `-- xfs_stage1_5
|-- initrd-2.6.29.5-191.fc11.x86_64.img
|-- initrd-2.6.30.img
|-- vmlinuz-2.6.29.5-191.fc11.x86_64
`-- vmlinuz-2.6.30

For each kernel release, you will typically see a vmlinuz, System.map, initrd and config file. The vmlinuz file contain the actual Linux kernel, which is loaded and executed by grub. The System.map file contains a list of kernel symbols and the addresses these symbols are located at. The initrd file is the initial ramdisk used to preload modules, and contains the drivers and supporting infrastructure (keyboard mappings, etc.) needed to manage your keyboard, serial devices and block storage early on in the boot process. The config file contains a list of kernel configuration options, which is useful for understanding which features were compiled into the kernel, and which features were built as modules. I am going to type up a separate post with my notes on grub, especially those related to solving boot related issues.

Listing file system lock files on Linux hosts

I mentioned in a previous post that I was using the Linux flock utility to ensure that only one copy of yum would run at any given point in time (well, theoretically someone could call yum from outside of the script, but there are only so many use cases you can protect against). The lock files that are created by flock reside on a file system, and can be viewed with the lslk utility:

$ lslk

SRC        PID DEV     INUM SZ TY M ST WH END LEN NAME
(unknown) 1536 8,1   927544     w 0  0  0   0   0 / (rootfs)
atd       1785 8,1   927573  5  w 0  0  0   0   0 /var/run/atd.pid
(unknown) 2034 8,1 14963655     w 0  0  0   0   0 / (rootfs)

If the file name doesn’t appear in the lslk output, you can use the find utilities “-inum” option (find a file by its inode number) to locate the file using the inode number listed in the 4th column:

$ find . -inum 14963655
./lock

If the process name doesn’t show up, you can use the lsof utility along with the name of the lock to see which process has the lock file open:

$ lsof | awk ‘$9 ~ /lock/ { print }’

test      2033      root  200w      REG                8,1        0   14963655 /tmp/lock
sleep     2035      root  200w      REG                8,1        0   14963655 /tmp/lock

I have been using lslk off and on for years, and it’s a SUPER useful tool for debugging issues with file system lock files. Nice!

Testing out live CD distributions with KVM

I read over the latest KVM putback log last night, and saw that KVM now supports booting from ISO images that are accessible via HTTP (it uses libcurl under the covers). This is pretty fricking cool, and allow you to boot in recovery mode without requiring local media or PXE, and provides a super easy way to play around with live CD distributions. To use this feature, you will first need to install KVM build 87 from source. Once that is complete, you can add the URL to your qemu-kvm command line:

$ qemu-kvm -cdrom http://disarm.prefetch.net/isos/fedoralivecd.dvd …

You can also fire up virsh and “edit” the URL into the XML data. KVM is pretty rad, and it’s amazing how many cool features the KVM developers keep kicking out!

Accessing the Python help facility from the Python shell

Python has a ton of useful modules, and the built-in help facility is extremely useful for gaining quick access to a description of methods in a given module. Once a module is imported with import:

>>> import pexpect

You can run dir(MODULE_NAME) to view the list of methods in the module:

>>> dir(pexpect)
['EOF', 'ExceptionPexpect', 'TIMEOUT', '__all__', '__builtins__', '__doc__', '__file__', '__loader__', '__name__', '__revision__', '__version__', 'errno', 'fcntl', 'os', 'pty', 're', 'resource', 'run', 'searcher_re', 'searcher_string', 'select', 'signal', 'spawn', 'split_command_line', 'string', 'struct', 'sys', 'termios', 'time', 'traceback', 'tty', 'types', 'which']

To get help on a specific method, you can pass the module and method name to the built-in help function:

>>> help(pexpect.spawn)
Help on class spawn in module pexpect:

class spawn(__builtin__.object)
 |  This is the main class interface for Pexpect. Use this class to start
 |  and control child applications.
 |
 |  Methods defined here:

   .....

This is pretty sweet, and makes coding in Python super easy.

How does crossbow deal with MAC address conflicts?

When I gave my presentation on Solaris network virtualization a few months back, one of the folks in the audience asked me how Crossbow deals with duplicate MAC detection. I didn’t have a solid answer for the gentlemen that asked, but thanks to Nicolas Droux from Solaris kernel engineering, now I do. Here is what Nicolas had to say about this topic on the network-discuss mailing list:

“When a VNIC is created we have checks in place to ensure that the address doesn’t conflict with another MAC address defined on top of the same underlying NIC. When the MAC address is generated randomly, and the generated MAC address conflicts with another VNIC, we currently fail the whole operation. We should try another MAC address in that case, transparently to the user, I filed CR 6853771 to track this.

To reduce the risk of MAC address conflicts with physical NICs on other hosts on the network, we use by default a OUI with local bit set for random MAC addresses, and we let the administrator use a different OUI or prefix if desired. We currently don’t have a mechanism in place do perform automatic MAC address duplication detection between multiple hosts.”

I was under the impression that Crossbow used the ARP cache and DAD code to verify that a MAC address wasn’t in use on the network, but that doesn’t appear to be the case. Given this new information, I will need to modify my tools to assign a static MAC that is based off of the address assigned to th virtual NIC. Thanks Nicolas for the awesome reply to the list!