Is ZFS ready for primetime?

Over the course of the past four months, I have encountered a couple of ZFS bugs that led to corrupt pools. One of the bugs hit this weekend, and resulted in us having to recover the pool from a snapshot on our storage array (this experience has made me truly appreciate the 3par’s snapshot capability). These bugs have led me to question whether or not ZFS is ready for prime time, and if companies should be deploying it in production. If you have bumped into any ZFS bugs that led to data corruption, please leave me a comment.

Getting patches to install on Solaris 10 systems

There have been a number of threads in zones-discuss relating to Solaris 10 patching. Most of the feedback came from folks who ran into issues related to delayed activation patching, or patching zones that run on ZFS file system. Running zones on ZFS file systems is not currently supported by Sun, but it appears installing the latest version of 119254 or 119255 will help address a number of issues related to running zones on ZFS file systems. I am posting this here for future reference, and in an effort to help others who may be stumbling into issues applying patches to their Solaris 10 hosts.

Monitoring system utilization on Linux hosts

I am always on the look out for tools to analyze system performance. One nifty tool I recently came across is atop, which is an advanced system performance monitor for Linux. When atop is run, it displays overall system utilization in the header, and per-process CPU, memory, network or disk utilization information in the body (you need to patch your kernel to get disk and network utilization). Here is a sample atop session that shows just how awesome this utility is:

I really dig the headers, as well as the network and disk utilization menus. Atop rocks!

Viewing SMBIOS data

The SMBIOS specification allows applications and operating systems to retrieve and store hardware configuration settings. Retrieving hardware settings from a shell prompt can be extremely useful, since rebooting to view a given BIOS menu requires the machine to be taken offline. If you happen to be running Solaris 10 or Nevada, you can run the smbios utility to display the contents of the SMBIOS image on a machine:

$ smbios |more

ID    SIZE TYPE
0     60   SMB_TYPE_BIOS (BIOS information)

  Vendor: Sun Microsystems
  Version String: S39_3B17
  Release Date: 07/04/2007
  Address Segment: 0xf000
  ROM Size: 524288 bytes
  Image Size: 65536 bytes
  Characteristics: 0x17f8bde90

< ..... >

This is good stuff.

Simplifying Apache chroot creation with mod_chroot

Building and maintaining Apache chroot environments can be a royal pain. Creating a chroot environment for Apache requires you to first identify all the libraries and applications that are required to run the httpd processes. Once you identify the dependencies, you need to create a chroot environment that contains these files. After you successfully create the chroot environment, you need to update it when security and reliability updates are released. This can be a time consuming process, and even though several tools (e.g., mock, makejail, etc.) exist to ease this process, there is still a fair amount of work that needs to occur to get things running properly.

One way to get around the hassles of creating chroot environments is to use mod_chroot. Mod_chroot will issue the chroot() system call after the runtime linker loads dependent libraries, and Apache processes its configuration file and opens the access and error logs. Delaying the chroot() system call until after Apache is initialized can greatly reduce the amount of work required to configure the chroot environment, since libraries don’t need to be copied* into the jail, and logs and configuration files can live outside of the chroot environment.

Installing and configuring mod_chroot is a snap. To compile and install mod_chroot from source, you can use the apxs utility from the Apache installation you want to run in the chroot environment:

$ tar xfvz mod_chroot-0.5.tar.gz

$ apxs -cai mod_chroot-0.5/src/apache20/mod_chroot.c

This will compile mod_chroot and install it the Apache loadable modules directory. To configure mod_chroot, you will first need to add a “LoadModule” directive to your httpd.conf to load mod_chroot:

LoadModule chroot_module modules/mod_chroot.so

Next you will need to add a “ChrootDir” directive with the directory you want to chroot Apache to:

ChrootDir /var/chroot/apache

The chroot directory should contain the content your web server serves, and any dependencies that can’t be resolved prior to the web server starting. Configuration is extremely simple, though there are a few caveats to watch out for. The web server cannot be gracefully restarted unless the web server configuration file is moved into the chroot, and programs that lazily load shared libraries will fail. Mod_chroot is an incredible module, and can definitely make managing chroot environments a whole lot easier! Nice!

* If a program uses dlopen() to load a library, you will need to copy the library into the chroot environment, or use the Apache “LoadFile” directive to load it at initialization time.