I grew up loving rock and roll, and spent a fair amount of time in college listening to music (especially when I was writing code for class). One band that got a fair amount of play time on my home stero was Candlebox. They had a cool relaxing sound, and could jam with the best of them. So when Ticketmaster informed me that they would be in town playing a show at a local venue, I decided to get tickets. Once the opening acts left the stage, Candlebox played an audio clip, and then blasted into a version of “Arrow.” The band looked like they were having fun on stage, and proceeded to play all of their hits, including “Far Behind,” “You,” “Cover Me,” “Rain,” “Change,” and “Simple lessons.” The show was incredible, and being four rows back gave me a chance to watch the band jam close up. If you get a chance to see them. check ‘em out!
I use the MD (multiple device) logical volume manager to mirror the boot devices on the Linux servers I support. When I first started using MD, the mdadm utility was not available to manage and monitor MD devices. Since disk failures are relatively common in large shops, I used the shell script from my SysAdmin article Monitoring and Managing Linux Software RAID to send E-mail when a device entered the failed state. While reading through the mdadm(8) manual page, I came across the “–monitor” and “–mail” options. These options can be used to monitor the operational state of the MD devices in a server, and generate E-mail notifications if a problem is detected. E-mail notification support can be enabled by running mdadm with the “–monitor” option to monitor devices, the “–daemonise” option to create a daemon process, and the “–mail” option to generate E-mail:
$ /sbin/mdadm --monitor --scan --daemonise --mail=root@localhost
Once mdadm is daemonized, an E-mail similar to the following will be sent each time a failure is detected:
From: mdadm monitoring
To: root@localhost.localdomain
Subject: Fail event on /dev/md1:biscuit
This is an automatically generated mail message from mdadm
running on biscuit
A Fail event had been detected on md device /dev/md1.
Faithfully yours, etc.
I digs me some mdadm!
I recently started supporting several DNS servers running BIND 9. To ensure that these server are up and operational at all times, I wrote a small shell script named dns-check to test the operational state of each server. The script takes a file as an argument, and each line in the file contains the IP address of a DNS server (names will also work), a name to resolve, and the record type that should be requested. If the script is unable to resolve the name for one reason or another (any return code > 0 is a failure), the script will log a message to syslog, and send E-mail to the address listed in the $ADMIN variable, or an address passed to the “-e” option. Here is sample run:
$ cat dns-check-sites
ns1.fooby.net mail.fooby.net A
ns2.fooby.net mail.fooby.net A
$ dns-check -e dns-admin@prefetch.net -f dns-check-sites
The script is nothing special, but might be useful to folks running DNS servers.
I recently had a file system on a Solaris Volume Manager (SVM) metadevice fill up, and I needed to expand it to make room for some additional data. Since the expansion could potentially cause problems, I backed up the file system, and saved a copy of the metastat and df output to my local workstation. Having several backups always gives me a warm fuzzy, since I know I have a way to revert back to the old configuration if something goes awry. Once the configuration was in a safe place and the data backed up, I used the umount command to unmount the /data file system, which lives on metadevice d100:
$ df -h
Filesystem size used avail capacity Mounted on
/dev/dsk/c1t0d0s0 7.9G 2.1G 5.7G 27% /
/devices 0K 0K 0K 0% /devices
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 2.3G 600K 2.3G 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
/usr/lib/libc/libc_hwcap1.so.1
7.9G 2.1G 5.7G 27% /lib/libc.so.1
fd 0K 0K 0K 0% /dev/fd
/dev/dsk/c1t0d0s4 4.0G 154M 3.8G 4% /var
swap 2.3G 32K 2.3G 1% /tmp
swap 2.3G 24K 2.3G 1% /var/run
/dev/dsk/c1t0d0s3 19G 2.8G 17G 15% /opt
/dev/md/dsk/d100 35G 35G 120M 99% /data
$ umount /data
After the file system was unmounted, I had to run the metaclear utility to remove the metadevice from the meta state database:
$ metaclear D100
d100: Concat/Stripe is cleared
Now that the metadevice was removed, I needed to add it back with the desired layout. It is EXTREMELY important to place the device(s) back in the right order, and to ensure that the new layout doesn’t corrupt the data that exists on the device(s) that contain the file system (i.e., don’t create a RAID5 metadevice with the existing devices, since that will wipe your data when the RAID5 metadevice is initialized). In my case, I wanted to concatenate another hardware RAID protected LUN to the meta device d100. This was accomplished by running metainit with the “numstripes” equal to 2 to indicate a 2 stripe concatenation, and “width” equal to 1 to indicate that each stripe should have one member:
$ metainit d100 2 1 c1t1d0s0 1 c1t2d0s0
d100: Concat/Stripe is setup
Once the new metadevice was created, I ran the mount utility to remount the /data file system, and then executed growfs to expand the file system:
$ mount /dev/md/dsk/d100 /data
$ growfs -M /data /dev/md/rdsk/d100
Warning: 2778 sector(s) in last cylinder unallocated
/dev/md/rdsk/d100: 150721830 sectors in 24532 cylinders of 48 tracks, 128 sectors
73594.6MB in 1534 cyl groups (16 c/g, 48.00MB/g, 5824 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
32, 98464, 196896, 295328, 393760, 492192, 590624, 689056, 787488, 885920,
Initializing cylinder groups:
..............................
super-block backups for last 10 cylinder groups at:
149821984, 149920416, 150018848, 150117280, 150215712, 150314144, 150412576,
150511008, 150609440, 150707872
After the growfs operation completed, I had some breathing room on the /data file system:
$ df -h
Filesystem size used avail capacity Mounted on
/dev/dsk/c1t0d0s0 7.9G 2.1G 5.7G 27% /
/devices 0K 0K 0K 0% /devices
ctfs 0K 0K 0K 0% /system/contract
proc 0K 0K 0K 0% /proc
mnttab 0K 0K 0K 0% /etc/mnttab
swap 2.3G 600K 2.3G 1% /etc/svc/volatile
objfs 0K 0K 0K 0% /system/object
/usr/lib/libc/libc_hwcap1.so.1
7.9G 2.1G 5.7G 27% /lib/libc.so.1
fd 0K 0K 0K 0% /dev/fd
/dev/dsk/c1t0d0s4 4.0G 154M 3.8G 4% /var
swap 2.3G 32K 2.3G 1% /tmp
swap 2.3G 24K 2.3G 1% /var/run
/dev/dsk/c1t0d0s3 19G 2.8G 17G 15% /opt
/dev/md/dsk/d100 71G 36G 35G 49% /data
The fact that you have to unmount the file system to grow a metadevice is somewhat frustrating, since every other LVM package I have used allows volumes and file system to be expanded on the fly (it’s a good thing ZFS is shipping with Solaris). As with all data migrations, you should test storage expansion operations prior to performing them on production systems.
I have been using the Ultime Boot Disk for the past few months to test x86 and X64 hardware. The disk contains numerous awesome utilities that can be used to test memory, disks and CPUs. The following packages come on the CD, and are four of my personal favorites:
Memtest86+ to test memory
Darik’s Boot and Nuke to securely erase data from a disk drive
CPU burn to test CPUs
PCI sniffer to identify the type of card in a system