Using ZFS clones to simplify upgrades

One cool ZFS feature that was introduced in Solaris 10 11/06 is the ability to clone a file system. This is a truly powerful tool, since it allows admins to create copies of file systems prior to performing upgrades, and a way to roll a file system back to a previous state (this provides a simple lightweight method to back out changes if upgrades go awry). To illustrated just how useful this is, let’s create a clone of the file system oradata. To clone oradata, we first need to take a sapshot of the oradata file system:

$ zfs snapshot striped/oradata@ordata_snap

Once a snapshot is created, the zfs utilities’ “clone” option can be used to create a clone of the file system from the snaphost we just created:

$ zfs clone striped/oradata@ordata_snap striped/oradata_clone

After the clone is created, it can be “promoted” to a file system with the zfs utilities’ “promote” option:

$ zfs promote striped/oradata_clone

If everything works correctly (which it should), the clone and the file system the clone was created from will appear in the df output:

$ df -h

Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c0d0s0         17G   2.9G    14G    18%    /
/devices                 0K     0K     0K     0%    /devices
/dev                     0K     0K     0K     0%    /dev
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                  1016M   768K  1015M     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
/usr/lib/libc/libc_hwcap1.so.1
                        17G   2.9G    14G    18%    /lib/libc.so.1
fd                       0K     0K     0K     0%    /dev/fd
swap                  1015M     0K  1015M     0%    /tmp
swap                  1015M    24K  1015M     1%    /var/run
striped                 52G    27K    50G     1%    /striped
striped/oradata         52G   300M    50G     1%    /striped/oradata
striped/oradata_clone
                        52G   300M    50G     1%    /striped/oradata_clone

Now let’s pretend that a DBA accidentally deletes the Oracle datafiles in /striped/oradata:

$ rm -f /striped/oradata/*

Oops! Since oradata_clone is a clone of the oradata file system, we can restore oradata by first deleting the original file system (you could also rename it if you wanted to keep it around):

$ zfs destroy -f striped/oradata

And then renaming the clone to take it’s place:

$ zfs rename striped/oradata_clone striped/oradata

Once the rename occurs, everything is as it once was:

$ df -h

Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c0d0s0         17G   2.9G    14G    18%    /
/devices                 0K     0K     0K     0%    /devices
/dev                     0K     0K     0K     0%    /dev
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                  1015M   768K  1014M     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
/usr/lib/libc/libc_hwcap1.so.1
                        17G   2.9G    14G    18%    /lib/libc.so.1
fd                       0K     0K     0K     0%    /dev/fd
swap                  1014M     0K  1014M     0%    /tmp
swap                  1014M    24K  1014M     1%    /var/run
striped                 52G    26K    50G     1%    /striped
striped/oradata         52G   300M    50G     1%    /striped/oradata

$ ls -la /striped/oradata

total 614555
drwxr-xr-x   2 root     sys            5 Dec 24 11:49 .
drwxr-xr-x   3 root     sys            3 Dec 24 11:59 ..
-rw-r--r--   1 root     root     104857600 Dec 24 11:47 iklts01.dbf
-rw-r--r--   1 root     root     104857600 Dec 24 11:48 iklts02.dbf
-rw-r--r--   1 root     root     104857600 Dec 24 11:49 iklts03.dbf

This has so many uses, and once ZFS boot is incorporated, using clones to protect yourself from rogue patches will be one of the coolest things ever! Shibby!

Disabling old wordpress comments

As with most individuals that manage a blog, I get inundated with comment spam. Based on some research I did, it looks like almost all of the blog spam occurs for older posts, especially those over 30 days old. Based on this information, I decided to disable comments for all old posts older than 30 days. It turns out that you can’t do this directly from wordpress without a plugin, so I decided to adjust the “comment_status” column in the wp_posts table directly. To remove comment spam for all posts older than 30 days, I first used the GNU date utility to find the date 30 days ago:

$ date –date=”30 days ago” “+%Y-%m-%d”
2006-11-29

Once I had the date string from 30 days ago, I connected to the MySQL database that hosts my blog, and ran the following SQL query to disable posts for all entries greater than the date returned form GNU date:

$ mysql -umatty -p -h mysql.prefetch.net
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 16155903 to server version: 4.0.27-standard-log

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the buffer.

mysql> use matty
Database changed

mysql> UPDATE wp_posts SET comment_status = ‘closed’ WHERE post_date < '2006-11-29' AND post_status = 'publish'

mysql> quit
Bye

Now that old posts don’t have comments enabled, I am curious to see how the comment spammers react.

Configuring crontab mail destinations with MAILTO

When scripts executing from cron write to stdout or stderr, the results are emailed to the user the job is running in the context of. One way to control where the mail is sent is by adding a “MAILTO” line to the top of the crontab entry for the user that is running the cron job. Here is an example:

$ crontab -l
MAILTO=matty
* * * * * echo “hi”

Each time this job runs, the user matty will get emailed with the results of the script. You can also use /etc/aliases to assist with this on a global scale, but sometimes that isn’t an option.

Verifying GPG signatures

One of my friends recently asked me how to verify a signature that is distributed with an opensource application. Since I didn’t have a machine handy to show him, I thought I would jot this down for him in my blog. The first step in verifying a signature requires locating the public key of the individual who signed the binary. Once you locate the key and validate it’s authenticity, you can use the gpg “–import” option to import the key. Once the key is imported, you can use the gpg “–verify” option to verify that the signature is legit:

$ gpg –verify postfix-2.3.2.tar.gz.sig postfix-2.3.2.tar.gz

gpg: WARNING: using insecure memory!
gpg: please see http://www.gnupg.org/faq.html for more information
gpg: Signature made Thu Jul 27 12:49:07 2006 EDT using RSA key ID C12BCD99
gpg: Good signature from "Wietse Venema "
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: FF 96 4A 8C 96 88 7C 6E  A4 EF AD BF 48 34 E1 BB

In the above example, I used Wietse Venema’s public key (which I had previously imported) to verify the signature stored in the file postfix-2.3.2.tar.gz.sig against the source code I downloaded. Gpg is a swell piece of software!

Securing CentOS Linux installations by disabling unneeded services

To ensure that my CentOS machines run as efficiently and securely as possible, I disable a number of services after each installation. The end result is a system that accepts ssh connections on TCP port 22, and on one or more service ports that are in use by the applications hosted on the platform. To get to this state, I go through and disable numerous services that come enabled by default. Here is the default list of services that are enabled after a CentOS 4.4 installation:

$ chkconfig –list | grep on

atd             0:off   1:off   2:off   3:on    4:on    5:on    6:off
messagebus      0:off   1:off   2:off   3:on    4:on    5:on    6:off
smartd          0:off   1:off   2:on    3:on    4:on    5:on    6:off
portmap         0:off   1:off   2:off   3:on    4:on    5:on    6:off
sendmail        0:off   1:off   2:on    3:on    4:on    5:on    6:off
netfs           0:off   1:off   2:off   3:on    4:on    5:on    6:off
cups            0:off   1:off   2:on    3:on    4:on    5:on    6:off
irqbalance      0:off   1:off   2:off   3:on    4:on    5:on    6:off
rpcgssd         0:off   1:off   2:off   3:on    4:on    5:on    6:off
xfs             0:off   1:off   2:on    3:on    4:on    5:on    6:off
isdn            0:off   1:off   2:on    3:on    4:on    5:on    6:off
autofs          0:off   1:off   2:off   3:on    4:on    5:on    6:off
gpm             0:off   1:off   2:on    3:on    4:on    5:on    6:off
apmd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
crond           0:off   1:off   2:on    3:on    4:on    5:on    6:off
acpid           0:off   1:off   2:off   3:on    4:on    5:on    6:off
microcode_ctl   0:off   1:off   2:on    3:on    4:on    5:on    6:off
pcmcia          0:off   1:off   2:on    3:on    4:on    5:on    6:off
cpuspeed        0:off   1:on    2:on    3:on    4:on    5:on    6:off
xinetd          0:off   1:off   2:off   3:on    4:on    5:on    6:off
rpcidmapd       0:off   1:off   2:off   3:on    4:on    5:on    6:off
readahead_early 0:off   1:off   2:off   3:off   4:off   5:on    6:off
readahead       0:off   1:off   2:off   3:off   4:off   5:on    6:off
sshd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
anacron         0:off   1:off   2:on    3:on    4:on    5:on    6:off
network         0:off   1:off   2:on    3:on    4:on    5:on    6:off
kudzu           0:off   1:off   2:off   3:on    4:on    5:on    6:off
syslog          0:off   1:off   2:on    3:on    4:on    5:on    6:off
nfslock         0:off   1:off   2:off   3:on    4:on    5:on    6:off
rawdevices      0:off   1:off   2:off   3:on    4:on    5:on    6:off
mdmonitor       0:off   1:off   2:on    3:on    4:on    5:on    6:off
haldaemon       0:off   1:off   2:off   3:on    4:on    5:on    6:off

Several of these services are required, but several others serve no purpose in my environment, and use CPU and memory resources that would be better allocated to my applications. Since I don’t use RPC services, autofs or NFS, those are the first to get disabled:

$ /sbin/chkconfig –level 0123456 portmap off
$ /sbin/chkconfig –level 0123456 nfslock off
$ /sbin/chkconfig –level 0123456 netfs off
$ /sbin/chkconfig –level 0123456 rpcgssd off
$ /sbin/chkconfig –level 0123456 rpcidmapd off
$ /sbin/chkconfig –level 0123456 autofs off

I also don’t allow individual hosts to receive mail from the outside world, so sendmail gets nixed next:

$ /sbin/chkconfig –level 0123456 sendmail off

On server platforms, who needs printing?:

$ /sbin/chkconfig –level 0123456 cups off

Now we get to the font server, isdn capabilities, console mouse and pcmcia support. I don’t use these services on my servers, so they get disabled as well:

$ /sbin/chkconfig –level 0123456 xfs off
$ /sbin/chkconfig –level 0123456 isdn off
$ /sbin/chkconfig –level 0123456 gpm off
$ /sbin/chkconfig –level 0123456 pcmcia off

Once these services are disabled (and optionally stopped with the service command or a reboot), my netstat output looks nice and clean:

$ netstat –tcp –udp –listening

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             State      
tcp        0      0 *:ssh                       *:*                         LISTEN   

This has served me well over the years, since it reduces boot time (less rc scripts to run), and frees up additional resources for my applications (while this isn’t substantial, every page of memory helps!).

Viewing Linux partitions with fdisk and partprobe

While reading up on the Linux parted utility, I came across partprobe. Partprobe can be used to display the number of partitions on a device, and is useful for quickly viewing the layout of a given device:

$ /sbin/partprobe -s /dev/hda
/dev/hda: msdos partitions 1 2

If your looking for details on the layout of a device, I reckon fdisk is a better bet:

$ /sbin/fdisk -l /dev/hda

Disk /dev/hda: 8590 MB, 8590417920 bytes
255 heads, 63 sectors/track, 1044 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1   *           1         978     7855753+  83  Linux
/dev/hda2             979        1043      522112+  82  Linux swap

I really dig the fdisk “-l” option!