Ridding your Solaris host of zombie processes

We encountered a nasty bug in our backup software this week. When this bug is triggered, each job (one process is created per job) that completes will turn into a zombie. After a few days we will have hundreds or even thousands of zombie processes, which if left unchecked will eventually lead to the system-side process table filling up. Solaris comes with a nifty tool to help deal with zombies (no, they don’t ship you a shotgun with your media kit), and it comes by the name preap. To use preap, you can pass it the PID of the zombie process you want to reap:

$ ps -ef | grep defunct

    root   646   426   0        - ?           0:00 <defunct>
    root  1489 12335   0 09:32:54 pts/1       0:00 grep defunct

$ preap 646
646: exited with status 0

This will cause the process to exit, and the kernel can then free up the resources that were allocated by that process. On a related note, if you haven’t seen the movie zombieland you are missing out!!!! That movie is hilarious!

Sorting data by dates, numbers and much much more

Every year or two I try to re-read manual pages and documentation about my favorite UNIX tools (bash, awk, sed, grep, etc.). Each time I do this I pick up some cool new nugget of information, and refresh my mind on things that I may have forgot. While reading through an article on sort, I came across the following note about the sort “-k” (field to sort by) option:

“Further modifications of the sorting algorithm are possible with these options: -d (use only letters, digits, and blanks for sort keys), -f (turn off case recognition and treat lowercase and uppercase characters as identical), -i (ignores non-printing ASCII characters), -M (sorts lines using three-letter abbreviations of month names: JAN, FEB, MAR, …), -n (sorts lines using only digits, -, and commas, or other thousands separator). These options, as well as -b and -r, can be used as part of a key number, in which case they apply to that key only and not globally, like they do when they are used outside key definitions.”

This is crazy useful, and I didn’t realize sort could be used to sort by date. I put this to use today, when I had to sort a slew of data that looked similar to this:

Jun 10 05:17:47 some_data_string
May 20 05:17:48 some_data_string2
Jun 17 05:17:49 some_data_string0

I was able to first sort by the month, and then by the day of the month:

$ awk ‘{printf “%-3s %-2s %-8s %-50s\n”, $1, $2, $3, $4 }’ data | sort -k1M -k2n
May 17 05:17:49 some_data_string0
Jun 01 05:17:47 some_data_string
Jun 20 05:17:48 some_data_string2

Awesome stuff, and I will definitely be using this again in the future!!!

Displaying GPG public keys in ASCII format

I was debugging a gpg issue earlier this week, and needed to dump the contents of a public key in some type of human readable form. After a bit of googling I came across the crazy awesome pgpdump utility, which provides a command line interface to display the contents of a GPG public key. To use this tool, you can pass the key file as an argument to pgpdump:

$ gpg –export -a > $HOME/pub.asc

$ pgpdump $HOME/pub.asc

Old: Public Key Packet(tag 6)(418 bytes)
	Ver 4 - new
	Public key creation time - Tue Jun 22 10:33:25 EDT 2010
	Pub alg - DSA Digital Signature Algorithm(pub 17)
	DSA p(1024 bits) - ...
	DSA q(160 bits) - ...
	DSA g(1021 bits) - ...
	DSA y(1023 bits) - ...
Old: User ID Packet(tag 13)(31 bytes)
	User ID - Test Key 
Old: Signature Packet(tag 2)(96 bytes)
	Ver 4 - new
	Sig type - Positive certification of a User ID and Public Key packet(0x13).
	Pub alg - DSA Digital Signature Algorithm(pub 17)
	Hash alg - SHA1(hash 2)
	Hashed Sub: signature creation time(sub 2)(4 bytes)
		Time - Tue Jun 22 10:33:25 EDT 2010
	Hashed Sub: key flags(sub 27)(1 bytes)
		Flag - This key may be used to certify other keys
		Flag - This key may be used to sign data
	Hashed Sub: preferred symmetric algorithms(sub 11)(5 bytes)
		Sym alg - AES with 256-bit key(sym 9)
		Sym alg - AES with 192-bit key(sym 8)
		Sym alg - AES with 128-bit key(sym 7)
		Sym alg - CAST5(sym 3)
		Sym alg - Triple-DES(sym 2)
	Hashed Sub: preferred hash algorithms(sub 21)(3 bytes)
		Hash alg - SHA1(hash 2)
		Hash alg - SHA256(hash 8)
		Hash alg - RIPEMD160(hash 3)
	Hashed Sub: preferred compression algorithms(sub 22)(3 bytes)
		Comp alg - ZLIB (comp 2)
		Comp alg - BZip2(comp 3)
		Comp alg - ZIP (comp 1)
	Hashed Sub: features(sub 30)(1 bytes)
		Flag - Modification detection (packets 18 and 19)
	Hashed Sub: key server preferences(sub 23)(1 bytes)
		Flag - No-modify
	Sub: issuer key ID(sub 16)(8 bytes)
		Key ID - 0xA7B71783E5016F25
	Hash left 2 bytes - ad 6b 
	DSA r(157 bits) - ...
	DSA s(159 bits) - ...
		-> hash(DSA q bits)
Old: Public Subkey Packet(tag 14)(525 bytes)
	Ver 4 - new
	Public key creation time - Tue Jun 22 10:33:25 EDT 2010
	Pub alg - ElGamal Encrypt-Only(pub 16)
	ElGamal p(2048 bits) - ...
	ElGamal g(3 bits) - ...
	ElGamal y(2046 bits) - ...
Old: Signature Packet(tag 2)(73 bytes)
	Ver 4 - new
	Sig type - Subkey Binding Signature(0x18).
	Pub alg - DSA Digital Signature Algorithm(pub 17)
	Hash alg - SHA1(hash 2)
	Hashed Sub: signature creation time(sub 2)(4 bytes)
		Time - Tue Jun 22 10:33:25 EDT 2010
	Hashed Sub: key flags(sub 27)(1 bytes)
		Flag - This key may be used to encrypt communications
		Flag - This key may be used to encrypt storage
	Sub: issuer key ID(sub 16)(8 bytes)
		Key ID - 0xA7B71783E5016F25
	Hash left 2 bytes - b1 38 
	DSA r(158 bits) - ...
	DSA s(159 bits) - ...
		-> hash(DSA q bits)

Pgpdump will display the algorithms used to create the key, as well as the key-lengths that were used. This is amazingly helpful when debugging key-related issues (hash algorithm mismatches, key-size discrepancies, etc.), and I will definitely be adding this tool to my SysAdmin toolkit!

Getting notified when new hosts appear on your network

I had to debug an interesting network problem a few weeks back, and wanted to see when new hosts appeared on my network. While debugging the issue, I needed to find a way to get notified when a new host appeared (I didn’t want to sit at a terminal reviewing the output from snoop and tcpdump). Enter arpwatch, which can be used to send alerts the first time a client issues an ARP request. This is actually quite handy, and the alerts you get my e-mail are rather useful:

$ arpwatch -i br0

From: root (Arpwatch)
To: root
Subject: new station

            hostname: foo.bar.com
          ip address: 192.168.1.18
    ethernet address: 0:1c:b3:c2:80:2f
     ethernet vendor: 
           timestamp: Friday, April 30, 2010 23:07:32 -0400

There are a slew of options to control who gets the e-mail, whether to use a saved packet capture instead of an active network connection, etc. Arpwatch is a pretty cool tool, and saved me a lot of time and hassle!