Enabling large file support dynamically with VxFS

I recently encountered a VxFS file system that didn’t support largefiles. This issue was causing one of our Oracle databases to complain, which was preventing us from using datafiles optimized for our application access patterns. Since the file system was a Veritas File System (VxFS), I was able to fix this problem with the fsadm utility:

$ /usr/lib/fs/vxfs/fsadm -F vxfs -o largefiles /u01

$ mount -p | grep u01
/dev/vx/dsk/oradg/oravol01 – /u01 vxfs – no rw,suid,delaylog,largefiles,ioerror=mwdisable

This operation can be run against mounted live file systems, which is great for production environments.

Verifying digital signatures

I recently downloaded the samhain file integrity verification suite, and wanted to verify the authenticity of the package. The samhain developers distribute samhain as tar archive, which includes the source code and a detached ASCII signature file:

$ /usr/sfw/bin/gtar tvfz samhain-current.tar.gz

-rw-r--r-- 1000/100    1302539 2005-09-22 16:05:35 samhain-2.0.10a.tar.gz
-rw-r--r-- 1000/100        189 2005-09-22 16:06:29 samhain-2.0.10a.tar.gz.asc

To verify the source code that is contained in the file samhain-2.0.10a.tar.gz using the detached ASCII signature stored in the file samhain-2.0.10a.tar.gz.asc, I downloaded the samhain developers public key, and imported it into my keychain with the gpg “–import” option:

$ gpg –import sh.pub

gpg: key 0F571F6C: public key "Rainer Wichmann " imported
gpg: Total number processed: 1
gpg:               imported: 1

Once the public key was imported, I used gpg’s “–fingerprint” option to extract a digital fingerprint from each public key:

$ gpg –fingerprint

/home/matty/.gnupg/pubring.gpg
------------------------------
pub  1024D/0F571F6C 1999-10-31 Rainer Wichmann 
     Key fingerprint = EF6C EF54 701A 0AFD B86A  F4C3 1AAD 26C8 0F57 1F6C
uid                            Rainer Wichmann 
sub  1024g/9DACAC30 1999-10-31

Once I had the fingerprint and the keyid, I verified both pieces of information with a trusted source. After I verified the authenticity of the key I imported, I used gpg’s “–verify” option along with the signature file (an encrypted hash) to verify the files contents with the key I just imported:

$ gpg –verify samhain-2.0.10a.tar.gz.asc samhain-2.0.10a.tar.gz

gpg: Signature made Thu 22 Sep 2005 04:06:29 PM EDT using DSA key ID 0F571F6C
gpg: Good signature from "Rainer Wichmann "
gpg:                 aka "Rainer Wichmann "
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: EF6C EF54 701A 0AFD B86A  F4C3 1AAD 26C8 0F57 1F6C

Verifying signatures can help ensure that the source code you download hasn’t been replaced with a malicious variant. The gpg(1) manual contains lots of information on signatures, and provides a detailed breakdown of the available gpg options.

Debugging Solaris in.rarpd issues

While performing a routine jumpstart this week, one of my friends was receiving a steady stream of Timeouts while attempting to jumpstart a system:

{0} ok boot net – install
Boot device: /pci@83,4000/network@1,1 File and args: – install
Timeout waiting for ARP/RARP packet
Timeout waiting for ARP/RARP packet
Timeout waiting for ARP/RARP packet
Timeout waiting for ARP/RARP packet
Timeout waiting for ARP/RARP packet
Timeout waiting for ARP/RARP packet
[ ….. ]

He had used the jumpstart server earlier in the week to build a system, and was uncertain why the server he was jumpstarting couldn’t get an IP address. He asked me if I could take a look at the problem, so I fired up snoop on the jumpstart server to see if the RARP requests were reaching the server:

$ snoop -d hme0 ether 0:e0:0:c4:d8:e3
OLD-BROADCAST -> (broadcast) RARP C Who is 0:e0:0:c4:d8:e3 ?
OLD-BROADCAST -> (broadcast) RARP C Who is 0:e0:0:c4:d8:e3 ?
OLD-BROADCAST -> (broadcast) RARP C Who is 0:e0:0:c4:d8:e3 ?
OLD-BROADCAST -> (broadcast) RARP C Who is 0:e0:0:c4:d8:e3 ?

The requests were indeed getting to the server, but for some reason the server wasn’t sending anything back to the client. I double checked all the configuration files (e.g., /etc/bootparams, /etc/ethers, /etc/hosts), and then double checked that the host had valid entry in /etc/tftpboot. Everything appeared to be correct, so I fired up truss to watch what in.rarp was doing:

$ truss -leaf -x all -p 8192

8192/2:         psargs: /usr/sbin/in.rarpd -a
8192/4:         getmsg(5, 0xFEEFBF94, 0xFEEFB788, 0xFEEFAF84) (sleeping...)
8192/3:         getmsg(4, 0xFEFFBF94, 0xFEFFB788, 0xFEFFAF84) (sleeping...)
8192/2:         lwp_park(0, 0x00000000, 0)      (sleeping...)
8192/4:         getmsg(5, 0xFEEFBF94, 0xFEEFB788, 0xFEEFAF84)   = 0
8192/4:         open(0xFEDD49AC, 0)                             = 7
8192/4:              0xFEDD49AC: "/etc/ethers"
8192/4:         fstat64(7, 0xFEEFA258)                          = 0
8192/4:         fstat64(7, 0xFEEFA100)                          = 0
8192/4:         ioctl(7, 0x00005401, 0xFEEFA1E4)                Err#25 ENOTTY
8192/4:         read(7, 0x0002697C, 8192)                       = 744
8192/4:              0x0002697C: " 8 : 0 : 2 0 : c 7 : 9 e"..
8192/4:         llseek(7, 0xFFFFFFFFFFFFFFE5, 1)                = 717
8192/4:         close(7)                                        = 0
8192/4:         door(6, 0xFEEF83B0)                             = 0
8192/4:         door(6, 0xFEEF8398)                             = 0
8192/4:         getmsg(5, 0xFEEFBF94, 0xFEEFB788, 0xFEEFAF84) (sleeping...)
8192/4:         getmsg(5, 0xFEEFBF94, 0xFEEFB788, 0xFEEFAF84)   = 0
8192/4:         open(0xFEDD49AC, 0)                             = 7
8192/4:              0xFEDD49AC: "/etc/ethers"
8192/4:         fstat64(7, 0xFEEFA258)                          = 0
8192/4:         fstat64(7, 0xFEEFA100)                          = 0
8192/4:         ioctl(7, 0x00005401, 0xFEEFA1E4)                Err#25 ENOTTY
8192/4:         read(7, 0x0002697C, 8192)                       = 744
8192/4:              0x0002697C: " 8 : 0 : 2 0 : c 7 : 9 e"..

The truss output indicated that certain RARP requests were being processed by in.rarpd, but the MAC address of the system my friend was attempting to jumpstart was not showing up. This led me to believe that a cable was unplugged, or something had happened to the jumpstart interface. I checked the ifconfig(1m) output and several files in /etc, and immediately noticed that the IP address assigned to ge0 had changed. A quick check of the in.rarpd source code* revealed that each physical interface is initialized when in.rarpd starts, so when the IP address changed, in.rarpd knew nothing about the new network, and therefore ignored RARP requests for that network. Once I knew exactly what caused the problem, I restarted in.rarpd with the “-a” (process requests on all interfaces) option, and the client was able to receive an IP address. I thoroughly enjoy debugging problems like this, and I had a fun time learning how in.rarpd works! Wh00t wh00t!!

* I will BLOG about my in.rarp source code disection in a later post.