Rereading the Solaris sd.conf dynamically

I ran into an issue this week where the Solaris device tree wasn’t updated to reflect two new targets we added. After a bit of poking around, I noticed that the new targets we created weren’t present in the sd.conf configuration file. To get the host to see the storage, I first used my sdcreate script to populate the sd.conf file with the targets and LUNs I had allocated to the host:

$ cp /kernel/drv/sd.conf /kernel/drv/sd.conf.removeme.01.21.2007

$ sd-create.sh -c 2 -n sd -p lpfc 10 15 >> /kernel/drv/sd.conf

Once the sd.conf file was populated to my liking, I ran update_drv to have sd reread it’s configuration file:

$ update_drv -f sd

After the update_drv command completed, I ran the devfsadm utility to populate the /dev entries:

$ defvsadm -C

As soon as devfsadm completed, I was able to access my new storage. Niiiice!

Determining file fragmentation on EXT3 file systems

EXT3, along with most other file systems, can incur file level fragmentation over time. To see how fragmented a file on an EXT3 file system is, the filefrag utility can be run with the “-v” (verbose) option and the name of a file to check for fragmentation:

$ filefrag -v ick

Checking ick
Filesystem type is: ef53
Filesystem cylinder groups is approximately 3832
Blocksize of file ick is 4096
File size of ick is 115910586 (28299 blocks)
Discontinuity: Block 858 is at 788667 (was 787807)
Discontinuity: Block 2716 is at 790536 (was 790527)
Discontinuity: Block 4754 is at 792592 (was 792575)
Discontinuity: Block 6784 is at 794632 (was 794623)
Discontinuity: Block 23144 is at 820195 (was 811007)
Discontinuity: Block 23149 is at 821452 (was 820199)
ick: 7 extents found, perfection would be 1 extent

The easiest way I have found to reduce file fragmentation is to copy a fragmented file to a new location in the file system hierarchy (i.e., a new directory), and use that file instead of the original. I really wish there was a tool similar to VxFS’s fsadm utility to defragment files w/o having to copy them (this is impractical for file systems that store lots of data).

Debugging a PAM LDAP password expiration problem

While testing out LDAP authentication on a CentOS 4.4 Linux host this week, I noticed that the “password” statements I added to /etc/pam.d/sshd weren’t taking effect:

password    requisite     /lib/security/$ISA/pam_cracklib.so retry=3
password    sufficient    /lib/security/$ISA/pam_unix.so nullok use_authtok md5 shadow
password    sufficient    /lib/security/$ISA/pam_ldap.so use_authtok
password    required      /lib/security/$ISA/pam_deny.so

After pondering the issue for a while, I eventually started to wonder if the “passwd” utility was called by sshd to change user passwords. To see if this was the case, I decided to expire a user’s password, and then strace sshd while I logged in as that user:

$ strace -f -e trace=execve -p 2616

Process 2616 attached - interrupt to quit
--- SIGCHLD (Child exited) @ 0 (0) ---
Process 26638 attached
[pid 26638] execve("/usr/sbin/sshd", ["/usr/sbin/sshd", "-R"], [/* 14 vars */]) = 0
Process 26639 attached
Process 26639 detached
[pid 26638] --- SIGCHLD (Child exited) @ 0 (0) ---
Process 26640 attached
Process 26641 attached
[pid 26641] execve("/usr/bin/passwd", ["passwd"], [/* 14 vars */]) = 0
Process 26641 detached
[pid 26640] --- SIGCHLD (Child exited) @ 0 (0) ---
Process 26640 detached
[pid 26638] --- SIGCHLD (Child exited) @ 0 (0) ---
Process 26638 detached
--- SIGCHLD (Child exited) @ 0 (0) ---
Process 2616 detached

Sure enough, /usr/bin/passwd is called to change an expired password. To verify that the sshd daemon was the entity invoking /usr/bin/passwd, I used the strings utility to see if the string “/usr/bin/passwd” resided in the data segment of the sshd executable:

$ strings sshd | grep passwd

kerberosorlocalpasswd
/usr/bin/passwd
auth2-passwd.c
%s: struct passwd size mismatch
sshpam_passwd_conv
sshpam_auth_passwd

Once I knew that sshd called /usr/bin/passwd, I added my changes to /etc/pam.d/system-auth (which is “stacked” by pam_stack.so in /etc/pam.d/passwd), and everything worked as expected. I kinda dig the stacking capabilities that come out of the box with CentOS 4.4, since you can make a change in one location (/etc/pam.d/system-auth), and it’s effects are propogated to all service definitions in /etc/pam.d.

Generating passwords for LDAP

When I generate LDIF entries for new LDAP users, I need to populate the userPassword field with a crypt password or an SHA1 or MD5 hash. Sun’s directory server comes with the getpwenc utility to assist with this:

$ getpwenc CRYPT password
{crypt}g72FJndhHyuiob

In addition to generating crypt passwords, it can also create MD5 and SHA1 hashes.

Debugging directory server issues with DTrace

Today while installing Sun’s directory server, I kept getting the error “No components have been selected for installation in the selected directory” after selecting numerous components for installation:

Directory Server components showing a checked box will be installed.

 [X]  1   Sun Java(TM) System Directory Suite    0 bytes
 [X]  2     Sun Java(TM) System Directory Server    0 bytes
 [ ]  3     Sun Java(TM) System Directory Console Support    0 bytes
 [ ]  4   Sun Java(TM) System Administration Services    0 bytes
 [ ]  5     Sun Java(TM) System Administration Server    0 bytes
 [ ]  6     Sun Java(TM) System Administration Console    0 bytes
 [ ]  7   Sun Java(TM) System Server Console    0 bytes
 [ ]  8     Sun Java(TM) System Server Console Core    0 bytes
 [ ]  9     Java Runtime Environment    0 bytes
 [ ]  10  Sun Java(TM) System Server Basic Libraries    0 bytes

   To check a particular component, enter its number, or 0 when you are
   finished [0] {"<" goes back, "!" exits}: 0

No components have been selected for installation in the selected directory.

Since I had previously installed the directory server on the machine, I started to wonder if the directory server had placed a package repository somewhere on the file system. To see if this was the case, I fired up opensnoop from the DTraceToolkit:

$ opensnoop

  UID    PID COMM          FD PATH
                 < ..... >
    0    657 java          16 /var/sadm/install/productregistry.access
    0    657 java          16 /var/sadm/install/productregistry.access.lock
    0    657 java          16 /var/sadm/install/productregistry.access
    0    657 java          18 /var/sadm/install/productregistry.access.tmp
    0    657 java          16 /var/sadm/install/productregistry.access
    0    657 java          16 /var/sadm/install/productregistry.access.lock

Sure enough, there is a product registry file in /var/sadm/install. Once I removed it with rm:

$ rm -f /var/sadm/install/productregistry

The installation went off without a hitch. DTrace truly does bring a lot to the table, and I feel like a fish out of water when debugging difficult issues on Solaris 9, AIX and Linux hosts.

Saving output with the EXT3 logsave utility

I was doing some console work this week, and needed a way to save the output from several commands to a file. File redirection and the script utility would have worked, but I decided to use the logsave utility instead:

$ logsave foo /bin/ls
anaconda-ks.cfg
apache
foo
install.log
install.log.syslog

$ more foo

Log of /bin/ls
Sun Jan 14 08:59:45 2007

anaconda-ks.cfg
apache
foo
install.log
install.log.syslog

Sun Jan 14 08:59:45 2007
—————-

I like the fact that logsave writes one timestamp before the output is written to the file, and a second timestamp once the write operation completes. If you ever need to time something, this is one way to do it (time and timex also work, but they don’t include the output).