Solaris nscd keep-hot-count issue

While reviewing the DNS logfiles on one of my name servers, I noticed that one of my Solaris hosts was querying getupdates.sun.com once per hour:

Sep 30 02:28:05 dns named[1097]: [ID 866145 local0.info] client 1.2.3.4#47644: view internal: query: getupdates.sun.com IN A +
Sep 30 03:27:35 dns named[1097]: [ID 866145 local0.info] client 1.2.3.4#47738: view internal: query: getupdates.sun.com IN A +
Sep 30 04:27:05 dns named[1097]: [ID 866145 local0.info] client 1.2.3.4#47841: view internal: query: getupdates.sun.com IN A +
[ ..... ]

Since I had recently applied the latest connection manager patches to the server, I started to wonder if Sun had installed a cron job (or started a daemon) to periodically poll the server that hosted getupdates.sun.com. To see what process was querying getupdates.sun.com, I first reviewed the cron jobs on the system, and then I perused the list of daemons that were running. Everything checked out fine, so I fired up DTrace to see which process was actually responsible for the DNS requests. It turns out these DNS queries were coming from the nscd daemon. After reading through the nscd.conf manual page, I came across a description of the “keep-hot-count” attribute:

“This attribute allows the administrator to set the number of entries nscd(1M) is to keep current in the specified cache. value is an integer number which should approximate the number of entries frequently used during the day.”

A quick peak at /etc/nscd.conf revealed that the value of keep-hot-count is set to 20 by default:

$ egrep “keep-hot.*host” /etc/nscd.conf
keep-hot-count hosts 20

It turns out nscd was refreshing the name service information entry for this host once per hour. To disable this behavior, I set the value of keep-hot-count to 0, and the DNS queries for getupdates.sun.com ceased. Numerous admins kill nscd on systems that don’t handle lots of login sessions, and I reckon I will need to investigate this option.

Moving Solaris zones

While messing around with zones and ZFS last weekend, I decided to create a new ZFS file system called /zones to store my zones. The /zones file system would be created as a striped ZFS file system, allowing me to take advantage of ZFS’s data checksumming, compression, snapshots and zone cloning features. Since I already had one zone installed in /zones, I needed to migrate the zone to another location while I setup the new ZFS file system. There are numerous ways to do this, but I decided to use the zoneadm “move” option since it was designed for this purpose.

To begin the migration, I ran the zoneadm utility to halt the zone I wanted to migrate:

$ zoneadm -z centos halt

Once the zone was halted, I executed zoneadm with the “move” option and the location where I wanted the zone moved:

$ zoneadm -z centos move /zones/centos

Moving across file systems; copying zonepath /zones/centos...
Cleaning up zonepath /zones/centos...

Now that the zone was in a safe temporary place, I ran the zpool utility to create a 2 disk striped pool (the underlying storage is RAID protected, so there is no need to protect the pool a second time):

$ zpool create zones c1d0 c1d1

$ zpool status -v

  pool: zones
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        zones       ONLINE       0     0     0
          c1d0      ONLINE       0     0     0
          c1d1      ONLINE       0     0     0

errors: No known data errors

Now that I had a file system to store all of my zones, I once again used the zoneadm “move” option to move the zone back to it’s rightful home:

$ zoneadm -z centos move /zones/centos

cannot create ZFS dataset zones/centos: 'sharenfs' must be a string
Moving across file systems; copying zonepath /centos...
Cleaning up zonepath /centos...

I am a big fan of zones, and they are an ideal solution for server consolidation projects!

Apple file vault and good backups save the day!

I frequently travel with my Apple powerbook, and have always been concerned that someone might steal my laptop, or the disk drive would fail (this is ahuge single point of failure). To address both concerns, I enabled file vault to encrypt my home folder, and backup my data once a week to a central file store. File vault does slow down certain operations, but it works great if you use your laptop for surfing the web, checking email, and chatting with folks online.

Well — this weekend one of my concerns came to fruition. My powerbook disk drive croaked. Sometimes the drive works, sometimes it doesn’t, and it is making all of those lovely noises that drives make when they are on the fritz. I am not sure why smartmontools didn’t pick up the failure, and unfortunately I can’t get the machine to stay up long enough to extract the SMART attributes. Luckly I have a service contract on my laptop, so getting the drive replaced should be relatively straight forward (I have never dealt with AppleCare, so we will see).

Since I use file vault and backup my data weekly to a safe location, I don’t have to worry about the technician or drive manufacturer looking through the data on my laptop (not that there is anything worth looking at, but I am a privacy nut), and I also don’t have much to do to get my machine back online when it comes back form AppleCare (it should be as easy as installing OS X Tiger, and scp’ing my data from my central file store).

As most people who have met me know, I am a FANATIC when it comes to backing up data. I typically backup my data once a week to a server with redundant disk drives, those disk drives are periodically swapped out, and then I archive the data on those drives to a one or more DVDs which I store in a safety deposit box. Now you might ask me, Matty, why are you such a fanatic when it comes to backups? Well, I have collected technical notes, contacts and documentation for the last ten years, and I couldn’t imagine losing all of the data I have collected. This backup strategy may sound excessive to some, but using this practice means that my powerbook disk failure is an inconvenience, and not a life traumatizing event. :)

P.S. Anyone use their parents house for DR? ;)

Running Linux applications in Solaris Linux branded zones

While playing around with the latest version of Nevada this week, I decided to see how well Linux branded zones work. In case your not following the Sun development efforts, Linux branded zones allow you to run Linux ELF executables unmodified on Solaris hosts. This is pretty interesting, and I definitely wanted to take this technology for a test drive. After reading through the documentation in the brandz community, I BFU’ed my Nevada machine to the latest nightly build, and installed the packages listed on the brandz download page. Since brandz currently only supports CentOS 3.0 – 3.7 and the Linux 2.4 kernel series, I first had to download the three CentoS 3.7 iso images (branded zones currently don’t support CentOS 3.8 without some hacking):

$ cd /home/matty/CentOS

$ wget http://www.gtlib.gatech.edu/pub/centos/3.7/isos/i386/CentOS-3.7-i386-bin1of3.iso

$ wget http://www.gtlib.gatech.edu/pub/centos/3.7/isos/i386/CentOS-3.7-i386-bin2of3.iso

$ wget http://www.gtlib.gatech.edu/pub/centos/3.7/isos/i386/CentOS-3.7-i386-bin3of3.iso

After I retrieved the ISO images, I needed to create a branded zone. Creating Linux branded zones is a piece of cake, and is accomplished by running the zonecfg utility with the “-z” option and a name to assign to your zone, and then specifying one or more parameters inside the zone configuration shell. Here is the configuration I used with my test zone:

$ zonecfg -z centostest

centostest: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:centostest> create -t SUNWlx
zonecfg:centostest> add net
zonecfg:centostest:net> set physical=ni0
zonecfg:centostest:net> set address=192.168.1.25
zonecfg:centostest:net> end
zonecfg:centostest> set zonepath=/zones/centostest
zonecfg:centostest> set autoboot=true
zonecfg:centostest> verify
zonecfg:centostest> commit
zonecfg:centostest> exit

This zone configuration is pretty basic. It contains one network interface (when you boot the zone, a virtual interface is configured on that interface with the address passed to the address attribute), a location to store the zone data, and it is configured to automatically boot when the system is bootstrapped. Next I needed to install the CentOS binaries in the zone. To install the CentOS 3.7 binaries in the new zone I created, I ran the zoneadm utility with the ‘install’ option, and passed the directory with the CentOS ISO images as an argument:

$ zoneadm -z centostest install -v -d /home/matty/CentOS

Verbose output mode enabled.
Installing zone "centostest" at root "/zones/centostest"
  Attempting ISO-based install from directory:
    "/home/matty/CentOS"
Checking possible ISO
  "/home/matty/CentOS/CentOS-3.7-i386-bin1of3.iso"...
    added as lofi device "/dev/lofi/1"
Attempting mount of device "/dev/lofi/1"
     on directory "/tmp/lxisos/iso.1"... succeeded.
Checking possible ISO
  "/home/matty/CentOS/CentOS-3.7-i386-bin2of3.iso"...
    added as lofi device "/dev/lofi/2"
Attempting mount of device "/dev/lofi/2"
     on directory "/tmp/lxisos/iso.2"... succeeded.
Checking possible ISO
  "/home/matty/CentOS/CentOS-3.7-i386-bin3of3.iso"...
    added as lofi device "/dev/lofi/3"
Attempting mount of device "/dev/lofi/3"
     on directory "/tmp/lxisos/iso.3"... succeeded.
Checking for distro "/usr/lib/brand/lx/distros/centos35.distro"...
Checking iso file mounted at "/tmp/lxisos/iso.1"...
read discinfo file "/tmp/lxisos/iso.1/.discinfo"
  ISO "/tmp/lxisos/iso.1": Serial "1144177644.47"
    Release "CentOS [Disc Set 1144177644.47]" Disc 1
Checking iso file mounted at "/tmp/lxisos/iso.2"...
read discinfo file "/tmp/lxisos/iso.2/.discinfo"
  ISO "/tmp/lxisos/iso.2": Serial "1144177644.47"
    Release "CentOS [Disc Set 1144177644.47]" Disc 2
Checking iso file mounted at "/tmp/lxisos/iso.3"...
read discinfo file "/tmp/lxisos/iso.3/.discinfo"
  ISO "/tmp/lxisos/iso.3": Serial "1144177644.47"
    Release "CentOS [Disc Set 1144177644.47]" Disc 3
Checking for distro "/usr/lib/brand/lx/distros/centos36.distro"...
Checking iso file mounted at "/tmp/lxisos/iso.1"...
read discinfo file "/tmp/lxisos/iso.1/.discinfo"
  ISO "/tmp/lxisos/iso.1": Serial "1144177644.47"
    Release "CentOS [Disc Set 1144177644.47]" Disc 1
Checking iso file mounted at "/tmp/lxisos/iso.2"...
read discinfo file "/tmp/lxisos/iso.2/.discinfo"
  ISO "/tmp/lxisos/iso.2": Serial "1144177644.47"
    Release "CentOS [Disc Set 1144177644.47]" Disc 2
Checking iso file mounted at "/tmp/lxisos/iso.3"...
read discinfo file "/tmp/lxisos/iso.3/.discinfo"
  ISO "/tmp/lxisos/iso.3": Serial "1144177644.47"
    Release "CentOS [Disc Set 1144177644.47]" Disc 3
Checking for distro "/usr/lib/brand/lx/distros/centos37.distro"...
Checking iso file mounted at "/tmp/lxisos/iso.1"...
read discinfo file "/tmp/lxisos/iso.1/.discinfo"
  ISO "/tmp/lxisos/iso.1": Serial "1144177644.47"
    Release "CentOS [Disc Set 1144177644.47]" Disc 1
Added ISO "/tmp/lxisos/iso.1" as disc 1
Checking iso file mounted at "/tmp/lxisos/iso.2"...
read discinfo file "/tmp/lxisos/iso.2/.discinfo"
  ISO "/tmp/lxisos/iso.2": Serial "1144177644.47"
    Release "CentOS [Disc Set 1144177644.47]" Disc 2
Added ISO "/tmp/lxisos/iso.2" as disc 2
Checking iso file mounted at "/tmp/lxisos/iso.3"...
read discinfo file "/tmp/lxisos/iso.3/.discinfo"
  ISO "/tmp/lxisos/iso.3": Serial "1144177644.47"
    Release "CentOS [Disc Set 1144177644.47]" Disc 3
Added ISO "/tmp/lxisos/iso.3" as disc 3
Installing distribution 'CentOS [Disc Set 1144177644.47]'...
Installing cluster 'desktop'
Installing zone miniroot.
Installing miniroot from ISO image 1 (of 3)
RPM source directory: "/tmp/lxisos/iso.1/RedHat/RPMS"
Attempting to expand 30 RPM names...
Installing RPM "SysVinit-2.85-4.4.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "basesystem-8.0-2.centos.0.noarch.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "bash-2.05b-41.5.centos.0.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "beecrypt-3.0.1-0.20030630.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "bzip2-libs-1.0.2-11.EL3.4.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "coreutils-4.5.3-28.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "elfutils-0.94-1.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "elfutils-libelf-0.94-1.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "filesystem-2.2.1-3.centos.1.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "glibc-2.3.2-95.39.i586.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "glibc-common-2.3.2-95.39.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "gpm-1.19.3-27.2.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "initscripts-7.31.30.EL-1.centos.1.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "iptables-1.2.8-12.3.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "iptables-ipv6-1.2.8-12.3.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "kernel-utils-2.4-8.37.14.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "laus-libs-0.1-70RHEL3.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "libacl-2.2.3-1.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "libattr-2.2.0-1.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "libgcc-3.2.3-54.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "libtermcap-2.0.8-35.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "ncurses-5.3-9.4.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "pam-0.75-67.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "popt-1.8.2-24_nonptl.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "rpm-4.2.3-24_nonptl.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "rpm-libs-4.2.3-24_nonptl.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "setup-2.5.27-1.noarch.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "termcap-11.0.1-17.1.noarch.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "zlib-1.1.4-8.1.i386.rpm" to miniroot at 
    "/zones/centostest"...
Installing RPM "centos-release-3-7.1.i386.rpm" to miniroot at 
    "/zones/centostest"...
Setting up the initial lx brand environment.
System configuration modifications complete!
Duplicating miniroot; this may take a few minutes...

Booting zone miniroot...
Miniroot zone setup complete.

Installing zone 'centostest' from ISO image 1.
RPM source directory: "/zones/centostest/root/iso/RedHat/RPMS"
Attempting to expand 667 RPM names...
Installing 433 RPM packages; this may take several minutes...

Preparing...                ##################################################
libgcc                      ##################################################
setup                       ##################################################
filesystem                  ##################################################
hwdata                      ##################################################
redhat-menus                ##################################################
mailcap                     ##################################################
XFree86-libs-data           ##################################################
basesystem                  ##################################################
gnome-mime-data             ##################################################

[.....]

After the brandz installer finished installing the CentOS 3.7 RPMs, I used the zoneadm ‘boot’ option to start the zone:

$ zoneadm -z centostest boot

To view the console output while the zone was booting, I immediately fired up the zlogin utility to console into the new Linux branded zone, and ran a few commands to see what the environment looked like after the zone was booted:

$ zlogin -C centostest

[Connected to zone 'centostest' console]  [  OK  ]
Activating swap partitions:  [  OK  ]
Checking filesystems [  OK  ]
Mounting local filesystems:  [  OK  ]
Enabling swap space:  [  OK  ]
modprobe: Can't open dependencies file /lib/modules/2.4.21/modules.dep (No such file or directory)
INIT: Entering runlevel: 3
Entering non-interactive startup
Starting sysstat:  [  OK  ]
Starting system logger: [  OK  ]
Starting kernel logger: [  OK  ]
Starting automount: No Mountpoints Defined[  OK  ]
Starting cups: [  OK  ]
Starting sshd:[  OK  ]
Starting crond: [  OK  ]
Starting atd: [  OK  ]
Rotating KDC list [  OK  ]


CentOS release 3.7 (Final)
Kernel 2.4.21 on an i686

centostest login: root

$ uname -a

Linux centos 2.4.21 BrandZ fake linux i686 i686 i386 GNU/Linux

$ cat /proc/cpuinfo

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 6
model name      : Intel Celeron(r)
stepping        : 5
cpu MHz         : 1662.136
cache size      : 2048 KB
fdiv_bug        : no
hlt_bug         : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
flags           : fpu pse tsc msr mce cx8 sep mtrr pge cmov mmx fxsr sse sse2 ss

Yum works swell in a branded zone, and most of the tools you typically use work out of the box. Linux branded zones are wicked cool, and I can see tons of uses for them. Some folks are dead set on running Linux instead of Solaris, which means they can’t take advantage of things like ZFS, FMA and DTrace. If you need to better understand your application and the way it interacts with the system, or if you want to take advantage of the stability the Solaris kernel brings to production system, you can fire up a branded zone and run your application transparently on a Solaris system. You can also easily transport your applications between a CentOS server and a branded zone, use DTrace to profile the application, and then take any performance wins back to your Linux server. Who can argue with that? :)

Mapping pfiles output to files

While I was analyzing the performance characteristics of one of my Solaris 9 Oracle database servers, I needed to map a file descriptor listed in the pfiles output to the actual file name on the file system (Solaris 10 provides this information in the pfiles output, which is yet another reason to run Solaris 10). When you run pfiles on a process or core file, it gives you the type (e.g., socket, fifo, regular file, etc) of file that the file descriptor references, and the inode number associated with the file:

$ pfiles 12345

  [ ..... ]

 269: S_IFREG mode:0660 dev:228,119007 ino:61 uid:102 gid:315 size:10729046016
      O_RDWR|O_DSYNC|O_LARGEFILE FD_CLOEXEC
  [ ..... ]

To map the inode listed after the “ino:” field to the file name, I first used the “dev:” identifier to locate the device the file was located on:

$ cd /dev/vx/dsk/foodg

$ ls -la

total 4
drwxr-xr-x   2 root     root         512 Sep 18  2005 .
drwxr-xr-x   5 root     root         512 Sep 18  2005 ..
brw-------   1 root     root     228,119000 Sep 18  2005 u16
brw-------   1 root     root     228,119001 Sep 18  2005 u17
brw-------   1 root     root     228,119002 Sep 18  2005 u18
brw-------   1 root     root     228,119003 Sep 18  2005 u19
brw-------   1 root     root     228,119004 Sep 18  2005 u20
brw-------   1 root     root     228,119005 Sep 18  2005 u57
brw-------   1 root     root     228,119006 Sep 18  2005 u58
brw-------   1 root     root     228,119007 Sep 18  2005 u59
brw-------   1 root     root     228,119008 Sep 18  2005 u60
brw-------   1 root     root     228,119009 Sep 18  2005 u61
brw-------   1 root     root     228,119010 Sep 18  2005 u62
brw-------   1 root     root     228,119011 Sep 18  2005 u63
brw-------   1 root     root     228,119012 Sep 18  2005 u64
brw-------   1 root     root     228,119013 Sep 18  2005 u65
brw-------   1 root     root     228,119014 Sep 18  2005 u66

Once I had the device name and associated mount point, I used the trusty old find utility to locate the file by it’s inode number:

$ find /u59 -inum 61 -ls

   61 10477592 -rw-rw----   1 oracle   dba      10729046016 Sep 18 13:15 /u59/oradata/blatch/data01.dbf

Some folks might be wondering why I didn’t use lsof. Well, unfortunately the version of lsof available in various package repositories isn’t able to interpret the VxFS metadata, and I would have spent more time building the new package than analayzing the problem I was trying to solve. Viva la Solaris!

Opensource rocks!

I received two more awesome patches for ssl-cert-check. Ken Gallo sent me a patch to allow ssl-cert-check to process certificates stored in PKCS#12 databases, and I am hoping to integrate his changes this upcoming weekend. Quanah Gibson-Mount sent me a patch that adds nagios support to ssl-cert-check, and also adjusts the default binary locations to work on more systems out of the box. Since numerous people are contributing patches to make opensource solutions better, I truly do think opensource is the way to go (I think closed source solutions have value as well)! With opensource solutions, you have access to the source, you can change things to suit your needs, and the changes you make will most likely benefit lots of people. You can download the new version of ssl-cert-check from my website, and I would like to thank Quanah and Ken for their awesome contributions!