Viewing Solaris security and reliability updates

I previously discussed using pca to get security updates. One thing I didn’t realize at the time was pca’s ability to list or install only the patches that are classified as security and reliability updates. This ability to filter patches is accomplished by adding the “r” (reliability updates) or “s” (security updates) character to one of the available patch group operands (e.g., missing, installed, all, total, unbundled, bad). The following example shows how the “r” and “s” characters can be used to list all patches that are classified as security and reliability updates:

$ pca -l missingrs

Using /var/tmp/patchdiag.xref from Jan/26/07
Host: tigger (SunOS 5.10/Generic_118833-24/sparc/sun4u)

Patch  IR   CR RSB Age Synopsis
------ -- - -- --- --- -------------------------------------------------------
118666 09 < 10 -S-  16 J2SE 5.0: update 10 patch (5.0u10)
118667 09 < 10 -S-  16 J2SE 5.0: update 10 patch (5.0u10), 64bit
119213 10 < 11 -S-  17 NSS_NSPR_JSS 3.11.4: NSPR 4.6.4 / NSS 3.11.4 / JSS 4.2.4
119254 32 < 34 RS-   2 SunOS 5.10: Install and Patch Utilities Patch
119850 21 < 22 R--  18 SunOS 5.10: mpt driver & picl plugins patch
120719 01 < 02 RS-  16 SunOS 5.10 : SunFreeware gzip patch
120824 -- < 07 R--  12 SunOS 5.10: SunBlade T6300 & Sun Fire (T1000, T2000) platform patc
121118 08 < 10 R--  25 SunOS 5.10: Sun Update Connection System Client 1.0.8
122032 02 < 03 R--  16 SunOS 5.10: Update timezones patch
124943 -- < 01 -S-  16 SunOS 5.10: SunFreeware gzip man pages patch
124997 -- < 01 RS-  10 SunOS 5.10: /usr/bin/tip patch

If you want to install all of the available security and reliability updates, you can specify the "r" or "s" character as part of the installation process:

$ pca -i missingrs

Using /var/tmp/patchdiag.xref from Jan/26/07
Host: tigger (SunOS 5.10/Generic_118833-24/sparc/sun4u)

Patch  IR   CR RSB Age Synopsis
------ -- - -- --- --- -------------------------------------------------------
118666 09 < 10 -S-  16 J2SE 5.0: update 10 patch (5.0u10)
                       Download 1/11: done
                       Install  1/11: done

118667 09 < 10 -S-  16 J2SE 5.0: update 10 patch (5.0u10), 64bit
                       Download 2/11: done
                       Install  2/11: done

119213 10 < 11 -S-  17 NSS_NSPR_JSS 3.11.4: NSPR 4.6.4 / NSS 3.11.4 / JSS 4.2.4
                       Download 3/11: done
                       Install  3/11: done
    < ..... >

I wish I would have noticed this earlier, since it would have saved me having to write a shell wrapper. :)

Fixing a broken Solaris zone

I applied the latest set of patches to my x86 Solaris 10 server this morning, and after the server was rebooted I noticed that my zones didn’t start. When I ran the zoneadm utility with the “list” option, all of the zones were in the “installed” state (they should be in the running state since the autoboot variables was set to true):

$ zoneadm list -vc

  ID NAME             STATUS         PATH                          
   0 global           running        /                             
   - z1-t             installed      /zones/z1-t               
   - z2-d             installed      /zones/z2-d               
   - z3-p             installed      /zones/z3-p             

At first I thought the zones service might be in the maintenance state, but after reviewing the output from the svcs command, that theory turned out to be incorrect:

$ svcs -a | grep zones

online          8:39:22 svc:/system/zones:default

Since the box contained several critical services, I decided to start the zones by hand and perform mostmortem analysis after the zones were back up and operational. When I ran zoneadm with the the “boot” option and the name of the zone to boot, I was greeted with the following error:

$ zoneadm -z dns boot

zoneadm: zone 'dns': Failed to initialize privileges: No such file or directory
zoneadm: zone 'dns': call to zoneadmd failed

Oh good grief! After reviewing my notes, I noticed that I had applied patch 122663-06 (a libezonecfg patch) as part of the patch bundle. Configurable zone privileges are coming as part of Solaris 10 update 3, and it looks like they prematurely made their way into a Solaris 10 patch. Since I have not had a chance to play with configurable privileges, I decided to create a new zone to see if zonecfg worked ok, and also to see if configurable privileges required additional attributes:

$ zonecfg -z test

test: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:test> create
zonecfg:test> info
zonepath: 
autoboot: false
pool: 
inherit-pkg-dir:
        dir: /lib
inherit-pkg-dir:
        dir: /platform
inherit-pkg-dir:
        dir: /sbin
inherit-pkg-dir:
        dir: /usr
zonecfg:test> set zonepath=/zones/test
zonecfg:test> commit
ld.so.1: zonecfg: fatal: relocation error: file /usr/sbin/zonecfg: symbol zonecfg_add_index: referenced symbol not found
Killed

Well that isn’t good, and the output from the info command doesn’t seem to indicate that new attributes were added. I needed to get the box up and running, so I decided to try to back out patch 122663-06. When I ran patchrm to remove the patch, it bombed out since it wasn’t able to start the zones:

$ patchrm 122663-06

Validating patches...

Loading patches installed on the system...

Done!

Checking patches that you specified for removal.

Done!

Approved patches will be removed in this order:

122663-06 
Preparing checklist for non-global zone check...

Checking non-global zones...

Booting non-global zone dns for patch check...
 ERROR: unable to boot zone: problem running  on zone : error 1
zoneadm: zone 'dns': Failed to initialize privileges: No such file or directory
zoneadm: zone 'dns': call to zoneadmd failed

Can not boot non-global zone dns

Gak! Once I realized that backing out the patch with patchrm wouldn’t work, I decided to back up the libzonecfg shared library that 122663-06 had installed, and copy the previous version over it. To find the previous version, I used the find command in the /var/sadm directory:

$ cd /var/sadm

$ find . -name 122663-06

./pkg/SUNWcsr/save/pspool/SUNWcsr/save/122663-06
./pkg/SUNWcsr/save/122663-06
./pkg/SUNWzoneu/save/pspool/SUNWzoneu/save/122663-06
./pkg/SUNWzoneu/save/122663-06
./patch/122663-06

After I located the patch directories, I looked for the file named undo.Z. The undo.Z file contains a backup of each file the patch overwrites, and is used by the patchrm utility to restore a server to it’s previosu state. To find the right undo.Z file, I ran the find command in the pkg/SUNWzoneu directory, and then used the ls “-i” (print inode) and “-l” (long output) options to print the inode number and size of each undo.Z file I found:

$ ls -li ./pkg/SUNWzoneu/save/pspool/SUNWzoneu/save/122663-06/*.Z ./pkg/SUNWzoneu/save/122663-06/*.Z

    102041 -rw-r--r--   1 root     root      158534 Oct 29 08:56 ./pkg/SUNWzoneu/save/122663-06/undo.Z
    101589 -rw-r--r--   1 root     root      158534 Oct 29 08:56 ./pkg/SUNWzoneu/save/pspool/SUNWzoneu/save/122663-06/undo.Z

Since the size and timestamps on the files I located were identical (as a side note — I am curious why Sun keeps two copies of the undo.Z file. If anyone knows, please add your thoughts to the comment section. ), I copied one of the files to /tmp, and used the uncompress and pkgadd utilities to extract the file to /tmp/u:

$ cp undo.Z /tmp

$ cd /tmp

$ uncompress undo.Z

$ pkgadd -s /tmp/u -d undo

The following packages are available:
  1  SUNWzoneu     Solaris Zones (Usr)
                   (i386) 11.10.0,REV=2005.01.21.16.34

Select package(s) you wish to process (or 'all' to process
all packages). (default: all) [?,??,q]: 
Transferring  package instance

Once the packages were extracted to /tmp/u, I started to poke around to see which files were included in the package:

$ cd /tmp/u/*

$ find .

.
./pkginfo
./pkgmap
./install
./install/checkinstall
./install/postinstall
./reloc
./reloc/usr
./reloc/usr/lib
./reloc/usr/lib/amd64
./reloc/usr/lib/amd64/libzonecfg.so.1
./reloc/usr/lib/libzonecfg.so.1
./reloc/usr/share
./reloc/usr/share/lib
./reloc/usr/share/lib/xml
./reloc/usr/share/lib/xml/dtd
./reloc/usr/share/lib/xml/dtd/zonecfg.dtd.1

Since the new version of libzonecfg.so was most likely the cause of my problems, I backed up the shared libraries the patch had installed in /usr/lib and /usr/lib/amd, and then replaced these with the versions I had extracted to /tmp/u:

$ cd /usr/lib

$ pwd
/usr/lib

$ cp libzonecfg.so.1 libzonecfg.so.1.orig

$ cp /tmp/u/SUNWzoneu/reloc/usr/lib/libzonecfg.so.1 .

$ cd /usr/lib/amd

$ pwd
/usr/lib/amd

$ cp libzonecfg.so.1 libzonecfg.so.1.orig

$ cp /tmp/u/SUNWzoneu/reloc/usr/lib/amd/libzonecfg.so.1 .

Once the old version of libzonecfg was in place, I was able to boot my zones without issue:

$ zoneadm -z dns boot

This experience once again leads me to wonder if Sun actually tests patches prior to sending them out to the public (this is my second bad experience in as many months). Now to schedule another downtime to properly back out the patch. :(

Generating E-mail when Solaris security patches are available

I wrote about pca a few weeks back, and absolutely love the capabilities it brings to the table. To keep my servers up date with the latest security patches, I added the following cron job to extract security patches from the pca output, and E-mail them to my account:

0 0 * * * pca -l | awk '$5 ~ /S/ { print $0}'  | mailx -s "Security updates for $HOSTNAME" root

This works pretty well, and I now get E-mailed when security patches are available.

Managing Solaris patches with pca

I have written repeatedly about the problems with the Solaris patch tools, and decided to test out the pca utility after Chris and Frank recommended it so highly. Pca is not only an awesome patching tool, but it blows away everything that is currently offered by Sun (pca has yet to throw Java exceptions and die in mysterious ways). To see what I mean, all you need to do is run pca with the help option:

$ pca -h

Usage: bin/pca [OPTION] .. [OPERAND] ..

Operands:
  patch group:    missing, installed, all, unbundled, bad
                  Add r, s or rs at the end to list Recommended,
                  Security or Recommended/Security patches only.
  patch ID:       123456, 123456-78
  patch file:     123456-78.zip, 123456-78.tar.Z
  file name:      patchlist.txt
  pattern:        /dtmail/

Options:
  -l              List patches
  -L              List patches, produce HTML output
  -d              Download patches
  -i              Install patches
  -I              Pretend to install patches
  -x              Download patch cross-reference file
  -y              Do not check for updated patch cross-reference file
  -X dir        Set location of patches cross-reference file
  -P dir        Set patch download directory
  -R dir        Set alternative root directory
  -n              Install only patches which do not require a reboot
  -k              Make patchadd not back up files to be patched
  -G              Make patchadd modify packages in the current zone only
  -a              Ask for SunSolve authentication data interactively
  -H              Don't display descriptive headers
  -r id         Display patch README
  -f dir        Read uname/showrev/pkginfo output from files in dir
  -h              Display this help
  -V              Display debug output
  -v              Display version information

Pca has several modes of operation. It can list patches that are outdated on your system, retrieve patches from Sunsolve, and most importantly it can be used to install individual patches and groups of patches on a server. To list patches that are available for a server, pca can be run with the “-l” option (or “-L” if you want HTML reports):

$ pca -l

Retrieving xref-file to /var/tmp/patchdiag.xref ... done
Using /var/tmp/patchdiag.xref from Jul/14/06
Host: neutron (SunOS 5.10/i386/i86pc)

Patch  IR   CR RSB Age Synopsis
------ -- - -- --- --- -------------------------------------------------------
117464 01 < 02 ---   2 SunOS 5.10_x86: passwdutil Patch
119131 21 < 22 R--   4 SunOS 5.10_x86: Sun Fibre Channel Device Drivers
119214 07 < 09 RSB   2 NSS_NSPR_JSS 3.11.2_x86: NSPR 4.6.2 / NSS 3.11.2 / JSS 4.2.4
119471 05 < 06 ---   5 SunOS 5.10_x86: Sun Enterprise Network Array firmware and utilitie
119686 06 < 07 ---   2 SunOS 5.10_x86: lib/svc/bin/svc.startd Patch
120037 03 < 05 ---   2 SunOS 5.10_x86: libldap patch
120053 02 < 03 ---   2 SunOS 5.10_x86: pam library patch
120200 04 < 05 ---   3 SunOS 5.10_x86: sysidtool Patch
121003 02 < 03 ---   2 SunOS 5.10_x86: pax patch
121005 01 < 02 RS-   2 SunOS 5.10_x86: sh patch
121011 01 < 02 ---   2 SunOS 5.10_x86: rpc.metad patch
123327 -- < 01 ---   5 SunOS 5.10_x86: tail patch
123521 -- < 01 ---   2 SunOS 5.10_x86: dirname & basename patch
123525 -- < 01 ---   2 SunOS 5.10_x86: psrinfo patch

The first column lists the patchid, the second column lists the version of the patch installed, the fourth column lists the updated version of the patch that is available on Sunsolve, and the fifth column indicates if the patch addresses a security or reliability problem. To install patches with pca, you can run pca with the "-i" option to install all available patches, or you can install individual patches by passing the patchid(s) to "-i":

$ pca -i 121005 119214

Retrieving xref-file to /var/tmp/patchdiag.xref ... done
Using /var/tmp/patchdiag.xref from Jul/14/06

Downloading patches to /home/matty
------------------------------------------------------------------------------
Retrieving 119214-09 (1/2) ... done
Retrieving 121005-02 (2/2) ... done

Summary: 2 total, 2 successful, 0 failed

Installing patches
------------------------------------------------------------------------------
Installing 119214-09 (1/2) ... done
Installing 121005-02 (2/2) ... done

Summary: 2 total, 2 successful, 0 skipped, 0 failed

This is a super useful piece of software, and I wish Sun would include something similar in Solaris (smpatch is not the answer!).

Woo hoo! smpatch lives!

After several frustrating days of debugging smpatch, I finally found a way to get it working on my X64 Solaris 10 server. I am not 100% certain why this fixed my problem, and since I don’t have access to the smpatch source code I will probably never know. To recap the problem, I was able to register my server with the Solaris sconadm utility, but was receiving the following error each time I ran smpatch to check for new updates:

$ smpatch analyze

Failure: Cannot connect to retrieve Database/current.zip:
This system is currently unregistered and is unable to retrieve patches from the Sun Update Connection.
Please register your system using the Update Manager.

To get smaptch to run successfully, I first had to remove my server’s assetid from the patch repository. This was accomplished with the ccr utilities “-r” option:

$ /usr/lib/cc-ccr/bin/ccr -r cns.assetid

Once the assetid was remove, I ran the sconadm utility to register my server:

$ sconadm register -a -r RegistrationProfile
sconadm is running
Authenticating user …
finish registration!

After these two tasks completed, smpatch ran successfully and reported a number of patches that needed to be applied. This experience really taught me how much I like yum, and I have since decided to replace smpatch with the pca (patch check advanced) utility after reading Chris’s blog and Frank’s comments. Pca is an AWESOME piece of software, and I wish I would have found it sooner. More to come on pca in future posts.

More smpatch madness

I have mentioned several problems with Sun’s smpatch utility in the past few blog entries, and it looks like I hit another nasty bug. The sconadm utility allows me to register a server:

$ sconadm register -a -r RegistrationProfile

sconadm is running
Authenticating user ...
finish registration!

But when I run the smpatch utility to check for updates, it says the system isn’t registered:

$ smpatch analyze

Failure: Cannot connect to retrieve Database/current.zip: 
This system is currently unregistered and is unable to retrieve patches from the Sun Update Connection. 
Please register your system using the Update Manager.

This is somewhat amusing, though it has caused me a good deal of consternation. I am extremely disappointed with the quality of Sun’s patch tools, and am posting this here in case other folks are encountering similar problems. Hopefully they will see that they are doing nothing wrong, and the tools they have been provided are broken.