New zone features in Solaris 10 update 9

I just finished reading through Frank Batschulat’s slide deck of new zone features in Solaris 10 update 9. The new update on attach enhancements look extremely useful, and I have a few systems I will have to try to p2v using the tools that now ship with Solaris. If you are using zones, you should check out his presentation. Good stuff!

Adding a file system to a running zone

I needed to add a second file system to one of my Solaris 10 zones this morning, and needed to do so without rebooting the zone. Since the global zone uses loopback mounts to present file systems to zones, adding a new file system was as easy as loopback mounting the file system into the zone’s file system:

$ mount -F lofs /filesystems/zone1oracle03 /zones/zone1/root/ora03

Once the file system was mounted, I added it to the zone configuration and then verified it was mounted:

$ mount | grep ora03

/filesystems/zone1oracle03 on filesystems/zone1oracle0 read/write/setuid/devices/nonbmand/exec/xattr/atime/dev=2d9000b on Sun Apr 12 10:43:19 2009
/zones/zone1/root/ora03 on /filesystems/zone1oracle03 read/write/setuid/devices/dev=2d9000b on Sun Apr 12 10:44:07 2009



Now to update my ASM disk group to use the storage.

Patching zones when they are attached to hosts

I recently patched one of my Solaris 10 hosts, and decided to test out the zone update on attach functionality that is now part of Solaris 10 update 6. The update on attach feature allows detached zones to get patched when they are attached to a host, which can be rather handy if you are moving zones around your infrastructure. To test this functionality, I first detached a zone from the host I was going to patch:

$ zoneadm -z zone detach

$ zoneadm list -vc

  ID NAME             STATUS     PATH                           BRAND    IP    
   0 global           running    /                              native   shared
   - zone1            configured /zones/zone1                   native   shared



Once the zone was detached, I applied the latest Solaris patch bundle and rebooted the server. When the system came back up, I tried to attach the zone:

$ zoneadm -z zone1 attach

These patches installed on the source system are inconsistent with this system:
	118668: version mismatch
		(17) (19)
	118669: version mismatch
		(17) (19)
	119060: version mismatch
		(44) (45)
	119091: version mismatch
		(31) (32)
	119214: version mismatch
		(17) (18)
	119247: version mismatch
		(34) (35)
	119253: version mismatch
		(29) (31)
	119255: version mismatch
		(59) (65)
	119314: version mismatch
		(24) (26)
	119758: version mismatch
		(12) (14)
	119784: version mismatch
		(07) (10)
	120095: version mismatch
		(21) (22)
	120200: version mismatch
		(14) (15)
	120223: version mismatch
		(29) (31)
	120273: version mismatch
		(23) (25)
	120411: version mismatch
		(29) (30)
	120544: version mismatch
		(11) (14)
	120740: version mismatch
		(04) (05)
	121119: version mismatch
		(13) (15)
	121309: version mismatch
		(14) (16)
	121395: version mismatch
		(01) (03)
	122213: version mismatch
		(28) (32)
	122912: version mismatch
		(13) (15)
	123896: version mismatch
		(05) (10)
	124394: version mismatch
		(08) (09)
	124629: version mismatch
		(09) (10)
	124631: version mismatch
		(19) (24)
	125165: version mismatch
		(12) (13)
	125185: version mismatch
		(08) (11)
	125333: version mismatch
		(03) (05)
	125540: version mismatch
		(04) (06)
	125720: version mismatch
		(24) (28)
	125732: version mismatch
		(02) (04)
	125953: version mismatch
		(17) (18)
	126364: version mismatch
		(06) (07)
	126366: version mismatch
		(12) (14)
	126420: version mismatch
		(01) (02)
	126539: version mismatch
		(01) (02)
	126869: version mismatch
		(02) (03)
	136883: version mismatch
		(01) (02)
	137122: version mismatch
		(03) (06)
	137128: version mismatch
		(02) (05)
	138224: version mismatch
		(02) (03)
	138242: version mismatch
		(01) (05)
	138254: version mismatch
		(01) (02)
	138264: version mismatch
		(02) (03)
	138286: version mismatch
		(01) (02)
	138372: version mismatch
		(02) (06)
	138628: version mismatch
		(02) (07)
	138857: version mismatch
		(01) (02)
	138867: version mismatch
		(01) (02)
	138882: version mismatch
		(01) (02)
These patches installed on this system were not installed on the source system:
	125556-02
	138889-08
	139100-01
	139463-02
	139482-01
	139484-05
	139499-04
	139501-02
	139561-02
	139580-02
	140145-01
	140384-01
	140456-01
	140775-03
	141009-01
	141015-01



As you can see in the above output, the zone refused to attach because the zone patch database differed from the global zone patch database. To synchronize the two, I added the “-u” option (update the zone when it is attached to a host) to the zoneadm command line:

$ zoneadm -z zone1 attach -u

Getting the list of files to remove
Removing 1209 files
Remove 197 of 197 packages
Installing 1315 files
Add 197 of 197 packages
Updating editable files

The file  within the zone contains a log of the zone update.



Once the zone was updated, I was able to boot the zone without issue:

$ zoneadm -z zone1 boot

$ zoneadm list -vc

  ID NAME             STATUS     PATH                           BRAND    IP    
   0 global           running    /                              native   shared
   4 zone1            running    /zones/zone1                   native   shared



This is pretty sweet, and I can see myself using this functionality (along with live upgrade) in the future!

Patching zones in parallel

While catching up on some of my favorite blogs, I came across a post pertaining to parallel zone patching by Jeff Victor. In his write up, Jeff discussed a couple of methods for decreasing the time it takes to patch a host with one more more zones. Of the methods listed, the “Zones Parallel patch” feature caught my eye:

“Sun has developed an enhancement to the Solaris patching tools which is intended to significantly decrease the elapsed time of patching. It is currently being tested at a small number of sites. After it’s released you can get the Zones Parallel Patching patch, described below. This solution decreases the elapsed time to patch a system. It can be combined with some of the solutions above, with varying benefits. For example, with Live Upgrade, parallel patching reduces the time to patch the ABE, but doesn’t reduce service downtime. Also, ZPP offers little benefit for the detach/attach-on-upgrade method. However, as a stand-alone method, ZPP offers significant reduction in elapsed time without changing your current patching process.”

It can take a looooooong time to patch hosts with multiple zones, so I can’t wait to get my hands on this!

Changing the zone path of a pre-existing zone

So, the hostname of one of our zones changed, and we create ZFS file systems for the zones using <path>/<zonename>

Easy enough of a fix…

root@db@blah-global:~#zoneadm list -cv
ID NAME             STATUS     PATH                           BRAND    IP
0 global           running    /                              native   shared
1 blah         running    /local2/data/zones/blah    native   shared
5 blah1          running    /local/data/zones/blah1      native   shared

root@db@blah-global:~#zonecfg -z blah1
zonecfg:blah1> set zonename=blah2
blah1: Zone state is invalid for the requested operation
So lets shut down the zone and change the zone name.

root@db@blah-global:~#zoneadm -z blah1 halt
root@db@blah-global:~#zonecfg -z blah1 info
zonename: blah1
zonepath: /local/data/zones/blah1

root@db@blah-global:~#zonecfg -z blah1
zonecfg:blah1> set zonename=blah2
zonecfg:blah2> verify
zonecfg:blah2> commit
zonecfg:blah2> exit

So now that we’ve changed the zone name, I also wanted to update the ZFS file system to reflect the new hostname…

root@db@blah-global:~#zfs rename pool0/local/data/zones/blah1 pool0/local/data/zones/blah2

Easy enough.  Lets boot the zone.

root@db@blah-global:~#zoneadm list -cv
ID NAME             STATUS     PATH                           BRAND    IP
0 global           running    /                              native   shared
1 blah         running    /local2/data/zones/blah    native   shared
– blah2    installed  /local/data/zones/blah1      native   shared

root@db@blah-global:~#zoneadm -z blah2 boot
zoneadm: /local/data/zones/blah1: No such file or directory
could not verify zonepath /local/data/zones/blah1 because of the above errors.
zoneadm: zone blah1 failed to verify

DOH.  Forgot to change the zone path.. So lets go fix this…

root@db@blah-global:~#zonecfg -z blah2
zonecfg:blah2> set zonepath=/local/data/zones/blah2
Zone blah2 already installed; set zonepath not allowed.

WHAT?!?!  Come on now zonecfg, this shouln’t be brain surgery….

So, lets go poke at the source of truth — /etc/zones.

root@db@blah-global:/etc/zones#grep blah1
index:blah2:installed:/local/data/zones/blah1:6b9891a3-7029-ef67-9581-aa01475c9b6e
blah2.xml:<zone name=”blah2″ zonepath=”/local/data/zones/blah1″ autoboot=”true”>

So, go and edit blah2.xml to change the zone path, as well as index.  (Make backup copies of these before doing so please)

Once the modifications were done, I was able to boot up the zone with the newly changed zonepath.

root@db@blah-global:/etc/zones#zoneadm list -cv
ID NAME             STATUS     PATH                           BRAND    IP
0 global           running    /                              native   shared
1 blah         running    /local2/data/zones/blah    native   shared
– blah2    installed  /local/data/zones/blah2 native   shared

root@db@blah-global:/etc/zones#zoneadm -z blah2 boot
root@db@blah-global:/etc/zones#zoneadm list -cv
ID NAME             STATUS     PATH                           BRAND    IP
0 global           running    /                              native   shared
1 blah         running    /local2/data/zones/blah    native   shared
6 blah2    running    /local/data/zones/blah2 native   shared
root@db@blah-global:/etc/zones#zlogin blah2
[Connected to zone ‘blah2’ pts/1]
Last login: Fri Mar 27 17:14:03 from blah-global
Sun Microsystems Inc.    SunOS 5.10    Generic    January 2005
#

Sun, could we please extend zonecfg to do this for us?  The header of /etc/zones/index is scary.

root@db@blah-global:~#cat /etc/zones/index
# Copyright 2004 Sun Microsystems, Inc.  All rights reserved.
# Use is subject to license terms.
#
# ident “@(#)zones-index        1.2     04/04/01 SMI”
#
# DO NOT EDIT: this file is automatically generated by zoneadm(1M)
# and zonecfg(1M).  Any manual changes will be lost.
#