Expanding Solaris metadevices

I recently had a file system on a Solaris Volume Manager (SVM) metadevice fill up, and I needed to expand it to make room for some additional data. Since the expansion could potentially cause problems, I backed up the file system, and saved a copy of the metastat and df output to my local workstation. Having several backups always gives me a warm fuzzy, since I know I have a way to revert back to the old configuration if something goes awry. Once the configuration was in a safe place and the data backed up, I used the umount command to unmount the /data file system, which lives on metadevice d100:

$ df -h

Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c1t0d0s0      7.9G   2.1G   5.7G    27%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                   2.3G   600K   2.3G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
/usr/lib/libc/libc_hwcap1.so.1
                       7.9G   2.1G   5.7G    27%    /lib/libc.so.1
fd                       0K     0K     0K     0%    /dev/fd
/dev/dsk/c1t0d0s4      4.0G   154M   3.8G     4%    /var
swap                   2.3G    32K   2.3G     1%    /tmp
swap                   2.3G    24K   2.3G     1%    /var/run
/dev/dsk/c1t0d0s3       19G   2.8G    17G    15%    /opt
/dev/md/dsk/d100        35G    35G   120M    99%    /data

$ umount /data

After the file system was unmounted, I had to run the metaclear utility to remove the metadevice from the meta state database:

$ metaclear D100
d100: Concat/Stripe is cleared

Now that the metadevice was removed, I needed to add it back with the desired layout. It is EXTREMELY important to place the device(s) back in the right order, and to ensure that the new layout doesn’t corrupt the data that exists on the device(s) that contain the file system (i.e., don’t create a RAID5 metadevice with the existing devices, since that will wipe your data when the RAID5 metadevice is initialized). In my case, I wanted to concatenate another hardware RAID protected LUN to the meta device d100. This was accomplished by running metainit with the “numstripes” equal to 2 to indicate a 2 stripe concatenation, and “width” equal to 1 to indicate that each stripe should have one member:

$ metainit d100 2 1 c1t1d0s0 1 c1t2d0s0
d100: Concat/Stripe is setup

Once the new metadevice was created, I ran the mount utility to remount the /data file system, and then executed growfs to expand the file system:

$ mount /dev/md/dsk/d100 /data

$ growfs -M /data /dev/md/rdsk/d100

Warning: 2778 sector(s) in last cylinder unallocated
/dev/md/rdsk/d100:      150721830 sectors in 24532 cylinders of 48 tracks, 128 sectors
        73594.6MB in 1534 cyl groups (16 c/g, 48.00MB/g, 5824 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 98464, 196896, 295328, 393760, 492192, 590624, 689056, 787488, 885920,
Initializing cylinder groups:
..............................
super-block backups for last 10 cylinder groups at:
 149821984, 149920416, 150018848, 150117280, 150215712, 150314144, 150412576,
 150511008, 150609440, 150707872

After the growfs operation completed, I had some breathing room on the /data file system:

$ df -h

Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c1t0d0s0      7.9G   2.1G   5.7G    27%    /
/devices                 0K     0K     0K     0%    /devices
ctfs                     0K     0K     0K     0%    /system/contract
proc                     0K     0K     0K     0%    /proc
mnttab                   0K     0K     0K     0%    /etc/mnttab
swap                   2.3G   600K   2.3G     1%    /etc/svc/volatile
objfs                    0K     0K     0K     0%    /system/object
/usr/lib/libc/libc_hwcap1.so.1
                       7.9G   2.1G   5.7G    27%    /lib/libc.so.1
fd                       0K     0K     0K     0%    /dev/fd
/dev/dsk/c1t0d0s4      4.0G   154M   3.8G     4%    /var
swap                   2.3G    32K   2.3G     1%    /tmp
swap                   2.3G    24K   2.3G     1%    /var/run
/dev/dsk/c1t0d0s3       19G   2.8G    17G    15%    /opt
/dev/md/dsk/d100        71G    36G    35G    49%    /data

The fact that you have to unmount the file system to grow a metadevice is somewhat frustrating, since every other LVM package I have used allows volumes and file system to be expanded on the fly (it’s a good thing ZFS is shipping with Solaris). As with all data migrations, you should test storage expansion operations prior to performing them on production systems.

Solaris Volume Manager vanity names

I’ve been a Solaris Volume Manager (formerly disksuite) user for the past 6-years, and have always been annoyed that I had to use dXXX and hspXXX to label mirrors, stripes, sub-devices and hot spares. As of Nevada build 37, you are no longer required to use dXXX and hspXXX, and are free to choose a name that fits the device:

$ metainit foomirror1 1 1 c1t5d0s0
foomirror1: Concat/Stripe is setup

$ metainit foomirror2 1 1 c1t6d0s0
foomirror2: Concat/Stripe is setup

$ metainit foovol -m foomirror1
foovol: Mirror is setup

$ metattach foovol foomirror2
foovol: submirror foomirror2 is attached

$ metastat -c

foovol           m   16GB foomirror1 foomirror2 (resync-0%)
    foomirror1   s   16GB c1t5d0s0
    foomirror2   s   16GB c1t6d0s0

W00t! W00t!

Bizarre SVM Issue

I had a disk drive fail in one of my machines this week, and used the typical drive replacement procedure (cfgadm / metadevadm / devfsadm) to replace the physical disk. Once the drive was replaced, I attempted to run metareplace to re-synchronize the two sub-mirrors:

$ metareplace -e d0 c0t1d0s0
d0: device c0t1d0s0 is enabled

$ metastat d0

d0: Mirror
    Submirror 0: d10
      State: Okay
    Submirror 1: d20
      State: Unavailable
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 8391880 blocks (4.0 GB)

d10: Submirror of d0
    State: Okay
    Size: 8391880 blocks (4.0 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c0t0d0s0          0     No            Okay   Yes


d20: Submirror of d0
    State: Unavailable
    Size: 8391880 blocks (4.0 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c0t1d0s0          0     No               -   Yes

Eh? For some reason d20 refused to re-synchronize and enter the Okay state, and repeated attempts to use metareplace led to the same behavior. This seemed odd, so I decided to detach d20 and re-attach it with metadetach and metattach:

$ metadetach d0 d20
d0: submirror d20 is detached

$ metattach d0 d20
d0: submirror d20 is attached

These operations completed successfully, and once the re-synchronization completed, the sub-mirror entered the “Okay” state:

$ metastat d0

d0: Mirror
    Submirror 0: d10
      State: Okay
    Submirror 1: d20
      State: Okay
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 8391880 blocks (4.0 GB)

d10: Submirror of d0
    State: Okay
    Size: 8391880 blocks (4.0 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c0t0d0s0          0     No            Okay   Yes


d20: Submirror of d0
    State: Okay
    Size: 8391880 blocks (4.0 GB)
    Stripe 0:
        Device     Start Block  Dbase        State Reloc Hot Spare
        c0t1d0s0          0     No            Okay   Yes

I am starting to speculate that this is a bug in metareplace, but wasn’t able to pinpoint anything specific on sunsolve. The moral of the story is use metattach/metadetach if you don’t want to waste lots of time. :)

Updating SVM/Disksuite device relocation information

While booting my Solaris 10 box today, I noticed the following error in /var/adm/messages:

Jul 23 11:15:53 tigger metadevadm: [ID 209699 daemon.error] Invalid device relocation 
information detected in Solaris Volume Manager
Jul 23 11:15:53 tigger metadevadm: [ID 912841 daemon.error] Please check the status of 
the following disk(s):
Jul 23 11:15:53 tigger metadevadm: [ID 702911 daemon.error]     c1t6d0

I recently replaced c1t6d0, but forgot to update the device relocation information in the meta state database. To fix this issue, I ran metadevadm(1m) with the “-u” (update diskid in the meta state database) option:

$ metadevadm -u c1t6d0

Updating Solaris Volume Manager device relocation information for c1t6d0
Old device reloc information:
        id1,sd@SSEAGATE_SX318203LC______LR834657____1024048T
New device reloc information:
        id1,sd@SSEAGATE_SX318203LC______LR22875200001004H76G

Now metadevadm doesn’t complain when the box boots! :)

Manually synchronizing Solaris meta devices

While performing routine maintenance today, I discovered that one of my hot spare drives kicked in to replace a faulted disk drive. Since the synchronization process had recently started, I decided to shutdown the box to replace the faulted drive. Once I booted the box back up, I noticed that the synchronization process didn’t start automatically:

$ metastat d5

d5: RAID
    State: Resyncing    
    Hot spare pool: hsp001
    Interlace: 128 blocks
    Size: 106085968 blocks (50 GB)
Original device:
    Size: 106086528 blocks (50 GB)
        Device     Start Block  Dbase        State Reloc  Hot Spare
        c1t1d0s0       6002        No         Okay   Yes 
        c1t2d0s0       4926        No    Resyncing   Yes c1t6d0s0
        c1t3d0s0       4926        No         Okay   Yes 
        c1t4d0s0       4926        No         Okay   Yes 

Under normal operation, a “Resync in progress” line would be listed. To manually start the syncrhonization process, I ran the metasync(1m) command by hand:

$ metasync -r 2048

Once the command was executed, the synchronization process started:

$ metastat d5

d5: RAID
    State: Resyncing    
    Resync in progress:  0.5% done
    Hot spare pool: hsp001
    Interlace: 128 blocks
    Size: 106085968 blocks (50 GB)
Original device:
    Size: 106086528 blocks (50 GB)
        Device     Start Block  Dbase        State Reloc  Hot Spare
        c1t1d0s0       6002        No         Okay   Yes 
        c1t2d0s0       4926        No    Resyncing   Yes c1t6d0s0
        c1t3d0s0       4926        No         Okay   Yes 
        c1t4d0s0       4926        No         Okay   Yes 

Since this is a software RAID5 meta device, the synchronization process ( read data and parity, calculate parity, write data, write parity) will take a looooooong time to complete.