Resizing Veritas volumes with vxresize

We were getting close to running out of space on one of our database volumes last week, and I needed to add some additional storage to ensure that things kept running smoothly. The admin who originally created the VxVM database volume only used half of each of the five disks that were associated with the volume / file system that were at capacity, which meant I had roughly 18GB of free space available on each device to work with:

$ vxdg free | egrep ‘(D01|D02|D03|D04|D05)’

GROUP        DISK         DEVICE       TAG          OFFSET    LENGTH    FLAGS
datadg       D01          c2t0d0s2     c2t0d0       35547981   35547981  -
datadg       D02          c2t1d0s2     c2t1d0       35547981   35547981  -
datadg       D03          c2t2d0s2     c2t2d0       35547981   35547981  -
datadg       D04          c2t3d0s2     c2t3d0       35547981   35547981  -
datadg       D05          c2t4d0s2     c2t4d0       35547981   35547981  -
datadg       D06          c2t5d0s2     c2t5d0       35547981   35547981  -

Now there are a number of ways to resize volumes and file systems with VxVM and VxFS. You can use vxassist to grow or shrink a volume, and then use the fsadm utility to extend the file system. You can also perform both of these operations with vxresize, which takes the name of the volume to resize, the disk group the volume is a part of, and a size parameter to indicate that you want to grow (you use a “+” to indicate that you want to grow the volume by the value immediately following the plus sign) or shrink (you use a “-” to indicate that you want to shrink the volume by the value immediately following the dash) the volume. Since I preferred the most efficient method to resize my volume, I fired up vxresize and told it to extend the volume named datavol01 by 35547981 blocks:

$ /etc/vx/bin/vxresize -g datadg -F vxfs datavol01 +35547981

Instead of specifying blocks, you can also use unit specifiers such as “m” to denote megabytes, and “g” to denote gigabytes. As with all operations that change the structure of storage, you should test any resizing operations on non-production systems prior to changing production systems.

Removing duplicate devices from vxdisk list

I replaced a disk in one of our A5200s last week, and noticed that vxdisk was displaying two entries for the device once I replaced it with vxdiskadm:

$ vxdisk list

DEVICE       TYPE      DISK         GROUP        STATUS
c7t21d0s2    sliced    disk01       oradg        online
c7t22d0s2    sliced    disk02       oradg        error
c7t22d0s2    sliced    -            -            error
c7t23d0s2    sliced    disk03       oradg        online

To fix this annoyance, I first removed the disk disk02 from the oradg disk group:

$ vxdg -g oradg rmdisk disk02

Once the disk was removed, I ran vxdisk “remove” two times to remove both disk access records:

$ vxdisk rm c7t22d0s2

$ vxdisk rm c7t22d0s2

After both device access records were removed, I executed ‘devfsadm -C’ to clean the Solaris device tree, and then ran ‘vxdctl enable’ to have Veritas update the list of devices it knows about. After these oeprations completed, the device showed up once in the vxdisk output:

$ vxdisk list

DEVICE       TYPE      DISK         GROUP        STATUS
c7t21d0s2    sliced    disk01       oradg        online
c7t22d0s2    sliced    disk02       oradg        online
c7t23d0s2    sliced    disk03       oradg        online

I have seen times where the Solaris device tree will hold on to old entries, which unfortunately requires a reboot to fix. Luckily for me, this wasn’t the case with my system. Shibby!

Recreating Veritas configurations with vxmake

Last year I wrote an article titled Veritas Volume Manager Recovery Features for SysAdmin magazine. In the article I described how to backup a Veritas configuration to a file with vxconfigbackup, and how to restore it with vxconfigrestore. One thing I didn’t touch on was selectively restoring individual volume configurations. This is easy to do, and I wrote the vxvmconfigbackup shell script to simplify capturing the data needed to restore a single Veritas Volume Manager volume. To illustrate just how easy this is, I added three disk drives to a disk group, and then created three volumes (datavol01, datavol02, datavol03). Here is the layout:

$ vxprint -hft

Disk group: datadg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         NVOLUME      KSTATE   STATE
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO

dg datadg       default      default  0        1158701822.18.snappy

dm hdb          hdb          auto     2074     8384848  -
dm hdc          hdc          auto     2074     8384848  -
dm hdd          hdd          auto     2074     8384848  -

v  datavol01    -            ENABLED  ACTIVE   2097152  SELECT    -        fsgen
pl datavol01-01 datavol01    ENABLED  ACTIVE   2097152  CONCAT    -        RW
sd hdb-01       datavol01-01 hdb      0        2097152  0         hdb      ENA
pl datavol01-02 datavol01    ENABLED  ACTIVE   2097152  CONCAT    -        RW
sd hdc-01       datavol01-02 hdc      0        2097152  0         hdc      ENA
pl datavol01-03 datavol01    ENABLED  ACTIVE   2097152  CONCAT    -        RW
sd hdd-01       datavol01-03 hdd      0        2097152  0         hdd      ENA

v  datavol02    -            ENABLED  ACTIVE   2097152  RAID      -        raid5
pl datavol02-01 datavol02    ENABLED  ACTIVE   2097152  RAID      3/32     RW
sd hdb-02       datavol02-01 hdb      2097152  1048576  0/0       hdb      ENA
sd hdc-02       datavol02-01 hdc      2097152  1048576  1/0       hdc      ENA
sd hdd-02       datavol02-01 hdd      2097152  1048576  2/0       hdd      ENA

v  datavol03    -            ENABLED  ACTIVE   2097152  SELECT    -        fsgen
pl datavol03-01 datavol03    ENABLED  ACTIVE   2097152  CONCAT    -        RW
sd hdb-03       datavol03-01 hdb      3145728  2097152  0         hdb      ENA

The vxvmconfigbackup shell script can be used to backup the configuration data for all disk groups, a single disk group, or a volume inside a disk group. To backup the configuration needed to recreate the volume datavol01, the volume and disk group names can be passed to the “-v” and “-g” options:

$ vxvmconfigbackup -g datadg -v datavol01 -d /tmp
Backing up volume datavol01 to /tmp/datadg.datavol01.09-19-2006-2658

Now that the configuration of datavol01 is backed up, let’s create a hypothetical problem. Let’s assume that it is 4am, you have been up debugging a problem for 48-hours straight, and you accidentally remove the volume datavol01 instead of datavol03 (I helped an admin recover from a similar situation a few years back, so I know it happens):

$ vxedit -rf rm datavol01

$ vxprint -hft

Disk group: datadg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         NVOLUME      KSTATE   STATE
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO

dg datadg       default      default  0        1158701822.18.snappy

dm hdb          hdb          auto     2074     8384848  -
dm hdc          hdc          auto     2074     8384848  -
dm hdd          hdd          auto     2074     8384848  -

v  datavol02    -            ENABLED  ACTIVE   2097152  RAID      -        raid5
pl datavol02-01 datavol02    ENABLED  ACTIVE   2097152  RAID      3/32     RW
sd hdb-02       datavol02-01 hdb      2097152  1048576  0/0       hdb      ENA
sd hdc-02       datavol02-01 hdc      2097152  1048576  1/0       hdc      ENA
sd hdd-02       datavol02-01 hdd      2097152  1048576  2/0       hdd      ENA

v  datavol03    -            ENABLED  ACTIVE   2097152  SELECT    -        fsgen
pl datavol03-01 datavol03    ENABLED  ACTIVE   2097152  CONCAT    -        RW
sd hdb-03       datavol03-01 hdb      3145728  2097152  0         hdb      ENA

If you weren’t prepared for this problem, you might respond by screaming a bunch of profanities, and yacking in the nearest garbage can. If you are backing up your configuration, you will most likely take a minute to laugh at yourself, and then you will run vxmake with the “-d” option and the configuration to restore:

$ vxmake -d /tmp/datadg.datavol01.09-19-2006-2658

Once the configuration is restored, the volume will be available for general purpose use:

$ vxprint -hft

Disk group: datadg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         NVOLUME      KSTATE   STATE
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO

dg datadg       default      default  0        1158701822.18.snappy

dm hdb          hdb          auto     2074     8384848  -
dm hdc          hdc          auto     2074     8384848  -
dm hdd          hdd          auto     2074     8384848  -

v  datavol01    -            ENABLED  ACTIVE   2097152  SELECT    -        fsgen
pl datavol01-01 datavol01    ENABLED  ACTIVE   2097152  CONCAT    -        RW
sd hdb-01       datavol01-01 hdb      0        2097152  0         hdb      ENA
pl datavol01-02 datavol01    ENABLED  ACTIVE   2097152  CONCAT    -        RW
sd hdc-01       datavol01-02 hdc      0        2097152  0         hdc      ENA
pl datavol01-03 datavol01    ENABLED  ACTIVE   2097152  CONCAT    -        RW
sd hdd-01       datavol01-03 hdd      0        2097152  0         hdd      ENA

v  datavol02    -            ENABLED  ACTIVE   2097152  RAID      -        raid5
pl datavol02-01 datavol02    ENABLED  ACTIVE   2097152  RAID      3/32     RW
sd hdb-02       datavol02-01 hdb      2097152  1048576  0/0       hdb      ENA
sd hdc-02       datavol02-01 hdc      2097152  1048576  1/0       hdc      ENA
sd hdd-02       datavol02-01 hdd      2097152  1048576  2/0       hdd      ENA

v  datavol03    -            ENABLED  ACTIVE   2097152  SELECT    -        fsgen
pl datavol03-01 datavol03    ENABLED  ACTIVE   2097152  CONCAT    -        RW
sd hdb-03       datavol03-01 hdb      3145728  2097152  0         hdb      ENA

$ mount -t vxfs /dev/vx/dsk/datadg/datavol01 /a

$ cd /a

$ ls -l

total 1536
-rw-r--r--  1 root root 524288 Sep 19 18:32 data01.dbf
-rw-r--r--  1 root root 524288 Sep 19 18:32 data02.dbf
-rw-r--r--  1 root root 524288 Sep 19 18:32 data03.dbf
drwxr-xr-x  2 root root     96 Sep 19 18:32 lost+found

In addition to be being able to restore the configuration with vxmake’s “-d” option, you can also piece each object back together manually. This is a tedious and error prone process (especially at 4am), so I tend to create configuration backups on all my servers just to be safe. As with all recovery scenarios, you should validate recovery procedures in a test environment prior to using them on a production system.

Tracing vxassist activity

While creating a few Veritas volumes last week, I wanted to see the commands that vxassist was executing under the covers. This was easily accomplished by adding the “-v” (trace commands executed by vxassist) option to the vxassist command line:

$ vxassist -b -v make datavol02 1g layout=mirror mirror=3
/usr/sbin/vxvol -g datadg -o bg -o plexfork=128 — start datavol02

VxVM is an awesome volume manager, and there are all kinds of cool things buried in the manual pages!

Using Veritas Volume Manager with RHAS 4.0

I have used Veritas Volume Manager (VxVM) and Veritas File system (VxFS) for as long as I can remember. All of my VxVM and VxFS experience has been on Solaris, so I thought I would install both products on a Linux host to see if anything was different. The Linux installation turned out to be nearly identical to the Solaris installation (the Linux installer install RPMs versus SVR4 packages), and the vx* commands are located in the same place in both operating systems. Since I wanted to get my hands dirty and play with VxVM and VxFS on a Linux host, I first ran the vxdisksetup utility to initialize a few devices:

$ /usr/lib/vxvm/bin/vxdisksetup -i hdc
VxVM vxdisksetup ERROR V-5-2-1814 hdc: Invalid disk device for ‘cdsdisk’ format

Gak! After pondering the error for a few minutes, it dawned on me that as of VxVM 4.0 the “cdsdisk” disk format is used by default. The new format allows devices to be transported between different operating systems and hardware architecures (e.g., you can deport a Solaris disk group on a SPARC host (big endian) and import and access it* on an x86 (little endian) Linux host). After sifting through the Veritas support site to see if cdsdisk had any limitations, I came across infodoc 278178. Infodoc 278178 states that the cdsdisk format can only be used on SCSI disks, and showed how to use the vxscsi command to see if the cdsdisk format could be used with a device:

$ /usr/lib/vxvm/diag.d/vxscsi -g sdd
Cannot get disk geometry on /dev/vx/rdmp/hdd !

The IDE disk drives I was using with the Linux host don’t fit into the cdsdisk supportability matrix, so I decided to the use the “sliced” format since the devices were used purely for testing:

$ /usr/lib/vxvm/bin/vxdisksetup -fi hdc format=sliced

$ /usr/lib/vxvm/bin/vxdisksetup -fi hdd format=sliced

Once the disks were initialized, I added them to a disk group, carved up a new volume, and created a VxFS file system on the volume:

$ vxdg init datadg hdc hdd cds=off

$ vxassist -g datadg make vol01 512m layout=concat

$ mkfs -t vxfs -o bsize=8192 /dev/vx/dsk/datadg/vol01

    version 6 layout
    8380416 sectors, 523776 blocks of size 8192, log size 2048 blocks
    largefiles supported

I plan to fire up my Sun multipack this weekend to see if the Solaris to Linux migration works as well as the folks at Veritas say (based on past experiences with the Veritas product suite, I am relatively certain it will work well).

* To deal with endianness issues, you need to use the fscdsconv utility.

Creating a Veritas Volume Manager volume from a plex

While cleaning up my Veritas VxVM notes this week, I came across one of my cheat sheets for creating volumes from plexes. This can be useful when you want to perform an offline backup of the data on an alternate server, or when you want to grab a spare copy of data prior to performing a major change. To create a new volume from a plex, you will first want to stash a copy of the vxprint output somewhere:

$ /usr/sbin/vxprint -hmQq > $HOME/vxprint.restore

Once you make a backup, you will want to run the human consumable form of vxprint to see which devices are available:

$ /usr/sbin/vxprint -hft

Disk group: oradg

DG NAME         NCONFIG      NLOG     MINORS   GROUP-I
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         NVOLUME      KSTATE   STATE
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO

dg oradg        default      default  10000    1127240283.19.winnie

dm c1t1d0       c1t1d0s2     auto     2048     35521408 -
dm c1t2d0       c1t2d0s2     auto     2048     35521408 -
dm c1t3d0       c1t3d0s2     auto     2048     35521408 -
dm c1t4d0       c1t4d0s2     auto     2048     35365968 -
dm c1t5d0       c1t5d0s2     auto     2048     35521408 -
dm c1t6d0       c1t6d0s2     auto     2048     35521408 -

v  oravol01     -            ENABLED  ACTIVE   20971520 SELECT    -        fsgen
pl oravol01-01  oravol01     ENABLED  ACTIVE   20971776 STRIPE    3/128    RW
sd c1t1d0-01    oravol01-01  c1t1d0   0        6990592  0/0       c1t1d0   ENA
sd c1t2d0-01    oravol01-01  c1t2d0   0        6990592  1/0       c1t2d0   ENA
sd c1t3d0-01    oravol01-01  c1t3d0   0        6990592  2/0       c1t3d0   ENA
pl oravol01-02  oravol01     ENABLED  ACTIVE   20971776 STRIPE    3/128    RW
sd c1t4d0-01    oravol01-02  c1t4d0   0        6990592  0/0       c1t4d0   ENA
sd c1t5d0-01    oravol01-02  c1t5d0   0        6990592  1/0       c1t5d0   ENA
sd c1t6d0-01    oravol01-02  c1t6d0   0        6990592  2/0       c1t6d0   ENA

After you locate the plex you want to turn intoa volume, you will need to disassociate it from the volume with the vxplex utility:

$ vxplex dis oravol01-02

Once the plex has been disassociated, you can then turn it into a volume with the vxmake utility:

$ vxmake -U gen vol oravol02 plex=oravol01-02

After the volume is created, you can start it and mount it just like all of the other volumes on your server:

$ vxvol start oravol02

$ mount -F vxfs /dev/vx/dsk/oradg/oravol02 /mnt

$ ls -la /mnt

total 8388690
drwxr-xr-x   3 root     root        8192 Oct 30 21:01 .
drwxr-xr-x  37 root     root        1024 Oct 27 12:27 ..
drwxr-xr-x   2 root     root          96 Oct 27 12:25 lost+found
-rw------T   1 root     root     1073741824 Oct 30 20:59 oradata01.dbf
-rw------T   1 root     root     1073741824 Oct 30 21:00 oradata02.dbf
-rw------T   1 root     root     1073741824 Oct 30 21:01 oradata03.dbf
-rw------T   1 root     root     1073741824 Oct 30 21:03 oradata04.dbf

This is a nifty feature, and now that Veritas Volume Manager and File System are free (thanks for the link CW!), you can test it out in your favorite lab!