Veritas Volume Manager Recovery Features


Veritas Volume Manager (VxVM) has become the standard Logical Volume Manager (LVM) in many enterprises for its robust feature set, its ability to run on multiple operating systems (e.g., HP-UX, Linux, Solaris, Windows), and the numerous scalability, availability, and recoverability features that come with the base product. The recoverability features help to ensure that data is protected when hardware platforms fail and to ease the process required to restore systems to an operational state.

This article will provide an introduction to two important and often overlooked recovery features: failure notifications and configuration database backups. The article will also provide two disaster-recovery case studies to show how these recovery features can be used to aide in recovering from disasters when they strike. A basic knowledge of Veritas Volume Manager will be assumed. If you are new to Veritas Volume Manager, consult the vxintro(1m) man page for an introduction to terminology and basic usage.

Veritas Disk Headers

When a device is initialized or encapsulated with Veritas Volume Manager, a disk header is written to the device’s private region. The disk header contains a diskid to uniquely identify the device, a disk group identifier to indicate which disk group the device is associated with, a set of flags to indicate the status and intended use (e.g., hot spare) of the device, and a hostid to indicate which host (if any) has the device imported. The disk header can be printed by passing the Veritas device name to vxdisk(1m)‘s “list” option:

$ vxdisk list c1t1d0

Device:    c1t1d0s2
devicetag: c1t1d0
type:      auto
hostid:    pooh
disk:      name=c1t1d0 id=1123602295.10.pooh
group:     name=oradg id=1123603158.13.pooh
info:      format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags:     online ready private autoconfig autoimport imported
pubpaths:  block=/dev/vx/dmp/c1t1d0s2 char=/dev/vx/rdmp/c1t1d0s2
version:   3.1
iosize:    min=512 (bytes) max=2048 (blocks)
public:    slice=2 offset=2304 len=35365968 disk_offset=0
private:   slice=2 offset=256 len=2048 disk_offset=0
update:    time=1123603160 seqno=0.6
ssb:       actual_seqno=0.0
headers:   0 240
configs:   count=1 len=1280
logs:      count=1 len=192
Defined regions:
 config   priv 000048-000239[000192]: copy=01 offset=000000 enabled
 config   priv 000256-001343[001088]: copy=01 offset=000192 enabled
 log      priv 001344-001535[000192]: copy=01 offset=000000 enabled
 lockrgn  priv 001536-001679[000144]: part=00 offset=000000
Multipathing information:
numpaths:   1
c1t1d0s2        state=enabled

Due to the critical nature of the configuration information stored in the disk headers, it is important to periodically back up this information. As I will discuss in the section Configuration Backups, Veritas introduced new features in Veritas Volume Manager 4.X to ease the backup process.

Veritas Configuration Database

When new objects (e.g., subdisks, plexes, volumes) are created with the Veritas CLI or GUI, Veritas will write the configuration data to describe these objects to a configuration database. The configuration database is stored in the private region of several devices in each disk group for redundancy. To list the devices in a disk group with a copy of the configuration database, the vxdg(1m) utility can be invoked with the “list” option and a disk group to query:

$ vxdg list oradg | egrep "config disk.*clean online"

config disk c1t1d0s2 copy 1 len=1280 state=clean online
config disk c1t2d0s2 copy 1 len=1280 state=clean online
config disk c1t3d0s2 copy 1 len=1280 state=clean online
config disk c1t4d0s2 copy 1 len=1280 state=clean online
config disk c1t5d0s2 copy 1 len=1280 state=clean online

To print the contents of the configuration database, the vxprint(1m) utility can be used. The following example uses the vxprint(1m) “-h” (list record hierarchies) and “-t” (use one-line format tailored for each record type) options to display the configuration database with an informational header and descriptive records:

$ vxprint -ht

DG NAME  NCONFIG     NLOG     MINORS   GROUP-ID
ST NAME  STATE       DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME  DEVICE      TYPE     PRIVLEN  PUBLEN   STATE
RV NAME  RLINK_CNT   KSTATE   STATE    PRIMARY  DATAVOLS SRL
RL NAME  RVG         KSTATE   STATE    REM_HOST REM_DG   REM_RLNK
CO NAME  CACHEVOL    KSTATE   STATE
VT NAME  NVOLUME     KSTATE   STATE
V  NAME  RVG/VSET/CO KSTATE   STATE    LENGTH   READPOL PREFPLEX UTYPE
PL NAME  VOLUME      KSTATE   STATE    LENGTH   LAYOUT  NCOL/WID  MODE
SD NAME  PLEX        DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE  MODE
SV NAME  PLEX        VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM   MODE
SC NAME  PLEX        CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE  MODE
DC NAME  PARENTVOL   LOGVOL
SP NAME  SNAPVOL     DCO

dg oradg     default      default  46000    1123603158.13.pooh

dm c1t1d0    c1t1d0s2     auto     2048     35365968 -
dm c1t2d0    c1t2d0s2     auto     2048     35521408 -
dm c1t3d0    c1t3d0s2     auto     2048     35521408 -
dm c1t4d0    c1t4d0s2     auto     2048     35521408 -
dm c1t5d0    c1t5d0s2     auto     2048     35365968 -
dm c1t6d0    c1t6d0s2     auto     2048     35521408 -

v  oravol01     -        ENABLED  ACTIVE   41943040 SELECT    oravol01-03 fsgen
pl oravol01-03  oravol01 ENABLED  ACTIVE    41943168 STRIPE  3/128  RW
sv oravol01-S01 oravol01-03  oravol01-L01 1  13981056 0/0    2/2   ENA
sv oravol01-S02 oravol01-03  oravol01-L02 1  13981056 1/0    2/2   ENA
sv oravol01-S03 oravol01-03  oravol01-L03 1  13981056 2/0    2/2   ENA

v  oravol01-L01 -        ENABLED  ACTIVE   13981056 SELECT     - fsgen
pl oravol01-P01 oravol01-L01 ENABLED  ACTIVE  13981056 CONCAT  -    RW
sd c1t1d0-02    oravol01-P01 c1t1d0   0       13981056 0   c1t1d0  ENA
pl oravol01-P02 oravol01-L01 ENABLED  ACTIVE  13981056 CONCAT  -    RW
sd c1t4d0-02    oravol01-P02 c1t4d0   0       13981056 0   c1t4d0  ENA

v  oravol01-L02 -            ENABLED  ACTIVE  13981056 SELECT  - fsgen
pl oravol01-P03 oravol01-L02 ENABLED  ACTIVE  13981056 CONCAT  -    RW
sd c1t2d0-02    oravol01-P03 c1t2d0   0       13981056 0   c1t2d0  ENA
pl oravol01-P04 oravol01-L02 ENABLED  ACTIVE  13981056 CONCAT  -    RW
sd c1t5d0-02    oravol01-P04 c1t5d0   0       13981056 0   c1t5d0  ENA

v  oravol01-L03 -            ENABLED  ACTIVE  13981056 SELECT  - fsgen
pl oravol01-P05 oravol01-L03 ENABLED  ACTIVE  13981056 CONCAT  -    RW
sd c1t3d0-02    oravol01-P05 c1t3d0   0       13981056 0   c1t3d0  ENA
pl oravol01-P06 oravol01-L03 ENABLED  ACTIVE  13981056 CONCAT  -    RW
sd c1t6d0-02    oravol01-P06 c1t6d0   0       13981056 0   c1t6d0  ENA

The object types (e.g., subdisks, disk groups, plexes, volumes, etc.), block offsets, object names, and locations can be useful for understanding the Veritas Volume Manager layout and reassembling the configuration layout during a disaster. Due to the importance of this information, backing up the configuration database is imperative! As you will see in the next section, Veritas Volume Manager makes backing up the configuration information a snap.

Configuration Backups

The vxconfigbackupd(1m) daemon was introduced in Veritas Volume Manager 4.X to provide an automated way to back up disk headers and the configuration database. Vxconfigbackupd(1m) is started automatically at system boot time and actively monitors the configuration database and disk headers. When vxconfigbackupd(1m) detects a change to the active configuration, vxconfigbackupd(1m) will write the configuration to a set of files in the /etc/vx/cbr/bk/. directory. (“” identifies the disk group, and “” identifies the disk group id.) The vxconfigbackupd(1m) man page provides the following descriptions for the files in the /etc/vx/cbr/bk/. directory:

/etc/vx/cbr/bk/<dgname>.<dgid>/<dgid>.diskinfo
    Location of backup file for disk attributes.

/etc/vx/cbr/bk/<dgname>>.<dgid>/<dgid>.binconfig
    Location of backup file for  binary  configuration copy.

/etc/vx/cbr/bk/<dgname>>.<dgid>/<dgid>.cfgrec
    Location of backup file for configuration  records
    in vxprint -m format.

In addition to the automated backups that are performed by vxconfigbackupd(1m), manual backups can be performed with the vxconfigbackup(1m) utility. This is valuable for backing up specific versions of the configuration and archiving the configuration to a user-supplied destination. The following example uses the vxconfigbackup(1m) “-l” (location to store backup) to back up the current configuration to the /etc/dgbackups directory:

$ /usr/lib/vxvm/bin/vxconfigbackup -l /etc/dgbackups

Start backing up diskgroup oradg to \
  /etc/dgbackups/oradg.1127240283.19.winnie ...
VxVM  NOTICE V-5-2-3100 Backup complete for diskgroup: oradg

If you are using a version of Veritas Volume Manager prior to 4.X, you will be unable to use the vxconfigbackupd(1m) and vxconfigbackup(1m) utilities to manage backups. Don’t fear – the sample shell script in Listing 1 can be used to archive the configuration to a text file, which can be passed to vxmake(1m)‘s “-d” option to re-create the configuration if disaster were to strike.

Restoring Configuration Data

When configuration information is lost due to a system outage or a mistake at the keyboard, the configuration can often be recovered to a point in time (preferably to a point in time when a known working backup was taken with vxconfigbackup(1m)) with the vxconfigrestore(1m) utility. The following example shows how to restore the contents on the oradg disk group using the configuration backup in the /etc/dgbackups directory:

$ /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups oradg

Diskgroup oradg configuration restoration started ......

Installing volume manager disk header for c1t1d0s2 ...
Installing volume manager disk header for c1t2d0s2 ...
Installing volume manager disk header for c1t3d0s2 ...
Installing volume manager disk header for c1t4d0s2 ...
Installing volume manager disk header for c1t5d0s2 ...
Installing volume manager disk header for c1t6d0s2 ...
-
oradg's diskgroup configuration is restored (in a precommitted state).
Diskgroup can be accessed in read only and can be examined using
vxprint(1m) in this state.

Run:
  vxconfigrestore -l /etc/dgbackups/ -c oradg ==> to commit the restoration.
  vxconfigrestore -l /etc/dgbackups/ -d oradg ==> to abort the restoration.

When vxconfigrestore(1m) is invoked, the configuration will be staged to a transient location, allowing the administrator to validate the configuration with the vxprint(1m) utility. If the configuration looks sound, the configuration can be committed to the disk headers and the configuration database by invoking vxconfigrestore(1m) with the “-c” (commit) option:

$ /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups/ -c oradg

Committing configuration restoration for diskgroup oradg ....
oradg's diskgroup configuration restoration is committed.

Once the configuration is restored, the volumes can be started, and the file systems that reside on those volume can be mounted.

Getting Notified When Failures Occur

When Veritas volume manager is installed with the default options, the vxrelocd(1m) failure event detection and subdisk relocation daemon is started at boot time. This daemon is responsible for detecting device failures, relocating subdisks on failed devices to alternate devices, and notifying personnel that a device or Veritas object failed. The notifications are sent to the root user by default and look similar to the following:

To: root
Subject: Volume Manager failures on host winnie
Content-Length: 240

Failures have been detected by the VERITAS Volume Manager:

failed disks:
 c1t4d0

failed plexes:
raid5vol-P08

Since e-mail addressed to the root user is directed to the local mail spool by default, it is advantageous to forward all messages addressed to root to an actual e-mail address. This can be accomplished by enabling email forwarding and adding the following alias to /etc/aliases:

root: admin@mycompany.com

If you would prefer to use a separate account for notifications, you can append the account name to the line that starts vxrelocd(1m) in the vxvm-recover init script:

$ grep vxrelocd /etc/init.d/S95vxvm-recover

vxrelocd root storageproblems &amp;

Configuring the notification facility to route messages to a valid account is crucial and ensures timely notifications when failures are detected.

Case Study #1: Recovering from a Power Failure

While sitting at my desk on a Friday afternoon (around 6pm), my e-mail client chimed to alert me to the following message:

To: root
Subject: Volume Manager failures on host winnie
Content-Length: 240

Failures have been detected by the VERITAS Volume Manager:

failed volumes:
 oravol01

I immediately logged into winnie and received the following error when trying to list the directory that was on the failed volume oravol01:

$ ls -la /u01

.: I/O error

A quick check of the system logs revealed numerous SCSI error messages:

$ tail /var/adm/messages

Jul 26 17:49:37 winnie scsi: [ID 107833 kern.warning] WARNING: /pci&#64;1f,0/pci&#64;1/scsi&#64;4/sd&#64;1,0 (sd1):
Jul 26 17:49:37 winnie  SCSI transport failed: reason 'incomplete': retrying command
Jul 26 17:49:37 winnie scsi: [ID 107833 kern.warning] WARNING: /pci&#64;1f,0/pci&#64;1/scsi&#64;4/sd&#64;4,0 (sd5):
Jul 26 17:49:37 winnie  SCSI transport failed: reason 'incomplete': retrying command
Jul 26 17:49:40 winnie scsi: [ID 107833 kern.warning] WARNING: /pci&#64;1f,0/pci&#64;1/scsi&#64;4/sd&#64;1,0 (sd1):
Jul 26 17:49:40 winnie  disk not responding to selection
Jul 26 17:49:40 winnie scsi: [ID 107833 kern.warning] WARNING: /pci&#64;1f,0/pci&#64;1/scsi&#64;4/sd&#64;4,0 (sd5):
Jul 26 17:49:40 winnie  disk not responding to selection
Jul 26 17:49:42 winnie scsi: [ID 107833 kern.warning] WARNING: /pci&#64;1f,0/pci&#64;1/scsi&#64;4/sd&#64;1,0 (sd1):
Jul 26 17:49:42 winnie  disk not responding to selection
Jul 26 17:49:44 winnie scsi: [ID 107833 kern.warning] WARNING: /pci&#64;1f,0/pci&#64;1/scsi&#64;4/sd&#64;1,0 (sd1):
Jul 26 17:49:44 winnie  disk not responding to selection

The Veritas vxprint(1m) utility was also unable to display the disk group configuration (since the configuration database was unavailable):

$ vxprint -g oradg -ht
$

I raced to the machine room to check the status of the hardware and started to speculate that a SCSI cable or the storage array had failed. When I checked the hardware, I noticed that the storage array was powered off, and the equipment in the surrounding racks was working correctly. Based on my findings, I thought that a power cord failed. Once I replaced the cord, the storage array came back to life, but the oradg disk group I needed to access was in the disabled state:

$ vxdg list

NAME         STATE         ID
oradg        disabled      1123603158.13.winnie

A quick check of the disks showed that they were online and associated with the disabled disk group:

$ vxdisk list

DEVICE       TYPE            DISK         GROUP      STATUS
c0t0d0s2     auto:none       -            -          online invalid
c0t1d0s2     auto:none       -            -          online invalid
c1t1d0s2     auto:cdsdisk    c1t1d0       oradg      online dgdisabled
c1t2d0s2     auto:cdsdisk    c1t2d0       oradg      online dgdisabled
c1t3d0s2     auto:cdsdisk    c1t3d0       oradg      online dgdisabled
c1t4d0s2     auto:cdsdisk    c1t4d0       oradg      online dgdisabled
c1t5d0s2     auto:cdsdisk    c1t5d0       oradg      online dgdisabled
c1t6d0s2     auto:cdsdisk    c1t6d0       oradg      online dgdisabled

In some situations, Veritas may report offline devices as “failed was: cXtXdXs2”. When this happens, the vxreattach(1m) command can reconnect Veritas Volume Manager to “lost” devices. Luckily, in our case, Veritas was able to reconnect to the devices so I unmounted the file system, deported the disk group, and imported the disk group to enable the oradg disk group:

$ umount /u01

$ vxdg deport oradg

$ vxdg import oradg

The deport and import operations are required to fix a disabled disk group and will validate that the disk group configuration records are consistent. Once the disk group was imported, I ran vxinfo(1m) to view the volume and plex status:

$ vxinfo -g oradg -p

vol  oravol01       fsgen    Startable
plex oravol01-03    ACTIVE
vol  oravol01-L01   fsgen    Startable
plex oravol01-P01   ACTIVE
plex oravol01-P02   ACTIVE
vol  oravol01-L02   fsgen    Startable
plex oravol01-P03   ACTIVE
plex oravol01-P04   ACTIVE
vol  oravol01-L03   fsgen    Startable
plex oravol01-P05   ACTIVE
plex oravol01-P06   ACTIVE

The “Startable” flag indicates that the volume and plexes are in a startable state, so I executed the vxvol(1m) utility to start the volume:

$ vxvol -g oradg start oravol01

Once the volume came online, I ran fsck(1m) to replay the transactions in the VxFS journal:

$ fsck -F vxfs /dev/vx/dsk/oof/oravol01

log replay in progress
replay complete - marking super-block as CLEAN

After fsck(1m) finished the consistency check, I mounted the file system, applied the archive logs, and was able to bring the database back up to an operational state. Due to the recovery features built into Veritas volume manager, Veritas file system, and Oracle, we were able to avoid a full file system restore! Since I received the failure notification immediately after Veritas detected a problem with the volume, I was able to reduce the time it took to recover the faulted system.

Case Study #2: Recovering a Deleted Volume

While debugging a C program on a Friday afternoon (around 6pm), one of my colleagues came to my desk and told me he had accidentally destroyed the production disk group instead of the test disk group he thought he was working on. A quick check of the Veritas configuration verified this:

$ vxprint -hft
$

Since we used vxconfigbackup(1m) to take regular backups of the disk group configuration, we located the last known good working backup and passed this location to vxconfigrestore(1m)‘s “-l” (location) option:

$ /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups/oradg_0612205 oradg

Diskgroup oradg configuration restoration started ......

Installing volume manager disk header for c1t1d0s2 ...
Installing volume manager disk header for c1t2d0s2 ...
Installing volume manager disk header for c1t3d0s2 ...
Installing volume manager disk header for c1t4d0s2 ...
Installing volume manager disk header for c1t5d0s2 ...
Installing volume manager disk header for c1t6d0s2 ...

oradg's diskgroup configuration is restored (in precommitted state).
Diskgroup can be accessed in read only and can be examined using
vxprint(1m) in this state.

Run:
  vxconfigrestore -l /etc/dgbackups/oradg_0612205 -c oradg ==> to commit the restoration.
  vxconfigrestore -l /etc/dgbackups/oradg_0612205 -d oradg ==> to abort the restoration.

I verified the configuration was consistent by running vxprint(1m), and committed the configuration to the disk headers and configuration database by invoking vxconfigrestore(1m) with the “-c” option:

$ /usr/lib/vxvm/bin/vxconfigrestore -c -l /var/tmp/back oradg

Committing configuration restoration for diskgroup oradg ....

oradg's diskgroup configuration restoration is committed.

Once we committed the configuration, the layout matched our last known good configuration backup:

$ vxprint -ht

Disk group: oradg

DG NAME  NCONFIG     NLOG    MINORS   GROUP-ID
ST NAME  STATE       DM_CNT  SPARE_CNT         APPVOL_CNT
DM NAME  DEVICE      TYPE    PRIVLEN  PUBLEN   STATE
RV NAME  RLINK_CNT   KSTATE  STATE    PRIMARY  DATAVOLS  SRL
RL NAME  RVG         KSTATE  STATE    REM_HOST REM_DG    REM_RLNK
CO NAME  CACHEVOL    KSTATE  STATE
VT NAME  NVOLUME     KSTATE  STATE
V  NAME  RVG/VSET/CO KSTATE  STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME  VOLUME      KSTATE  STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME  PLEX        DISK    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME  PLEX        VOLNAME NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME  PLEX        CACHE   DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME  PARENTVOL   LOGVOL
SP NAME  SNAPVOL     DCO

dg oradg    default     default  10000    1127240283.19.winnie

v  oravol01  -   ENABLED  ACTIVE   41943040 SELECT    oravol01-03 fsgen
pl oravol01-03  oravol01     ENABLED ACTIVE  41943168 STRIPE 3/128  RW
sv oravol01-S01 oravol01-03  oravol01-L01 1  13981056 0/0    2/2    ENA
sv oravol01-S02 oravol01-03  oravol01-L02 1  13981056 1/0    2/2    ENA
sv oravol01-S03 oravol01-03  oravol01-L03 1  13981056 2/0    2/2    ENA

v  oravol01-L01 -            ENABLED  SYNC   13981056 SELECT -    fsgen
pl oravol01-P01 oravol01-L01 ENABLED  ACTIVE 13981056 CONCAT -      RW
sd c1t1d0-02    oravol01-P01 c1t1d0   0      13981056 0    c1t1d0   ENA
pl oravol01-P02 oravol01-L01 ENABLED  ACTIVE 13981056 CONCAT -      RW
sd c1t4d0-02    oravol01-P02 c1t4d0   0      13981056 0    c1t4d0   ENA

v  oravol01-L02 -            ENABLED  SYNC   13981056 SELECT -    fsgen
pl oravol01-P03 oravol01-L02 ENABLED  ACTIVE 13981056 CONCAT -      RW
sd c1t2d0-02    oravol01-P03 c1t2d0   0      13981056 0    c1t2d0   ENA
pl oravol01-P04 oravol01-L02 ENABLED  ACTIVE 13981056 CONCAT -       RW
sd c1t5d0-02    oravol01-P04 c1t5d0   0      13981056 0    c1t5d0   ENA

v  oravol01-L03 -            ENABLED  SYNC   13981056 SELECT -    fsgen
pl oravol01-P05 oravol01-L03 ENABLED  ACTIVE 13981056 CONCAT -       RW
sd c1t3d0-02    oravol01-P05 c1t3d0   0      13981056 0    c1t3d0   ENA
pl oravol01-P06 oravol01-L03 ENABLED  ACTIVE 13981056 CONCAT -       RW
sd c1t6d0-02    oravol01-P06 c1t6d0   0      13981056 0    c1t6d0   ENA

To ensure that no damage had occurred to the file system, we used fsck(1m) to consistency check the file system:

$ fsck -F vxfs /dev/vx/rdsk/oradg/oravol01

and got our Oracle DBA to verify the integrity of each data file with Oracle’s database verification utilities. Because accidents and disasters can strike at any time, it is important to perform regular backups of the Veritas configuration and store this data in a safe location (having a configuration database backup saved us from having to restore the database or re-create the Veritas objects by hand).

Conclusion

This article provided an introduction to two important Veritas Volume Manager recovery techniques and discussed ways to ensure recoverability after a disaster. When dealing with difficult Veritas situations, it is imperative to know what function a command performs before executing it. It is also helpful to engage Veritas support or the folks on the Veritas mailing lists when major disasters occur. This will ensure that a second set of eyes is available to review the problem, and will ensure the quickest and safest methods are used to recover a faulted system. I tested all commands on a Solaris 10 server running Veritas Volume Manager 4.1, and I highly recommend testing all configuration changes on non-production systems.

References

The following references were used while writing this article:

Acknowledgements

Ryan would like to thank the individuals who maintain the SSA helpful hints site, as well as the members of the Veritas Volume Manager list.

Originally published in the January ‘06 issue of SysAdmin Magazine