Veritas Volume Manager (VxVM) has become the standard Logical Volume Manager (LVM) in many enterprises for its robust feature set, its ability to run on multiple operating systems (e.g., HP-UX, Linux, Solaris, Windows), and the numerous scalability, availability, and recoverability features that come with the base product. The recoverability features help to ensure that data is protected when hardware platforms fail and to ease the process required to restore systems to an operational state.
This article will provide an introduction to two important and often overlooked recovery features: failure notifications and configuration database backups. The article will also provide two disaster-recovery case studies to show how these recovery features can be used to aide in recovering from disasters when they strike. A basic knowledge of Veritas Volume Manager will be assumed. If you are new to Veritas Volume Manager, consult the vxintro(1m) man page for an introduction to terminology and basic usage.
When a device is initialized or encapsulated with Veritas Volume Manager, a disk header is written to the device’s private region. The disk header contains a diskid to uniquely identify the device, a disk group identifier to indicate which disk group the device is associated with, a set of flags to indicate the status and intended use (e.g., hot spare) of the device, and a hostid to indicate which host (if any) has the device imported. The disk header can be printed by passing the Veritas device name to vxdisk(1m)‘s “list” option:
$ vxdisk list c1t1d0
Device: c1t1d0s2
devicetag: c1t1d0
type: auto
hostid: pooh
disk: name=c1t1d0 id=1123602295.10.pooh
group: name=oradg id=1123603158.13.pooh
info: format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags: online ready private autoconfig autoimport imported
pubpaths: block=/dev/vx/dmp/c1t1d0s2 char=/dev/vx/rdmp/c1t1d0s2
version: 3.1
iosize: min=512 (bytes) max=2048 (blocks)
public: slice=2 offset=2304 len=35365968 disk_offset=0
private: slice=2 offset=256 len=2048 disk_offset=0
update: time=1123603160 seqno=0.6
ssb: actual_seqno=0.0
headers: 0 240
configs: count=1 len=1280
logs: count=1 len=192
Defined regions:
config priv 000048-000239[000192]: copy=01 offset=000000 enabled
config priv 000256-001343[001088]: copy=01 offset=000192 enabled
log priv 001344-001535[000192]: copy=01 offset=000000 enabled
lockrgn priv 001536-001679[000144]: part=00 offset=000000
Multipathing information:
numpaths: 1
c1t1d0s2 state=enabled
Due to the critical nature of the configuration information stored in the disk headers, it is important to periodically back up this information. As I will discuss in the section Configuration Backups, Veritas introduced new features in Veritas Volume Manager 4.X to ease the backup process.
When new objects (e.g., subdisks, plexes, volumes) are created with the Veritas CLI or GUI, Veritas will write the configuration data to describe these objects to a configuration database. The configuration database is stored in the private region of several devices in each disk group for redundancy. To list the devices in a disk group with a copy of the configuration database, the vxdg(1m) utility can be invoked with the “list” option and a disk group to query:
$ vxdg list oradg | egrep "config disk.*clean online"
config disk c1t1d0s2 copy 1 len=1280 state=clean online
config disk c1t2d0s2 copy 1 len=1280 state=clean online
config disk c1t3d0s2 copy 1 len=1280 state=clean online
config disk c1t4d0s2 copy 1 len=1280 state=clean online
config disk c1t5d0s2 copy 1 len=1280 state=clean online
To print the contents of the configuration database, the vxprint(1m) utility can be used. The following example uses the vxprint(1m) “-h” (list record hierarchies) and “-t” (use one-line format tailored for each record type) options to display the configuration database with an informational header and descriptive records:
$ vxprint -ht
DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME NVOLUME KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO
dg oradg default default 46000 1123603158.13.pooh
dm c1t1d0 c1t1d0s2 auto 2048 35365968 -
dm c1t2d0 c1t2d0s2 auto 2048 35521408 -
dm c1t3d0 c1t3d0s2 auto 2048 35521408 -
dm c1t4d0 c1t4d0s2 auto 2048 35521408 -
dm c1t5d0 c1t5d0s2 auto 2048 35365968 -
dm c1t6d0 c1t6d0s2 auto 2048 35521408 -
v oravol01 - ENABLED ACTIVE 41943040 SELECT oravol01-03 fsgen
pl oravol01-03 oravol01 ENABLED ACTIVE 41943168 STRIPE 3/128 RW
sv oravol01-S01 oravol01-03 oravol01-L01 1 13981056 0/0 2/2 ENA
sv oravol01-S02 oravol01-03 oravol01-L02 1 13981056 1/0 2/2 ENA
sv oravol01-S03 oravol01-03 oravol01-L03 1 13981056 2/0 2/2 ENA
v oravol01-L01 - ENABLED ACTIVE 13981056 SELECT - fsgen
pl oravol01-P01 oravol01-L01 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t1d0-02 oravol01-P01 c1t1d0 0 13981056 0 c1t1d0 ENA
pl oravol01-P02 oravol01-L01 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t4d0-02 oravol01-P02 c1t4d0 0 13981056 0 c1t4d0 ENA
v oravol01-L02 - ENABLED ACTIVE 13981056 SELECT - fsgen
pl oravol01-P03 oravol01-L02 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t2d0-02 oravol01-P03 c1t2d0 0 13981056 0 c1t2d0 ENA
pl oravol01-P04 oravol01-L02 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t5d0-02 oravol01-P04 c1t5d0 0 13981056 0 c1t5d0 ENA
v oravol01-L03 - ENABLED ACTIVE 13981056 SELECT - fsgen
pl oravol01-P05 oravol01-L03 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t3d0-02 oravol01-P05 c1t3d0 0 13981056 0 c1t3d0 ENA
pl oravol01-P06 oravol01-L03 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t6d0-02 oravol01-P06 c1t6d0 0 13981056 0 c1t6d0 ENA
The object types (e.g., subdisks, disk groups, plexes, volumes, etc.), block offsets, object names, and locations can be useful for understanding the Veritas Volume Manager layout and reassembling the configuration layout during a disaster. Due to the importance of this information, backing up the configuration database is imperative! As you will see in the next section, Veritas Volume Manager makes backing up the configuration information a snap.
The vxconfigbackupd(1m) daemon was introduced in Veritas Volume Manager 4.X to provide an automated way to back up disk headers and the configuration database. Vxconfigbackupd(1m) is started automatically at system boot time and actively monitors the configuration database and disk headers. When vxconfigbackupd(1m) detects a change to the active configuration, vxconfigbackupd(1m) will write the configuration to a set of files in the /etc/vx/cbr/bk/. directory. (“” identifies the disk group, and “” identifies the disk group id.) The vxconfigbackupd(1m) man page provides the following descriptions for the files in the /etc/vx/cbr/bk/. directory:
/etc/vx/cbr/bk/<dgname>.<dgid>/<dgid>.diskinfo
Location of backup file for disk attributes.
/etc/vx/cbr/bk/<dgname>>.<dgid>/<dgid>.binconfig
Location of backup file for binary configuration copy.
/etc/vx/cbr/bk/<dgname>>.<dgid>/<dgid>.cfgrec
Location of backup file for configuration records
in vxprint -m format.
In addition to the automated backups that are performed by vxconfigbackupd(1m), manual backups can be performed with the vxconfigbackup(1m) utility. This is valuable for backing up specific versions of the configuration and archiving the configuration to a user-supplied destination. The following example uses the vxconfigbackup(1m) “-l” (location to store backup) to back up the current configuration to the /etc/dgbackups directory:
$ /usr/lib/vxvm/bin/vxconfigbackup -l /etc/dgbackups
Start backing up diskgroup oradg to \
/etc/dgbackups/oradg.1127240283.19.winnie ...
VxVM NOTICE V-5-2-3100 Backup complete for diskgroup: oradg
If you are using a version of Veritas Volume Manager prior to 4.X, you will be unable to use the vxconfigbackupd(1m) and vxconfigbackup(1m) utilities to manage backups. Don’t fear – the sample shell script in Listing 1 can be used to archive the configuration to a text file, which can be passed to vxmake(1m)‘s “-d” option to re-create the configuration if disaster were to strike.
When configuration information is lost due to a system outage or a mistake at the keyboard, the configuration can often be recovered to a point in time (preferably to a point in time when a known working backup was taken with vxconfigbackup(1m)) with the vxconfigrestore(1m) utility. The following example shows how to restore the contents on the oradg disk group using the configuration backup in the /etc/dgbackups directory:
$ /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups oradg
Diskgroup oradg configuration restoration started ......
Installing volume manager disk header for c1t1d0s2 ...
Installing volume manager disk header for c1t2d0s2 ...
Installing volume manager disk header for c1t3d0s2 ...
Installing volume manager disk header for c1t4d0s2 ...
Installing volume manager disk header for c1t5d0s2 ...
Installing volume manager disk header for c1t6d0s2 ...
-
oradg's diskgroup configuration is restored (in a precommitted state).
Diskgroup can be accessed in read only and can be examined using
vxprint(1m) in this state.
Run:
vxconfigrestore -l /etc/dgbackups/ -c oradg ==> to commit the restoration.
vxconfigrestore -l /etc/dgbackups/ -d oradg ==> to abort the restoration.
When vxconfigrestore(1m) is invoked, the configuration will be staged to a transient location, allowing the administrator to validate the configuration with the vxprint(1m) utility. If the configuration looks sound, the configuration can be committed to the disk headers and the configuration database by invoking vxconfigrestore(1m) with the “-c” (commit) option:
$ /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups/ -c oradg
Committing configuration restoration for diskgroup oradg ....
oradg's diskgroup configuration restoration is committed.
Once the configuration is restored, the volumes can be started, and the file systems that reside on those volume can be mounted.
When Veritas volume manager is installed with the default options, the vxrelocd(1m) failure event detection and subdisk relocation daemon is started at boot time. This daemon is responsible for detecting device failures, relocating subdisks on failed devices to alternate devices, and notifying personnel that a device or Veritas object failed. The notifications are sent to the root user by default and look similar to the following:
To: root
Subject: Volume Manager failures on host winnie
Content-Length: 240
Failures have been detected by the VERITAS Volume Manager:
failed disks:
c1t4d0
failed plexes:
raid5vol-P08
Since e-mail addressed to the root user is directed to the local mail spool by default, it is advantageous to forward all messages addressed to root to an actual e-mail address. This can be accomplished by enabling email forwarding and adding the following alias to /etc/aliases:
root: admin@mycompany.com
If you would prefer to use a separate account for notifications, you can append the account name to the line that starts vxrelocd(1m) in the vxvm-recover init script:
$ grep vxrelocd /etc/init.d/S95vxvm-recover
vxrelocd root storageproblems &
Configuring the notification facility to route messages to a valid account is crucial and ensures timely notifications when failures are detected.
While sitting at my desk on a Friday afternoon (around 6pm), my e-mail client chimed to alert me to the following message:
To: root
Subject: Volume Manager failures on host winnie
Content-Length: 240
Failures have been detected by the VERITAS Volume Manager:
failed volumes:
oravol01
I immediately logged into winnie and received the following error when trying to list the directory that was on the failed volume oravol01:
$ ls -la /u01
.: I/O error
A quick check of the system logs revealed numerous SCSI error messages:
$ tail /var/adm/messages
Jul 26 17:49:37 winnie scsi: [ID 107833 kern.warning] WARNING: /pci@1f,0/pci@1/scsi@4/sd@1,0 (sd1):
Jul 26 17:49:37 winnie SCSI transport failed: reason 'incomplete': retrying command
Jul 26 17:49:37 winnie scsi: [ID 107833 kern.warning] WARNING: /pci@1f,0/pci@1/scsi@4/sd@4,0 (sd5):
Jul 26 17:49:37 winnie SCSI transport failed: reason 'incomplete': retrying command
Jul 26 17:49:40 winnie scsi: [ID 107833 kern.warning] WARNING: /pci@1f,0/pci@1/scsi@4/sd@1,0 (sd1):
Jul 26 17:49:40 winnie disk not responding to selection
Jul 26 17:49:40 winnie scsi: [ID 107833 kern.warning] WARNING: /pci@1f,0/pci@1/scsi@4/sd@4,0 (sd5):
Jul 26 17:49:40 winnie disk not responding to selection
Jul 26 17:49:42 winnie scsi: [ID 107833 kern.warning] WARNING: /pci@1f,0/pci@1/scsi@4/sd@1,0 (sd1):
Jul 26 17:49:42 winnie disk not responding to selection
Jul 26 17:49:44 winnie scsi: [ID 107833 kern.warning] WARNING: /pci@1f,0/pci@1/scsi@4/sd@1,0 (sd1):
Jul 26 17:49:44 winnie disk not responding to selection
The Veritas vxprint(1m) utility was also unable to display the disk group configuration (since the configuration database was unavailable):
$ vxprint -g oradg -ht
$
I raced to the machine room to check the status of the hardware and started to speculate that a SCSI cable or the storage array had failed. When I checked the hardware, I noticed that the storage array was powered off, and the equipment in the surrounding racks was working correctly. Based on my findings, I thought that a power cord failed. Once I replaced the cord, the storage array came back to life, but the oradg disk group I needed to access was in the disabled state:
$ vxdg list
NAME STATE ID
oradg disabled 1123603158.13.winnie
A quick check of the disks showed that they were online and associated with the disabled disk group:
$ vxdisk list
DEVICE TYPE DISK GROUP STATUS
c0t0d0s2 auto:none - - online invalid
c0t1d0s2 auto:none - - online invalid
c1t1d0s2 auto:cdsdisk c1t1d0 oradg online dgdisabled
c1t2d0s2 auto:cdsdisk c1t2d0 oradg online dgdisabled
c1t3d0s2 auto:cdsdisk c1t3d0 oradg online dgdisabled
c1t4d0s2 auto:cdsdisk c1t4d0 oradg online dgdisabled
c1t5d0s2 auto:cdsdisk c1t5d0 oradg online dgdisabled
c1t6d0s2 auto:cdsdisk c1t6d0 oradg online dgdisabled
In some situations, Veritas may report offline devices as “failed was: cXtXdXs2”. When this happens, the vxreattach(1m) command can reconnect Veritas Volume Manager to “lost” devices. Luckily, in our case, Veritas was able to reconnect to the devices so I unmounted the file system, deported the disk group, and imported the disk group to enable the oradg disk group:
$ umount /u01
$ vxdg deport oradg
$ vxdg import oradg
The deport and import operations are required to fix a disabled disk group and will validate that the disk group configuration records are consistent. Once the disk group was imported, I ran vxinfo(1m) to view the volume and plex status:
$ vxinfo -g oradg -p
vol oravol01 fsgen Startable
plex oravol01-03 ACTIVE
vol oravol01-L01 fsgen Startable
plex oravol01-P01 ACTIVE
plex oravol01-P02 ACTIVE
vol oravol01-L02 fsgen Startable
plex oravol01-P03 ACTIVE
plex oravol01-P04 ACTIVE
vol oravol01-L03 fsgen Startable
plex oravol01-P05 ACTIVE
plex oravol01-P06 ACTIVE
The “Startable” flag indicates that the volume and plexes are in a startable state, so I executed the vxvol(1m) utility to start the volume:
$ vxvol -g oradg start oravol01
Once the volume came online, I ran fsck(1m) to replay the transactions in the VxFS journal:
$ fsck -F vxfs /dev/vx/dsk/oof/oravol01
log replay in progress
replay complete - marking super-block as CLEAN
After fsck(1m) finished the consistency check, I mounted the file system, applied the archive logs, and was able to bring the database back up to an operational state. Due to the recovery features built into Veritas volume manager, Veritas file system, and Oracle, we were able to avoid a full file system restore! Since I received the failure notification immediately after Veritas detected a problem with the volume, I was able to reduce the time it took to recover the faulted system.
While debugging a C program on a Friday afternoon (around 6pm), one of my colleagues came to my desk and told me he had accidentally destroyed the production disk group instead of the test disk group he thought he was working on. A quick check of the Veritas configuration verified this:
$ vxprint -hft
$
Since we used vxconfigbackup(1m) to take regular backups of the disk group configuration, we located the last known good working backup and passed this location to vxconfigrestore(1m)‘s “-l” (location) option:
$ /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups/oradg_0612205 oradg
Diskgroup oradg configuration restoration started ......
Installing volume manager disk header for c1t1d0s2 ...
Installing volume manager disk header for c1t2d0s2 ...
Installing volume manager disk header for c1t3d0s2 ...
Installing volume manager disk header for c1t4d0s2 ...
Installing volume manager disk header for c1t5d0s2 ...
Installing volume manager disk header for c1t6d0s2 ...
oradg's diskgroup configuration is restored (in precommitted state).
Diskgroup can be accessed in read only and can be examined using
vxprint(1m) in this state.
Run:
vxconfigrestore -l /etc/dgbackups/oradg_0612205 -c oradg ==> to commit the restoration.
vxconfigrestore -l /etc/dgbackups/oradg_0612205 -d oradg ==> to abort the restoration.
I verified the configuration was consistent by running vxprint(1m), and committed the configuration to the disk headers and configuration database by invoking vxconfigrestore(1m) with the “-c” option:
$ /usr/lib/vxvm/bin/vxconfigrestore -c -l /var/tmp/back oradg
Committing configuration restoration for diskgroup oradg ....
oradg's diskgroup configuration restoration is committed.
Once we committed the configuration, the layout matched our last known good configuration backup:
$ vxprint -ht
Disk group: oradg
DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME NVOLUME KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO
dg oradg default default 10000 1127240283.19.winnie
v oravol01 - ENABLED ACTIVE 41943040 SELECT oravol01-03 fsgen
pl oravol01-03 oravol01 ENABLED ACTIVE 41943168 STRIPE 3/128 RW
sv oravol01-S01 oravol01-03 oravol01-L01 1 13981056 0/0 2/2 ENA
sv oravol01-S02 oravol01-03 oravol01-L02 1 13981056 1/0 2/2 ENA
sv oravol01-S03 oravol01-03 oravol01-L03 1 13981056 2/0 2/2 ENA
v oravol01-L01 - ENABLED SYNC 13981056 SELECT - fsgen
pl oravol01-P01 oravol01-L01 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t1d0-02 oravol01-P01 c1t1d0 0 13981056 0 c1t1d0 ENA
pl oravol01-P02 oravol01-L01 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t4d0-02 oravol01-P02 c1t4d0 0 13981056 0 c1t4d0 ENA
v oravol01-L02 - ENABLED SYNC 13981056 SELECT - fsgen
pl oravol01-P03 oravol01-L02 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t2d0-02 oravol01-P03 c1t2d0 0 13981056 0 c1t2d0 ENA
pl oravol01-P04 oravol01-L02 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t5d0-02 oravol01-P04 c1t5d0 0 13981056 0 c1t5d0 ENA
v oravol01-L03 - ENABLED SYNC 13981056 SELECT - fsgen
pl oravol01-P05 oravol01-L03 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t3d0-02 oravol01-P05 c1t3d0 0 13981056 0 c1t3d0 ENA
pl oravol01-P06 oravol01-L03 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t6d0-02 oravol01-P06 c1t6d0 0 13981056 0 c1t6d0 ENA
To ensure that no damage had occurred to the file system, we used fsck(1m) to consistency check the file system:
$ fsck -F vxfs /dev/vx/rdsk/oradg/oravol01
and got our Oracle DBA to verify the integrity of each data file with Oracle’s database verification utilities. Because accidents and disasters can strike at any time, it is important to perform regular backups of the Veritas configuration and store this data in a safe location (having a configuration database backup saved us from having to restore the database or re-create the Veritas objects by hand).
This article provided an introduction to two important Veritas Volume Manager recovery techniques and discussed ways to ensure recoverability after a disaster. When dealing with difficult Veritas situations, it is imperative to know what function a command performs before executing it. It is also helpful to engage Veritas support or the folks on the Veritas mailing lists when major disasters occur. This will ensure that a second set of eyes is available to review the problem, and will ensure the quickest and safest methods are used to recover a faulted system. I tested all commands on a Solaris 10 server running Veritas Volume Manager 4.1, and I highly recommend testing all configuration changes on non-production systems.
The following references were used while writing this article:
Ryan would like to thank the individuals who maintain the SSA helpful hints site, as well as the members of the Veritas Volume Manager list.
Originally published in the January ‘06 issue of SysAdmin Magazine