Top Button Left Second Button Left Third Button Left Fourth Button Left Fifth Button Left Sixth Button left Seventh Button left Eight Button Left

Veritas Volume Relayout

When provisioning storage and creating volumes, there are times when you create a volume for a specific workload, and things change after the fact. Veritas volume manager can easily deal with changing requirements, and allows you to convert between volume types ( e.g., convert a RAID5 volume to a striped mirrored volume) on the fly. Veritas performs this operation in most cases with layered volumes, and requires a chunk of free space to complete the relayout operation. The VxVM users guide describes the supported relayout operations, and also provides disk space requirements.

To illustrate just how useful the relayout operation is, let's say your manager just finished reading a Gartner report that criticizes RAID5. He comes over to your desk and asks you to convert the Oracle data volume from a 4-column RAID5 volume to a 2-column striped-mirror volume. Since you despise software RAID5, you set down UNIX File systems and run vxassist(1m) with the "relayout" keyword, the "layout" to convert to, and the number of columns to use (the ncols option is only used with striped volumes):

$ vxassist -g oof relayout oravol01 layout=stripe-mirror ncol=2

The relayout operation requires a temporary region to copy data to (marked with a state of TMP in vxprint) prior to migrating data it to it's final destination. If sufficent space isn't available, vxassist will display an error similar to the following and exit:

VxVM vxassist ERROR V-5-1-6345 Cannot allocate 15728640 blocks of disk space \
                      required by the relayout operation for column addition
VxVM vxassist ERROR V-5-1-4037 Relayout operation aborted. (7)

Once the relayout begins, the vxrelayout(1m) and vxtask(1m) utilities can be used to monitor the progress of the relayout operations:

$ vxrelayout -g oof status oravol01
RAID5,  columns=4,  stwidth=32 -->  STRIPED-MIRROR,  columns=2,  stwidth=128
 Relayout running,  10.02% completed.

$ vxtask list
TASKID  PTID TYPE/STATE    PCT   PROGRESS
  2125         RELAYOUT/R 14.45% 0/41943168/6061184 RELAYOUT oravol01 oof

When the relayout operation completes, the volume will be converted to a 2-column striped-mirrored volume:

$ vxprint -hft
Disk group: oof

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         NVOLUME      KSTATE   STATE
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO

dg oof          default      default  21000    1120604922.22.tigger

dm c1t1d0       c1t1d0s2     auto     2048     35365968 -
dm c1t2d0       c1t2d0s2     auto     2048     35521408 -
dm c1t3d0       c1t3d0s2     auto     2048     35521408 -
dm c1t4d0       c1t4d0s2     auto     2048     35521408 -
dm c1t5d0       c1t5d0s2     auto     2048     35521408 -
dm c1t6d0       c1t6d0s2     auto     2048     35521408 -

v  oravol01     -            ENABLED  ACTIVE   20971584 SELECT    oravol01-01 fsgen
pl oravol01-01  oravol01     ENABLED  ACTIVE   20971776 STRIPE    2/128    RW
sv oravol01-Ds01 oravol01-01 oravol01-d01 1    10485888 0/0       3/3      ENA
sv oravol01-Ds02 oravol01-01 oravol01-d02 1    10485888 1/0       3/3      ENA

v  oravol01-d01 -            ENABLED  ACTIVE   10485888 SELECT    -        fsgen
pl oravol01-dp01 oravol01-d01 ENABLED ACTIVE   10485888 CONCAT    -        RW
sd c1t1d0-04    oravol01-dp01 c1t1d0  0        10485888 0         c1t1d0   ENA
pl oravol01-dp02 oravol01-d01 ENABLED ACTIVE   10485888 CONCAT    -        RW
sd c1t6d0-03    oravol01-dp02 c1t6d0  2096640  10485888 0         c1t6d0   ENA
pl oravol01-dp03 oravol01-d01 ENABLED ACTIVE   LOGONLY  CONCAT    -        RW
sd c1t3d0-02    oravol01-dp03 c1t3d0  6990528  528      LOG       c1t3d0   ENA

v  oravol01-d02 -            ENABLED  ACTIVE   10485888 SELECT    -        fsgen
pl oravol01-dp04 oravol01-d02 ENABLED ACTIVE   10485888 CONCAT    -        RW
sd c1t2d0-04    oravol01-dp04 c1t2d0  0        10485888 0         c1t2d0   ENA
pl oravol01-dp05 oravol01-d02 ENABLED ACTIVE   10485888 CONCAT    -        RW
sd c1t5d0-04    oravol01-dp05 c1t5d0  2100480  10485888 0         c1t5d0   ENA
pl oravol01-dp06 oravol01-d02 ENABLED ACTIVE   LOGONLY  CONCAT    -        RW
sd c1t3d0-03    oravol01-dp06 c1t3d0  6991056  528      LOG       c1t3d0   ENA

Once the relayout completes, you pick up the wonderful Linker's & Libraries guide and begin reading. Just as you get to chapter 7 (Object file formats), your manager comes back to you and asks you to convert the 2-column striped-mirror volume to a 3-column striped-mirror volume. You are eager to use the remaining spindles in your storage array, so you execute vxassist(1m) with the relayout and ncol options:

$ vxassist -b -g oof relayout oravol01 layout=mirror-stripe ncol=3

When adding additional columns to a striped volume with the relayout option, ncol refers to the total number of columns, not the number to add. Once the relayout process completes, you have a 3-column striped-mirrored volume:

$ vxprint -hft
Disk group: oof

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         NVOLUME      KSTATE   STATE
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO

dg oof          6            default  21000    1120604922.22.tigger

dm c1t1d0       c1t1d0s2     auto     2048     35365968 -
dm c1t2d0       c1t2d0s2     auto     2048     35521408 -
dm c1t3d0       c1t3d0s2     auto     2048     35521408 -
dm c1t4d0       c1t4d0s2     auto     2048     35521408 -
dm c1t5d0       c1t5d0s2     auto     2048     35521408 -
dm c1t6d0       c1t6d0s2     auto     2048     35521408 -

v  oravol01     -            ENABLED  ACTIVE   20971520 SELECT    oravol01-03 fsgen
pl oravol01-03  oravol01     ENABLED  ACTIVE   20971776 STRIPE    3/128    RW
sv oravol01-S01 oravol01-03  oravol01-L01 1    6990592  0/0       2/2      ENA
sv oravol01-S02 oravol01-03  oravol01-L02 1    6990592  1/0       2/2      ENA
sv oravol01-S03 oravol01-03  oravol01-L03 1    6990592  2/0       2/2      ENA

v  oravol01-L01 -            ENABLED  SYNC     6990592  SELECT    -        fsgen
pl oravol01-P01 oravol01-L01 ENABLED  ACTIVE   6990592  CONCAT    -        RW
sd c1t1d0-02    oravol01-P01 c1t1d0   0        6990592  0         c1t1d0   ENA
pl oravol01-P02 oravol01-L01 ENABLED  ACTIVE   6990592  CONCAT    -        RW
sd c1t4d0-02    oravol01-P02 c1t4d0   0        6990592  0         c1t4d0   ENA

v  oravol01-L02 -            ENABLED  SYNC     6990592  SELECT    -        fsgen
pl oravol01-P03 oravol01-L02 ENABLED  ACTIVE   6990592  CONCAT    -        RW
sd c1t2d0-02    oravol01-P03 c1t2d0   0        6990592  0         c1t2d0   ENA
pl oravol01-P04 oravol01-L02 ENABLED  ACTIVE   6990592  CONCAT    -        RW
sd c1t5d0-02    oravol01-P04 c1t5d0   0        6990592  0         c1t5d0   ENA

v  oravol01-L03 -            ENABLED  SYNC     6990592  SELECT    -        fsgen
pl oravol01-P05 oravol01-L03 ENABLED  ACTIVE   6990592  CONCAT    -        RW
sd c1t3d0-02    oravol01-P05 c1t3d0   0        6990592  0         c1t3d0   ENA
pl oravol01-P06 oravol01-L03 ENABLED  ACTIVE   6990592  CONCAT    -        RW
sd c1t6d0-02    oravol01-P06 c1t6d0   0        6990592  0         c1t6d0   ENA

While reading through the DTRACE users guide, you notice that the io and syscall providers can be used to determine application I/O patterns. Once you get your D script working, you notice that the Java application running on your server is reading and writing data in 96k chunks. To optimize write performance, you decide to adjust the stripunit size to be a multiple of the application I/O size. This is accomplished with vxassist(1m)'s stripeunit option:

$ vxassist -g oof relayout oravol01 stripeunit=32k

Once this operation completes, you have a 3-column striped-mirrored volume with a 32k stripe unit size (since we have 3-columns, and the application is reading and writing 96k chunks, a 32k stripeunit is optimal):

$ vxprint -hft

Disk group: oof

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         NVOLUME      KSTATE   STATE
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO

dg oof          default      default  21000    1120604922.22.tigger

dm c1t1d0       c1t1d0s2     auto     2048     35365968 -
dm c1t2d0       c1t2d0s2     auto     2048     35521408 -
dm c1t3d0       c1t3d0s2     auto     2048     35521408 -
dm c1t4d0       c1t4d0s2     auto     2048     35521408 -
dm c1t5d0       c1t5d0s2     auto     2048     35521408 -
dm c1t6d0       c1t6d0s2     auto     2048     35521408 -

v  oravol01     -            ENABLED  ACTIVE   20971584 SELECT    oravol01-01 fsgen
pl oravol01-01  oravol01     ENABLED  ACTIVE   20971584 STRIPE    3/64     RW
sv oravol01-Ds01 oravol01-01 oravol01-d01 1    6990528  0/0       3/3      ENA
sv oravol01-Ds02 oravol01-01 oravol01-d02 1    6990528  1/0       3/3      ENA
sv oravol01-Ds06 oravol01-01 oravol01-d06 1    6990528  2/0       3/3      ENA

v  oravol01-d01 -            ENABLED  ACTIVE   6990528  SELECT    -        fsgen
pl oravol01-dp01 oravol01-d01 ENABLED ACTIVE   6990528  CONCAT    -        RW
sd c1t1d0-03    oravol01-dp01 c1t1d0  0        6990528  0         c1t1d0   ENA
pl oravol01-dp02 oravol01-d01 ENABLED ACTIVE   6990528  CONCAT    -        RW
sd c1t6d0-01    oravol01-dp02 c1t6d0  2096640  6990528  0         c1t6d0   ENA
pl oravol01-dp03 oravol01-d01 ENABLED ACTIVE   LOGONLY  CONCAT    -        RW
sd c1t3d0-01    oravol01-dp03 c1t3d0  0        528      LOG       c1t3d0   ENA

v  oravol01-d02 -            ENABLED  ACTIVE   6990528  SELECT    -        fsgen
pl oravol01-dp04 oravol01-d02 ENABLED ACTIVE   6990528  CONCAT    -        RW
sd c1t2d0-03    oravol01-dp04 c1t2d0  0        6990528  0         c1t2d0   ENA
pl oravol01-dp05 oravol01-d02 ENABLED ACTIVE   6990528  CONCAT    -        RW
sd c1t5d0-01    oravol01-dp05 c1t5d0  2100480  6990528  0         c1t5d0   ENA
pl oravol01-dp06 oravol01-d02 ENABLED ACTIVE   LOGONLY  CONCAT    -        RW
sd c1t3d0-02    oravol01-dp06 c1t3d0  528      528      LOG       c1t3d0   ENA

v  oravol01-d06 -            ENABLED  ACTIVE   6990528  SELECT    -        fsgen
pl oravol01-dp16 oravol01-d06 ENABLED ACTIVE   6990528  CONCAT    -        RW
sd c1t4d0-01    oravol01-dp16 c1t4d0  2096640  6990528  0         c1t4d0   ENA
pl oravol01-dp17 oravol01-d06 ENABLED ACTIVE   6990528  CONCAT    -        RW
sd c1t3d0-04    oravol01-dp17 c1t3d0  6991584  6990528  0         c1t3d0   ENA
pl oravol01-dp18 oravol01-d06 ENABLED ACTIVE   LOGONLY  CONCAT    -        RW
sd c1t3d0-03    oravol01-dp18 c1t3d0  1056     528      LOG       c1t3d0   ENA

Now a week or two goes by and your manager comes to you and tells you he wants to use concatenated-mirrored volumes since the back-end is already striped with EMC meta volumes. Since you have noted little to no benefit with host side striping, you set down Solaris Systems Programming, and fire up vxassist to convert the striped-mirrored volume to a concat-mirrored volume:

$ vxassist -g oof relayout oravol01 layout=concat-mirror

Once this operation completes, you have a shiny new concat-mirrored volume:

$ vxprint -hft
Disk group: oof

dg oof          default      default  21000    1120604922.22.tigger

dm c1t1d0       c1t1d0s2     auto     2048     35365968 -
dm c1t2d0       c1t2d0s2     auto     2048     35521408 -
dm c1t3d0       c1t3d0s2     auto     2048     35521408 -
dm c1t4d0       c1t4d0s2     auto     2048     35521408 -
dm c1t5d0       c1t5d0s2     auto     2048     35521408 -
dm c1t6d0       c1t6d0s2     auto     2048     35521408 -

v  oravol01     -            ENABLED  ACTIVE   20971520 SELECT    -        fsgen
pl oravol01-01  oravol01     ENABLED  ACTIVE   20971520 CONCAT    -        RW
sv oravol01-Ds03 oravol01-01 oravol01-d03 1    20971520 0         3/3      ENA

v  oravol01-d03 -            ENABLED  ACTIVE   20971520 SELECT    -        fsgen
pl oravol01-dp07 oravol01-d03 ENABLED ACTIVE   20971520 CONCAT    -        RW
sd c1t1d0-04    oravol01-dp07 c1t1d0  0        20971520 0         c1t1d0   ENA
pl oravol01-dp08 oravol01-d03 ENABLED ACTIVE   20971520 CONCAT    -        RW
sd c1t4d0-04    oravol01-dp08 c1t4d0  2880     20971520 0         c1t4d0   ENA
pl oravol01-dp09 oravol01-d03 ENABLED ACTIVE   LOGONLY  CONCAT    -        RW
sd c1t3d0-02    oravol01-dp09 c1t3d0  2097152  528      LOG       c1t3d0   ENA

Once the change is implemented, things run smoothly for a few weeks. Then one day your manager comes to you and tells you times are tight, and asks you to convert the mirrored-concatenated volume back to a RAID5 volume. Since this will save the company money (which will keep you employed), you set down the Veritas Volume Manager Troubleshooting Guide and fire up vxassist to perform the conversion:

$ vxassist -g oof relayout oravol01 layout=raid5 ncol=5

Once the relayout process completes, you have a nice 5-column RAID5 volume:

$ vxprint -hft
Disk group: oof

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         NVOLUME      KSTATE   STATE
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO

dg oof          default      default  21000    1120604922.22.tigger

dm c1t1d0       c1t1d0s2     auto     2048     35365968 -
dm c1t2d0       c1t2d0s2     auto     2048     35521408 -
dm c1t3d0       c1t3d0s2     auto     2048     35521408 -
dm c1t4d0       c1t4d0s2     auto     2048     35521408 -
dm c1t5d0       c1t5d0s2     auto     2048     35521408 -
dm c1t6d0       c1t6d0s2     auto     2048     35521408 -

v  oravol01     -            ENABLED  ACTIVE   20971520 RAID      -        raid5
pl oravol01-Dp02 oravol01    ENABLED  LOG      1600     CONCAT    -        RW
sd c1t5d0-02    oravol01-Dp02 c1t5d0  5242880  1600     0         c1t5d0   ENA
pl oravol01-03  oravol01     ENABLED  ACTIVE   20971520 RAID      5/32     RW
sd c1t1d0-03    oravol01-03  c1t1d0   0        5242880  0/0       c1t1d0   ENA
sd c1t5d0-03    oravol01-03  c1t5d0   0        5242880  1/0       c1t5d0   ENA
sd c1t6d0-02    oravol01-03  c1t6d0   0        5242880  2/0       c1t6d0   ENA
sd c1t3d0-03    oravol01-03  c1t3d0   2096640  5242880  3/0       c1t3d0   ENA
sd c1t4d0-03    oravol01-03  c1t4d0   2096640  5242880  4/0       c1t4d0   ENA

Now due to some budget tightening, your manager is let go, and a consultant is brought in to provide "performance" recommendations. After reviewing the system configuration, he recommends converting the RAID5 volume to a pure striped volume. Since this provides absolutely _NO_ protection for your data, you question his decision, but are forced to implement the change. Since you are a devoted company guy, you execute vxassist to convert the volume to a pure striped volume:

$ vxassist -g oof relayout oravol01 layout=striped ncol=6

Once this operation completes, you have a non-protected 6-column striped volume:

$ vxprint -hft
Disk group: oof

DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         NVOLUME      KSTATE   STATE
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO

dg oof          default      default  21000    1120604922.22.tigger

dm c1t1d0       c1t1d0s2     auto     2048     35365968 -
dm c1t2d0       c1t2d0s2     auto     2048     35521408 -
dm c1t3d0       c1t3d0s2     auto     2048     35521408 -
dm c1t4d0       c1t4d0s2     auto     2048     35521408 -
dm c1t5d0       c1t5d0s2     auto     2048     35521408 -
dm c1t6d0       c1t6d0s2     auto     2048     35521408 -

v  oravol01     -            ENABLED  ACTIVE   20971520 SELECT    oravol01-03 fsgen
pl oravol01-03  oravol01     ENABLED  ACTIVE   20971776 STRIPE    6/128    RW
sd c1t1d0-03    oravol01-03  c1t1d0   0        3495296  0/0       c1t1d0   ENA
sd c1t5d0-02    oravol01-03  c1t5d0   0        3495296  1/0       c1t5d0   ENA
sd c1t6d0-02    oravol01-03  c1t6d0   0        3495296  2/0       c1t6d0   ENA
sd c1t3d0-02    oravol01-03  c1t3d0   2096640  3495296  3/0       c1t3d0   ENA
sd c1t4d0-02    oravol01-03  c1t4d0   2096640  3495296  4/0       c1t4d0   ENA
sd c1t2d0-03    oravol01-03  c1t2d0   2096640  3495296  5/0       c1t2d0   ENA

Since this configuration makes you extremely nervous, you decide to update your resume, and eventually leave to take a position with a different company. After settling into your new job, you get a frantic call from the consultant at your previous job. He informs you that one of disks failed, and users are unable to get to the MySQL database. Being the nice guy you are, you explain how to create a redundant high performant volume with vxassist, and then walk him through using mysqlimport to restore the data that was being backed up with the mysqldump utility. You also tell him to stay tuned to this website for a future entry on recovery fundamentals (or the lack of them in this case!).

Conclusion

All of the operations described here were performed with an active MySQL database accessing the volume that was used in the examples. It is imperative to test all configuration changes in a non-production environment prior to implementing in production. While the relayout process should work flawlessly in other environments, I accept no responsibility for lost or corrupted data. If you have questions or comments on the article, please feel free to E-mail the author.