Expanding VMWare guest storage and extending LVM volumes and ext3 file systems to use the new storage

One of my production vSphere virtual machines came close to running out of space this past week. Expanding guest storage with vSphere is a breeze, and I wanted to jot down my notes in this blog post. To expand the size of a virtual disk, you will need to open the virtual infrastructure client, right click your guest and click “Edit Settings”. Once the virtual machine settings window opens, you will need to navigate to the disk that you want to expand. On the right side of the screen, you can click in the “Provisioned Size” text box and specify the new size of the virtual disk:


null

Now, assuming this is a Linux host, you can run fdisk to see the current size of the disk (replace /dev/sdb with the device that corresponds to the virtual disk you expanded):

$ fdisk -l /dev/sdb

Disk /dev/sdb: 19.3 GB, 19327352832 bytes
255 heads, 63 sectors/track, 2349 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn't contain a valid partition table

Fdisk still shows the disk as 18GB in size, so we will need to rescan the SCSI bus to pick up the changes:

$ /usr/bin/rescan-scsi-bus.sh -r -i

Host adapter 0 (mptspi) found.
Scanning SCSI subsystem for new devices
 and remove devices that have disappeared
Scanning host 0 for  SCSI target IDs  0 1 2 3 4 5 6 7, all LUNs
Scanning for device 0 0 0 0 ...
OLD: Host: scsi0 Channel: 00 Id: 00 Lun: 00
      Vendor: VMware   Model: Virtual disk     Rev: 1.0
      Type:   Direct-Access                    ANSI SCSI revision: 02
Scanning for device 0 0 1 0 ...
OLD: Host: scsi0 Channel: 00 Id: 01 Lun: 00
      Vendor: VMware   Model: Virtual disk     Rev: 1.0
      Type:   Direct-Access                    ANSI SCSI revision: 02
0 new device(s) found.
0 device(s) removed.

If the rescan completes without issue, the new space should be available:

$ fdisk -l /dev/sdb

Disk /dev/sdb: 38.6 GB, 38654705664 bytes
255 heads, 63 sectors/track, 4699 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/sdb doesn't contain a valid partition table

This is rad, though the physical volume, logical volume and file system know nothing about the new space. To expand the physical volume to make use of the space, we can run pvresize:

$ pvresize -v /dev/sdb

    Using physical volume(s) on command line
    Archiving volume group "data_vg" metadata (seqno 4).
    No change to size of physical volume /dev/sdb.
    Resizing volume "/dev/sdb" to 75497088 sectors.
    Updating physical volume "/dev/sdb"
    Creating volume group backup "/etc/lvm/backup/data_vg" (seqno 5).
  Physical volume "/dev/sdb" changed
  1 physical volume(s) resized / 0 physical volume(s) not resized

$ pvdisplay /dev/sdb

  --- Physical volume ---
  PV Name               /dev/sdb
  VG Name               data_vg
  PV Size               36.00 GB / not usable 3.81 MB
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              9215
  Free PE               0
  Allocated PE          9215
  PV UUID               dNSA1X-KxHX-g0kq-ArpY-umVO-buN3-NrPoAt

As you can see above, the physical volume is now 36GB in size. To expand the logical volume that utilizes this physical volume, we can run lvextend with the physical extent values listed in the pvdisplay output:

$ lvextend -l 9215 /dev/data_vg/datavol01

  Extending logical volume datavol01 to 36.00 GB
  Logical volume datavol01 successfully resized

$ lvdisplay /dev/data_vg/datavol01

  --- Logical volume ---
  LV Name                /dev/data_vg/datavol01
  VG Name                data_vg
  LV UUID                E2RB58-4Vac-tKSA-vmnW-nqL5-Sr0N-IR4lum
  LV Write Access        read/write
  LV Status              available
  # open                 0
  LV Size                36.00 GB
  Current LE             9215
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:3

Sweet! Now that the logical volume is 36GB in size, we can use the resize2fs command to increase the size of the ext3 file system that resides on the logical volume:

$ resize2fs /dev/data_vg/datavol01

resize2fs 1.39 (29-May-2006)
Filesystem at /dev/data_vg/datavol01 is mounted on /datavol01; on-line resizing required
Performing an on-line resize of /dev/data_vg/datavol01 to 9436160 (4k) blocks.

The resize2fs utility will take a few minutes to run, and should tell you that it is increasing the size of the file system to some number of blocks. If this step completes without error, you should see the new space in the df output (this assumes the file system is mounted):

$ df -h /datavol01

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/data_vg-datavol01
                       36G   17G   18G  49% /datavol01

Using the steps listed above, I was able to double the size the virtual disk attached to my virtual machine. While this procedure should work flawlessly, I provide zero guarantees that it will. Use this procedure at your own risk, and make sure to test it out in a non-production environment before doing ANYTHING with your production hosts!! I really really dig vSphere, and hope to blog about it a ton in the near future.

Building clusters with shared disk using VMWare server 2.0

I have a couple of lab machines that are running VMWare server 2.X under 64-bit CentOS 5.2. VMWare server has a cool feature where you can create “clusters in a box.” The cluster in a box feature allows you to share a virtual disk between more than one virtual machine, and since it support SCSI persistent reservations, you can truly simulate a real cluster. I have used this to deploy Oracle RAC, Solaris Cluster and Redhat Cluster server in my lab environment.

Based on the cluster in a box documentation on the VMWare website, sharing a disk between multiple nodes can be achieved by creating a virtual disk, and then importing that disk into each virtual machine. Once the disk is imported, you can add (or modify if they exist) the “scsiX.sharedBus = virtual” (X should match the controller that is used to host the shared disk) and “disk.locking = false” directives to the shared disk stanzas in each virtual machines vmx file. Here is a sample from a node I set up this weekend:

scsi1.present = “TRUE”
scsi1.sharedBus = “virtual”
scsi1.virtualDev = “lsilogic”
scsi1:0.present = “TRUE”
scsi1:0.fileName = “/data/vmware/shared/solariscluster/Disk1.vmdk”
disk.locking = “false”

This has served me well in my lab environment for quite some time, though I provide ZERO guarantees that it will work in yours (and there is always the possibility that sharing disk between multiple nodes will corrupt data). I am documenting this here as a reference for myself, though feel free to use this information at your own risk.

Getting VMWare server installed on a Fedora Core 6 desktop

I came across two bizarre issues last week while installing VMWare server on a Fedora Core 6 desktop. The first issue occurred when the vmware-config.pl script attempted to build the vmmon kernel module:

$ vmware-config.pl

Making sure services for VMware Server are stopped.

Stopping VMware services:
   Virtual machine monitor                                 [  OK  ]

Configuring fallback GTK+ 2.4 libraries.

In which directory do you want to install the mime type icons? 
[/usr/share/icons] 

What directory contains your desktop menu entry files? These files have a 
.desktop file extension. [/usr/share/applications] 

In which directory do you want to install the application's icon? 
[/usr/share/pixmaps] 

Trying to find a suitable vmmon module for your running kernel.

None of the pre-built vmmon modules for VMware Server is suitable for your 
running kernel.  Do you want this program to try to build the vmmon module for 
your system (you need to have a C compiler installed on your system)? [yes] 

Using compiler "/usr/bin/gcc". Use environment variable CC to override.

What is the location of the directory of C header files that match your running
kernel? [/usr/src/kernels/2.6.18-1.2869.fc6-i686/include] 

Extracting the sources of the vmmon module.

Building the vmmon module.

Using 2.6.x kernel build system.
make: Entering directory `/tmp/vmware-config0/vmmon-only'
make -C /usr/src/kernels/2.6.18-1.2869.fc6-i686/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. mo
dules
make[1]: Entering directory `/usr/src/kernels/2.6.18-1.2869.fc6-i686'
  CC [M]  /tmp/vmware-config0/vmmon-only/linux/driver.o
  CC [M]  /tmp/vmware-config0/vmmon-only/linux/hostif.o
  CC [M]  /tmp/vmware-config0/vmmon-only/common/cpuid.o
  CC [M]  /tmp/vmware-config0/vmmon-only/common/hash.o
  CC [M]  /tmp/vmware-config0/vmmon-only/common/memtrack.o
  CC [M]  /tmp/vmware-config0/vmmon-only/common/phystrack.o
  CC [M]  /tmp/vmware-config0/vmmon-only/common/task.o
  CC [M]  /tmp/vmware-config0/vmmon-only/common/vmx86.o
  CC [M]  /tmp/vmware-config0/vmmon-only/vmcore/moduleloop.o
  LD [M]  /tmp/vmware-config0/vmmon-only/vmmon.o
  Building modules, stage 2.
  MODPOST
  CC      /tmp/vmware-config0/vmmon-only/vmmon.mod.o
  LD [M]  /tmp/vmware-config0/vmmon-only/vmmon.ko
make[1]: Leaving directory `/usr/src/kernels/2.6.18-1.2869.fc6-i686'
cp -f vmmon.ko ./../vmmon.o
make: Leaving directory `/tmp/vmware-config0/vmmon-only'
Unable to make a vmmon module that can be loaded in the running kernel:
insmod: error inserting '/tmp/vmware-config0/vmmon.o': -1 Invalid module format
There is probably a slight difference in the kernel configuration between the 
set of C header files you specified and your running kernel.  You may want to 
rebuild a kernel based on that directory, or specify another directory.

For more information on how to troubleshoot module-related problems, please 
visit our Web site at "http://www.vmware.com/download/modules/modules.html" and
"http://www.vmware.com/support/reference/linux/prebuilt_modules_linux.html".

Execution aborted.

After a bit of poking around in /tmp/vmware-config0/vmmon-only, I figured out that I didn’t have the correct architecture of the redhat kernel source installed (my system had the i686 kernel development package instead of the i586 kernel development package). Once I removed the kernel-devel-i686 package and added the kernel-devel-i586 package, the kernel module built fine, but the vmnet module failed to build:

$ vmware-config.pl

< ..... >

Building the vmnet module.

Using 2.6.x kernel build system.
make: Entering directory `/tmp/vmware-config1/vmnet-only'
make -C /lib/modules/2.6.18-1.2869.fc6/build/include/.. SUBDIRS=$PWD SRCROOT=$PWD/. modul
es
make[1]: Entering directory `/usr/src/kernels/2.6.18-1.2869.fc6-i586'
  CC [M]  /tmp/vmware-config1/vmnet-only/driver.o
  CC [M]  /tmp/vmware-config1/vmnet-only/hub.o
  CC [M]  /tmp/vmware-config1/vmnet-only/userif.o
  CC [M]  /tmp/vmware-config1/vmnet-only/netif.o
  CC [M]  /tmp/vmware-config1/vmnet-only/bridge.o
  CC [M]  /tmp/vmware-config1/vmnet-only/procfs.o
/tmp/vmware-config1/vmnet-only/procfs.c:33:26: error: linux/config.h: No such file or dir
ectory
make[2]: *** [/tmp/vmware-config1/vmnet-only/procfs.o] Error 1
make[1]: *** [_module_/tmp/vmware-config1/vmnet-only] Error 2
make[1]: Leaving directory `/usr/src/kernels/2.6.18-1.2869.fc6-i586'
make: *** [vmnet.ko] Error 2
make: Leaving directory `/tmp/vmware-config1/vmnet-only'
Unable to build the vmnet module.

For more information on how to troubleshoot module-related problems, please 
visit our Web site at "http://www.vmware.com/download/modules/modules.html" and
"http://www.vmware.com/support/reference/linux/prebuilt_modules_linux.html".

Execution aborted.

After a bit of googling, it turns out that one of the headers is reliant on a deprecated configuration header. To fix this issue, I had to run vmware-config.pl again after touching config.h in the include directory of the kernel source that matched my running kernel (in this case, 2.6.18-1.2869.fc6-i586):

$ cd /usr/src/kernels/2.6.18-1.2869.fc6-i586/include/linux

$ touch config.h

Once I addressed these two issues, VMWare server ran perfectly!

Getting ESX server to recognize Clariion devices

While setting up two new ESX 3.0 server nodes, I ran into a bizarre problem where the VI client refused to initialize several Clariion CX700 devices. Since the VI client isn’t the best environment to debug problems, I ssh’ed into the service console, and began my research by running esxcfg-vmhbadevs to list the devices on the system:

$ esxcfg-vmhbadevs -q
vmhba0:0:0 /dev/sda
vmhba1:0:0 /dev/sdc
vmhba1:0:1 /dev/sdd
vmhba1:0:2 /dev/sde
vmhba1:0:3 /dev/sdf
vmhba1:0:4 /dev/sdg
vmhba1:0:5 /dev/sdh
vmhba1:0:6 /dev/sdi
vmhba1:0:7 /dev/sdj
vmhba1:0:8 /dev/sdk
vmhba1:0:9 /dev/sdl

This listed the correct number of devices, so I decided to use dd to read data from one of the devices that was causing problems:

$ dd if=/dev/sdl of=/dev/null
195728+0 records in
195728+0 records out

Dd worked fine, but I was still unable to create a file system with the VI client or vmkfstools:

$ vmkfstools -C vmfs3 /vmfs/devices/disks/vmhba1:0:0:9
Error: Bad file descriptor

To get a better idea of the disk device layout. I decided to run fdisk to list the label type and layout of each device:

$ fdisk -l

< ..... >

Disk /dev/sdl (Sun disk label): 256 heads, 10 sectors, 40958 cylinders
Units = cylinders of 2560 * 512 bytes

   Device Flag    Start       End    Blocks   Id  System
/dev/sdl3  u          0     40958  52426240    5  Whole disk
/dev/sdl4  u          0         1      1280    f  Unknown
/dev/sdl5  u          2     40958  52423680    e  Unknown

< ..... >

Hmmm — each Clariion device contained an SMI label, and I started to wonder if this was causing the problem. To test my hypothesis, I used the fdisk utility to write a DOS label to one of the problematic devices:

$ fdisk /dev/sdl

Command (m for help): o
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.

The number of cylinders for this disk is set to 40958.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

Once I labeled the device with a DOS label, vmkfstools and the VI client allowed me to initialize and use the device. I am not certain why ESX server was having an issue with the SMI label, but writing a DOS label looks to have fixed my problem.

Viewing VMWare ESX server configuration data from the command line

While poking around the ESX 3.0 service console last week, I came across the amazingly useful esxcfg-info utility. This utility can be used to extract statistics and configuration data for the network devices, hardware and storage that are attached to a server. To view the network configuration, you can invoke esxcfg-info with the “-n” option:

$ esxcfg-info -n |more

+NetworkInfoImpl :
   \==+Network Entities:
      \==+ConsoleNicInfoImpl :
         \==+Console Nics:
            \==+VnicImpl :
               |----Name............................................vswif0
               |----Port Group......................................portgroup0
               |----Port Id.........................................16777219
               |----Enabled.........................................true
               |----Ip Address......................................10.1.1.50
               |----Ip Netmask......................................255.255.255.0
               |----Ip Broadcast....................................10.1.1.255
               |----DHCP............................................false
               |----MAC Address.....................................00:50:56:48:2e:d3
               |----MTU.............................................1500
      \==+DnsConfig :
         |----Hostname..............................................foo01
         |----Domain................................................
         |----Search List...........................................
         |----Primary Name Server...................................10.1.1.28
         |----Secondary Name Server.................................10.1.1.26
      \==+RoutingInfoImpl :
         |----Kernel Gateway........................................10.1.1.1
         |----Console Gateway.......................................10.1.1.1
         |----Console Gateway Device................................vswif0

          [ ..... ]

To view the hardware present in a server, esxcfg-info can be invoked with the “-w” option:

$ esxcfg-info -w |more

+HardwareInfoImpl :
   |----BIOS UUID...................................................0x66 0x9d 0x71 0xf4 0xaa 0x38 0x11 0xd9 0xae 0x6c 0x0 0x30 
0x6e 0xc 0x67 0x4d
   \==+PciInfoImpl :
      \==+All Pci Devices:
         \==+PciDeviceImpl :
            |----Bus................................................0x00
            |----Slot...............................................0x06
            |----Function...........................................0x00
            |----Vendor Id..........................................0x1022
            |----Device Id..........................................0x7460
            |----Sub-Vendor Id......................................0x0000
            |----Sub-Device Id......................................0x0000
            |----Vendor Name........................................Advanced Micro Devices [AMD]
            |----Device Name........................................AMD-8111 PCI
            |----Device Class.......................................1540
            |----PIC Line...........................................255
            |----Service Console IRQ................................255
            |----Vector.............................................0
            |----PCI Pin............................................0
            |----Spawned Bus........................................1
            |----Flags..............................................258
            |----Module Id..........................................0
            |----Chassis............................................0
            |----Physical Slot......................................0
            |----VmKernel Device Name...............................
          
            [ ..... ]

And for all of the storage folks out there, esxcfg-info provides the “-s” option to print the storage configuration:

$ esxcfg-info -s |more

+StorageInfoImpl :
   \==+DiagnosticPartition :
   \==+DiskLunPartition :
      |----Name.....................................................vmhba0:0:0:7
      |----Start Sector.............................................143155278
      |----End Sector...............................................143364060
      |----Partition Type...........................................252
      |----Console Device.........................................../dev/sda7
      |----Size.....................................................106896384
      \==+Scsi Stats :
         |----Commands : ...........................................0
         |----Blocks Read: .........................................0
         |----Blocks Written: ......................................0
         |----Aborts: ..............................................0
         |----Resets: ..............................................0
         |----Read Operations: .....................................0
         |----Write Operations: ....................................0
         |----PAE commands: ........................................0
         |----PAE copies: ..........................................0
         |----Split commands: ......................................0
         |----Split copies: ........................................0
         |----Issue Time: ..........................................0
         |----Total Time: ..........................................0
         |----IsActive.................................................true
    
         [ ..... ]

There are some other useful options, but I will let folks investigate those when they play around with this super useful utility.