While setting up two new ESX 3.0 server nodes, I ran into a bizarre problem where the VI client refused to initialize several Clariion CX700 devices. Since the VI client isn’t the best environment to debug problems, I ssh’ed into the service console, and began my research by running esxcfg-vmhbadevs to list the devices on the system:
$ esxcfg-vmhbadevs -q
vmhba0:0:0 /dev/sda
vmhba1:0:0 /dev/sdc
vmhba1:0:1 /dev/sdd
vmhba1:0:2 /dev/sde
vmhba1:0:3 /dev/sdf
vmhba1:0:4 /dev/sdg
vmhba1:0:5 /dev/sdh
vmhba1:0:6 /dev/sdi
vmhba1:0:7 /dev/sdj
vmhba1:0:8 /dev/sdk
vmhba1:0:9 /dev/sdl
This listed the correct number of devices, so I decided to use dd to read data from one of the devices that was causing problems:
$ dd if=/dev/sdl of=/dev/null
195728+0 records in
195728+0 records out
Dd worked fine, but I was still unable to create a file system with the VI client or vmkfstools:
$ vmkfstools -C vmfs3 /vmfs/devices/disks/vmhba1:0:0:9
Error: Bad file descriptor
To get a better idea of the disk device layout. I decided to run fdisk to list the label type and layout of each device:
$ fdisk -l
< ..... >
Disk /dev/sdl (Sun disk label): 256 heads, 10 sectors, 40958 cylinders
Units = cylinders of 2560 512 bytes
Device Flag Start End Blocks Id System
/dev/sdl3 u 0 40958 52426240 5 Whole disk
/dev/sdl4 u 0 1 1280 f Unknown
/dev/sdl5 u 2 40958 52423680 e Unknown
Hmmm – each Clariion device contained an SMI label, and I started to wonder if this was causing the problem. To test my hypothesis, I used the fdisk utility to write a DOS label to one of the problematic devices:
$ fdisk /dev/sdl
Command (m for help): o
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
The number of cylinders for this disk is set to 40958.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
Once I labeled the device with a DOS label, vmkfstools and the VI client allowed me to initialize and use the device. I am not certain why ESX server was having an issue with the SMI label, but writing a DOS label looks to have fixed my problem.