The importance of keeping your storage array firmware up to date

A couple of weeks back I attempted to migrate a pair of clustered Solaris 10 servers to a new disk storage array. After rebooting into single user mode to pick up the new devices, I went to add the new quorum disk with clquorum. This resulted in both nodes panicking with the following panic string:

panic[cpu3]/thread=fffffe800125bc60: Reservation Conflict
Disk: /scsi_vhci/disk@g6000d310002c6700000000000000003e

fffffe800125ba40 fffffffff7959e39 ()
fffffe800125ba70 sd:sd_pkt_status_reservation_conflict+c8 ()
fffffe800125bab0 sd:sdintr+431 ()
fffffe800125bb50 scsi_vhci:vhci_intr+3da ()
fffffe800125bb70 fcp:ssfcp_post_callback+4a ()
fffffe800125bba0 fcp:ssfcp_cmd_callback+4c ()
fffffe800125bc00 qlc:ql_task_thread+756 ()
fffffe800125bc40 qlc:ql_task_daemon+94 ()
fffffe800125bc50 unix:thread_start+8 ()

At first I thought I was doing something wrong, but after a lot of research I figured out that there were a couple of Solaris-related bugs in the version of the storage array firmware we were using. One of the bugs was triggering the panic above, and after the array was patched everything worked as expected. Keeping up to date with firmware is just as important as keeping up to date with OS patches. It’s amazing how many firmware bugs there are, and they bite you in the oddest ways.

Locating physical disk drives in Solaris

On “enterprise” Sun hardware, you can do nifty tricks like blink LED lights on disks to identify where logical disk names like c8t2d0 resides as Matty pointed out in the blog post here.

But what if you’re stuck on crufty (cheaper) regular SATA drives without the sexy LED support? How do you find c8t2d0 amongst a ton of other disks? Using cfgadm -alv, you can print out the serial number of the drive. The serial number of the drive is usually printed on the external area that is viable (hopefully) or on top of the disk itself. Then, you can go SN hunting amongst all the other disk in your array. Niiice!

$ cfgadm -alv
Ap_Id                          Receptacle   Occupant     Condition  Information
When         Type         Busy     Phys_Id
sata1/0::dsk/c8t0d0            connected    configured   ok         Mod: WDC WD800JD-75HKA1 FRev: 14.03G14 SN: WD-WMAJ95141282
unavailable  disk         n        /devices/pci@0,0/pci108e,534a@7:0
sata1/1::dsk/c8t1d0            connected    configured   ok         Mod: ST31000528AS FRev: CC37 SN: 9VP21P37
unavailable  disk         n        /devices/pci@0,0/pci108e,534a@7:1
sata2/0::dsk/c3t0d0            connected    configured   ok         Mod: WDC WD10EARS-00Z5B1 FRev: 80.00A80 SN: WD-WMAVU1311029

Update: iostat -En also shows this serial number info as well. I often use iostat -En to check for transport errors and it didn’t dawn on me to look for SN info here. Thanks Mark!

$ iostat -En
c8t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0 
Vendor: ATA      Product: WDC WD800JD-75HK Revision: 3G14 Serial No:  
Size: 80.00GB 
Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0 
Illegal Request: 9 Predictive Failure Analysis: 0 

Cool blog post about aligning partitions for performance

I came across a link to Dave Lutz’s Partition Alignment Guidelines for Unified Storage blog post while catching up on zfs-discuss today. Aligning partitions correctly can make a HUGE performance difference, and it’s amazing how many people are unaware of the consequences of not laying out their partitions to align with the storage they are using (this includes segment size and partition alignment). If you are new to this topic, you should definitely read through Dave’s post!

Configuring and monitoring the T5220 hardware RAID controller

The Sun T5220 comes with a built-in RAID controller, which supports all of the standard RAID levels (0 – 6). Configuring one or more devices to participate in a RAID Configuration is dead simple, since you can use the Solaris raidctl utility. The last T5220 I configured had a root file system that was going to reside on the built-in RAID controller, so I had to boot into single user mode to create my volume. To create a RAID1 volume using the devices c1t0d0 and c1t1d0 (you can get the devices via format or raidctl), you can run raidctl with the “-c” (create raid volume) option, and the names of the disks to mirror:

$ raidctl -c c1t0d0 c1t1d0

Creating RAID volume will destroy all data on spare space of member disks, proceed (yes/no)? yes
/pci@0/pci@0/pci@2/scsi@0 (mpt0):
        Physical disk 0 created.
/pci@0/pci@0/pci@2/scsi@0 (mpt0):
        Physical disk 1 created.
/pci@0/pci@0/pci@2/scsi@0 (mpt0):
        Volume 0 created.
/pci@0/pci@0/pci@2/scsi@0 (mpt0):
        Physical disk (target 1) is |out of sync||online|
/pci@0/pci@0/pci@2/scsi@0 (mpt0):
        Volume 0 is |enabled||degraded|
/pci@0/pci@0/pci@2/scsi@0 (mpt0):
        Volume 0 is |enabled||resyncing||degraded|

I also wanted to be able to use the cache on the RAID controller, which can be enabled using the raidctl “-p” (set property) option:

$ raidctl -p “wp=on” c1t0d0

Once I had a working RAID1 volume, I created a label on the device with fdisk and proceeded to perform a Solaris 10 installation. After the volume synchronized and Solaris was re-installed, I was able to run raidctl with the “-l” option to display the state of the volume:

$ raidctl -l c1t0d0

Volume                  Size    Stripe  Status   Cache  RAID
        Sub                     Size                    Level
c1t0d0                  136.6G  N/A     OPTIMAL  ON     RAID1
                0.0.0   136.6G          GOOD    
                0.1.0   136.6G          GOOD    

The raidctl utility is rather handy, and I created a checklsi script that can be run from cron to check the status of your RAID controllers (from some limited testing it appears FMA doesn’t detect disk faults).

Backing up your COMSTAR storage configuration

I have been playing around with the COMSTAR iSCSI and FC port providers for the past few months, and other than a number of problems with the emlxs driver, they appear to work pretty well. As I’ve been experimenting, I wanted to back up my configuration in case something happened to my server. COMSTAR uses SMF and the underlying block device to store configuration data, so you can use svccfg to backup the configuration:

$ svccfg export -a stmf > comstar.bak.${DATE}

If you ever need to restore the configuration, you can attach the storage and run an import:

$ svccfg import comstar.bak.${DATE}

COMSTAR has some serious potential, and I’m looking forward to seeing the project grow in 2010!

Port multiplier support in opensolaris

I just saw the following opensolaris putback notice:

PSARC/2009/394 SATA Framework Port Multiplier Support
6422924 sata framework has to support port multipliers
6691950 ahci driver needs to support SIL3726/4726 SATA port multiplier

This is awesome news, and I’m hopeful this will allow me to use one of my external SATA enclosures with OpenSolaris. Time to live upgrade my systems!