An MD device that went too far

Recently I was approached to help debug a problem with a Linux MD device that wouldn’t start. When I ran raidstart to start the device, it spit out a number of errors on the console, and messages similar to the following were written to the system log:

ide: failed opcode was: unknown
hda: read_intr: status=0x59 { DriveReady SeekComplete DataRequest Error }
hda: read_intr: error=0x10 { SectorIdNotFound }, LBAsect=120101895, sector=120101895

After pondering the error for a bit, it dawned on me that the partition table might be fubar. My theory proved correct, since partition six (the one associated with the md device that wouldn’t start) had an ending cylinder count (7476) that was greater than the number of physical cylinders (7294) on the drive:

$ fdisk -l /dev/hda

Disk /dev/hda: 60.0 GB, 60000000000 bytes
255 heads, 63 sectors/track, 7294 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1   *           1          32      257008+  fd  Linux raid autodetect
/dev/hda2              33          65      265072+  fd  Linux raid autodetect
/dev/hda3              66         327     2104515   fd  Linux raid autodetect
/dev/hda4             328        7476    57424342+   5  Extended
/dev/hda5             328         458     1052226   fd  Linux raid autodetect
/dev/hda6             459         720     2104483+  fd  Linux raid autodetect
/dev/hda7             721        7476    54267538+  fd  Linux raid autodetect

Once I corrected the end cyclinder and recreated the file system, everything worked as expected. Now I have no idea how the system got into this state (I didn’t build the system), since the installers I tested display errors when you specify an ending cylinder count that is larger than the maximum number of cylinders available.

Adding status support to the Solaris dd utility

I really dig Solaris, but it is missing a few basic features I have come to rely on. One of these features is the ability to send the dd utility a signal to get the status of a copy operation. Since Solaris is now opensource (or mostly opensource), I thought I would hack some code together to implement this feature. After a bit of coding (and testing), I requested a sponsor to putback my change into opensolaris. One was assigned, and I sent my changes to him to review. I haven’t heard back from him in the past month or two, so I reckon he’s too busy to help me putback my changes. In case others are interested in this feature, I placed a diff with my changes on my website.

Solaris fibre channel management

With the introduction of Solaris 10, storage management has changed considerably. The storage foundation kit is now integrated into the base OS, the leadville driver has been expanded to include HBAs from Emulex, JNI and Qlogic, and the fcinfo utility as well as several mdb DCMDS were added to view fibre channel connectivity information. fcinfo is especially useful, since it provides a tool with the base Operating System to view HBA and connectivity information.

The fcinfo utility has two main options. The first option, “hba-port,” will display HBA port speeds, WWPNs, the HBA manufacturer, and the state of the HBA:

$ fcinfo hba-port

HBA Port WWN: 10000000c9327592
        OS Device Name: /dev/cfg/c2
        Manufacturer: Emulex
        Model: LP9002L
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb
        Current Speed: 2Gb
        Node WWN: 20000000c9327592

HBA Port WWN: 10000000c9327593
        OS Device Name: /dev/cfg/c3
        Manufacturer: Emulex
        Model: LP9002L
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb
        Current Speed: 2Gb
        Node WWN: 20000000c9327593

If the “-l” option is specified, fcinfo will also print link statistics (e.g., link failure, signal loss, etc.):

$ fcinfo hba-port -l

HBA Port WWN: 10000000c9327592
        OS Device Name: /dev/cfg/c2
        Manufacturer: Emulex
        Model: LP9002L
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb
        Current Speed: 2Gb
        Node WWN: 20000000c9327592
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 5
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 0
                Invalid CRC Count: 0

HBA Port WWN: 10000000c9327593
        OS Device Name: /dev/cfg/c3
        Manufacturer: Emulex
        Model: LP9002L
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb
        Current Speed: 2Gb
        Node WWN: 20000000c9327593
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 5
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 0
                Invalid CRC Count: 0

The second fcinfo option is “remote-port,” which will display information on remote targets. This information includes the storage manufacturer, the storage product type, WWPNs, and all of the SCSI targets that have been presented to the host:

$ fcinfo remote-port -slp 10000000c9327592

Remote Port WWN: 50060160082006e2
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060160882006e2
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 255
                Invalid CRC Count: 0
        LUN: 0
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c4t6006016061B71000AD0810C9979CD911d0s2
        LUN: 1
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c4t6006016061B7100055B12704989CD911d0s2
Remote Port WWN: 50060168082006e2
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060160882006e2
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 255
                Invalid CRC Count: 0
        LUN: 0
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c4t6006016061B71000AD0810C9979CD911d0s2
        LUN: 1
          Vendor: DGC
          Product: RAID 5
          OS Device Name: /dev/rdsk/c4t6006016061B7100055B12704989CD911d0s2

These changes should make managing storage on Solaris a bit easier.

Concert review: Buckcherry and Saliva

This past week I continued my quest to see every band live, and ventured out to see Saliva and Buckcherry at a relatively small venue. The music started around 9pm, when the Saliva lead singer came to the stage with one of the coolest mohawks I have seen in quite some time (I think it was colored bright pink). The band wasted no time getting the crowd energized, and cranked out one hit after another. This included their mega hit “Ladies And Gentlemen,” as well as “Click Click Boom,” “Always,” “Your disease,” “Survival Of The Sickest,” “Broken,” and “Raise Up.” Each song was filled with passion and a lot of aggression, and you could tell the crowd was loving the music!!

After a short intermission, Buckcherry took the stage. Prior to attending the concert, I think I heard two or three of the band’s songs. I knew next to nothing about them, but being an open minded person, I gave them the chance to make me a Buckcherry fan. Well, the band was absolutely incredible, and the lead singer Josh Todd has an amazing stage presence (even though he didn’t talk all that much). The band performed for about an hour and a half, and played their classi hit “Crazy Bitch,” as well as “Everything” and “Lit Up.” I especially enjoyed “Lit Up,” since the band played it with an amazing level of intensity, which got the crowd into the song (I don’t have the best vocals, but I was singing along with everyone around me).

Now I don’t give many show the Matty two thumbs of approval, but I can safely say this show gets it. The music was hard, both bands played with the utmost intensity, and most importantly, the crowd was into each and every song. If either of these bands wanders back down to the south to perform again, I will be sitting front and center watching them crank out some more awesome music! Niiiiiiiiiiiiiiiiiiiiiiiiice!

iSCSI slides

I gave a presentation on the Solaris iSCSI stack at the Atlanta opensolaris users group meeting this evening. If your curious what was discussed, you can look through the slides that were used for the presentation. I would like to thank everyone for coming out, and for being such a great audience. It was a pleasure talking about this super cool topic with everyone!!!

Taking IPMP managed interfaces online and offline

I use Solaris IPMP (IP multipathing) on several of my servers to allow them to keep operating in the event that a network interface or switch were to fail. Periodically I need to take IPMP managed interfaces offline, but I need to keep the IP addresses attached to those interface up and operational. Solaris comes with the if_mpadm utility, which provides a simple and straight forward way to take IPMP managed interfaces online and offline.

Prior to using the if_mpadm utility, it is useful to check the status of the interface you want to take online or offline. This can be done by running the ifconfig utility, and checking the status of the interface you are interested in taking online or offline (in this case ni0):

$ ifconfig -a

lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000 
ni0: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 2
        inet 192.168.1.5 netmask ffffff00 broadcast 192.168.1.255
        groupname ipmp0
        ether 0:45:e8:33:3c:97 
ni1: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 3
        inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
        groupname ipmp0
        ether 0:72:b6:d3:ee:35 
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
        inet6 ::1/128  

To take the interface ni0 offline for maintenance, the if_mpadm utility can be run with the “-d” option (take interface offline), and the name of the interface to take offline:

$ if_mpadm -d ni0

Once if_mpadm does it’s job, the interface will be in the OFFLINE state, and the IP addresses attached to that interface will have migrated to another device in the IPMP group:

$ ifconfig -a

lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000 
ni0: flags=289000842<BROADCAST,RUNNING,MULTICAST,IPv4,NOFAILOVER,OFFLINE,CoS> mtu 0 index 2
        inet 0.0.0.0 netmask 0 
        groupname ipmp0
        ether 0:45:e8:33:3c:97 
ni1: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 3
        inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
        groupname ipmp0
        ether 0:72:b6:d3:ee:35 
ni1:1: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 3
        inet 192.168.1.5 netmask ffffff00 broadcast 192.168.1.255
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
        inet6 ::1/128 

After you finish your maintenance, you can use the if_mpadm “-r” option (bring interface online) to bring the interface online:

$ if_mpadm -r ni0

Once if_mpadm completes, you can use the ifconfig utility to verify the interface is back up, and the IP addresses have migrated back to the original adaptor (you can disable automatic failback by setting FAILBACK to no in /etc/default/mpathd):

$ ifconfig -a

lo0: flags=2001000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4,VIRTUAL> mtu 8232 index 1
        inet 127.0.0.1 netmask ff000000 
ni0: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 2
        inet 192.168.1.5 netmask ffffff00 broadcast 192.168.1.255
        groupname ipmp0
        ether 0:45:e8:33:3c:97 
ni1: flags=201000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4,CoS> mtu 1500 index 3
        inet 0.0.0.0 netmask ff000000 broadcast 0.255.255.255
        groupname ipmp0
        ether 0:72:b6:d3:ee:35 
lo0: flags=2002000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv6,VIRTUAL> mtu 8252 index 1
        inet6 ::1/128 

IPMP is pretty neat, and the next major release has some impressive features (I especially dig ipmpstat)!