Blog O’ Matty

Eric Schrock has done some really cool work with integrating disk (SMART) /platform monitoring (IPMI) information into Opensolaris.   Just recently, he has extended FMA with a new technology called SES (SCSI Enclosure Services) into build 93 of OpenSolaris.

This looks like some really cool stuff.  The following was taken directly from his blog on the examples of using the new fmtopo utility to map out an external storage array.

# /usr/lib/fm/fmd/fmtopo
...

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:serial=2029QTF0000000002:part=Storage-J4400:revision=3R13/ses-enclosure=0

hc://:product-id=SUN-Storage-J4400:chassis-id=22029QTF0809QCK012:server-id=:part=123-4567-01/ses-enclosure=0/psu=0

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=:part=123-4567-01/ses-enclosure=0/psu=1

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=/ses-enclosure=0/fan=0

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=/ses-enclosure=0/fan=1

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=/ses-enclosure=0/fan=2

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=/ses-enclosure=0/fan=3

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=:serial=2029QTF0811RM0386:part=375-3584-01/ses-enclosure=0/controller=0

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=:serial=2029QTF0811RM0074:part=375-3584-01/ses-enclosure=0/controller=1

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=/ses-enclosure=0/bay=0

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=:serial=5QD0PC3X:part=SEAGATE-ST37500NSSUN750G-0720A0PC3X:revision=3.AZK/ses-enclosure=0/bay=0/disk=0

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=/ses-enclosure=0/bay=1

...

# fmtopo -V '*/ses-enclosure=0/bay=0/disk=0'
TIME                 UUID
Jul 14 03:54:23 3e95d95f-ce49-4a1b-a8be-b8d94a805ec8

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=:serial=5QD0PC3X:part=SEAGATE-ST37500NSSUN750G-0720A0PC3X:revision=3.AZK/ses-enclosure=0/bay=0/disk=0
  group: protocol                       version: 1   stability: Private/Private
    resource          fmri      hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=:serial=5QD0PC3X:part=SEAGATE-ST37500NSSUN750G-0720A0PC3X:revision=3.AZK/ses-enclosure=0/bay=0/disk=0
    ASRU              fmri      dev:///:devid=id1,sd@TATA_____SEAGATE_ST37500NSSUN750G_0720A0PC3X_____5QD0PC3X____________//scsi_vhci/disk@gATASEAGATEST37500NSSUN750G0720A0PC3X5QD0PC3X
    label             string    SCSI Device  0
    FRU               fmri      hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=:serial=5QD0PC3X:part=SEAGATE-ST37500NSSUN750G-0720A0PC3X:revision=3.AZK/ses-enclosure=0/bay=0/disk=0
  group: authority                      version: 1   stability: Private/Private
    product-id        string    SUN-Storage-J4400
    chassis-id        string    2029QTF0809QCK012
    server-id         string
  group: io                             version: 1   stability: Private/Private
    devfs-path        string    /scsi_vhci/disk@gATASEAGATEST37500NSSUN750G0720A0PC3X5QD0PC3X
    devid             string    id1,sd@TATA_____SEAGATE_ST37500NSSUN750G_0720A0PC3X_____5QD0PC3X____________
    phys-path         string[]  [ /pci@0,0/pci10de,377@a/pci1000,3150@0/disk@1c,0 /pci@0,0/pci10de,375@f/pci1000,3150@0/disk@1c,0 ]
  group: storage                        version: 1   stability: Private/Private
    logical-disk      string    c0tATASEAGATEST37500NSSUN750G0720A0PC3X5QD0PC3Xd0
    manufacturer      string    SEAGATE
    model             string    ST37500NSSUN750G 0720A0PC3X
    serial-number     string    5QD0PC3X
    firmware-revision string       3.AZK
    capacity-in-bytes string    750156374016

With the following putback:

PSARC 2006/373 Dynamic Lun Expansion
6241086 Format should allow label adjustment when disk/lun size changes
6430818 Solaris needs mechanism of dynamically increasing lun size

OpenSoaris now has the ability to detect and use space that is dynamically allocated to a LUN. This has been a bit of a stumbling block in the past, and the available solutions to expand storage weren’t for the faint of heart. Hopefully this feature works as advertised!!

In a SAN environment when dealing with external storage concepts such as EMC BCV’s, you’ll often have a request to create volumes on two different machines that are identical so replication on the back-end can occur.

 

When you look at a LUN presented to Solaris, it’ll appear with a cryptic name like the following:

 103. c20t60060480000190100665533030393836d0 <EMC-SYMMETRIX-5771 cyl 36826 alt 2 hd 60 sec 128>
          /scsi_vhci/ssd@g60060480000190100665533030353339

The c20 relates to the HBA (Fiber, SCSI, iSCSI) that provides a path to the device.  The “middle” sequence 60060480000190100665533030393836 between the “t (target)” and “d (device” is the WWN of the LUN.

 

Now, say your SAN engineer approaches you with some information like the following….

PROD DEV: 936

PROD LUN: 47

STAGE DEV: 986

STAGE LUN: 68

 

Ok… so what does that mean to us?  Using luxadm, we can probe a target to find out some specifc information about its back-end LUN name.

(root)> luxadm display /dev/rdsk/c20t60060480000190100665533030393836d0s2
DEVICE PROPERTIES for disk: /dev/rdsk/c20t60060480000190100665533030393836d0s2
  Vendor:               EMC
  Product ID:           SYMMETRIX
  Revision:             5771
  Serial Num:           xxxxxxxxxxx
  Unformatted capacity: 138105.000 MBytes
  Read Cache:           Enabled
    Minimum prefetch:   0×0
    Maximum prefetch:   0xffff
  Device Type:          Disk device
  Path(s):

  /dev/rdsk/c20t60060480000190100665533030393836d0s2
  /devices/scsi_vhci/ssd@g60060480000190100665533030393836:c,raw
   Controller           /devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0
    Device Address              50060482d52d2e76,68
    Host controller port WWN    210100e08ba0147a
    Class                       primary
    State                       ONLINE
   Controller           /devices/pci@17c,600000/SUNW,qlc@1,1/fp@0,0
    Device Address              50060482d52d2e59,68
    Host controller port WWN    210100e08ba0da73
    Class                       primary
    State                       ONLINE

The Device Address is the field we are interested in.  This shows us that the WWN of the Port we are plugged into is 50060482d52d2e59 and the LUN number is 68.

Now that we have the LUN number, we know what this LUN maps to.  We can then find the cooresponding LUN on the other machine (LUN Number 47) which maps to the BCV pair.

 

There are some other useful leadville commands that you may be interested in…

To display all HBAs available for use:

(root)> luxadm -e port
/devices/pci@15c,600000/SUNW,qlc@1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl                 CONNECTED
/devices/pci@15d,600000/SUNW,qlc@1,1/fp@0,0:devctl                 NOT CONNECTED
/devices/pci@15d,600000/SUNW,qlc@1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@17c,600000/SUNW,qlc@1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@17c,600000/SUNW,qlc@1,1/fp@0,0:devctl                 CONNECTED
/devices/pci@17d,600000/SUNW,qlc@1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@17d,600000/SUNW,qlc@1,1/fp@0,0:devctl                 NOT CONNECTED
/devices/pci@19c,600000/SUNW,qlc@1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@19c,600000/SUNW,qlc@1,1/fp@0,0:devctl                 NOT CONNECTED
/devices/pci@19d,600000/SUNW,qlc@1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@19d,600000/SUNW,qlc@1,1/fp@0,0:devctl                 NOT CONNECTED

Now that you have the device name, you can map that back to what device it cooralates to under /dev.

In this case since i’m using a Fiber channel HBA…

(root)> ls -l /dev/fc | grep /devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl
lrwxrwxrwx   1 root     root          55 Sep 26  2007 fp1 -> ../../devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl

Sweet.  So one of my HBAs is fp1.

Want to see more detailed information about what that Fiber HBA is connected to?

umt1a-bio-stg1:~
(root)> luxadm -e dump_map /devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl
Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type
0    611813  0         50060e8004769000 50060e8004769000 0×0  (Disk device)
1    624613  0         50060482d52d2e76 50060482d52d2e76 0×0  (Disk device)
2    617c13  0         210100e08ba0147a 200100e08ba0147a 0×1f (Unknown Type,Host Bus Adapter)

Note that the Port WWN 50060482d52d2e76 is the same WWN we saw above when looking for the LUN number. 

 

Want a dump of all LUNs attached to a controller?

(root)> cfgadm -o show_FCP_dev -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c4::50060482d52d2e76,0         disk         connected    configured   unknown
c4::50060482d52d2e76,69        disk         connected    configured   unknown
c4::50060482d52d2e76,70        disk         connected    configured   unknown
c4::50060482d52d2e76,71        disk         connected    configured   unknown
c4::50060482d52d2e76,72        disk         connected    configured   unknown
c4::50060482d52d2e76,73        disk         connected    configured   unknown
c4::50060482d52d2e76,74        disk         connected    configured   unknown
c4::50060482d52d2e76,75        disk         connected    configured   unknown
c4::50060482d52d2e76,76        disk         connected    configured   unknown
c4::50060482d52d2e76,77        disk         connected    configured   unknown
c4::50060482d52d2e76,78        disk         connected    configured   unknown
c4::50060482d52d2e76,79        disk         connected    configured   unknown
c4::50060482d52d2e76,80        disk         connected    configured   unknown

<snip>

….

Another super useful utility is the new fcinfo command which was introduced into Solaris 10.  The -l (linkstat) shows some detailed statistics on the HBA.

(root)> fcinfo hba-port -l
HBA Port WWN: 210100e08ba0147a
        OS Device Name: /dev/cfg/c4
        Manufacturer: QLogic Corp.
        Model: 375-3108-xx
        Firmware Version: 3.3.26
        FCode/BIOS Version:  fcode: 1.13;
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb
        Current Speed: 2Gb
        Node WWN: 200100e08ba0147a
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 0
                Invalid CRC Count: 0

        

With the WWN of the HBA, we can query the remote port information…

(root)> fcinfo remote-port -p 210100e08ba0147a
Remote Port WWN: 50060482d52d2e76
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060482d52d2e76
Remote Port WWN: 50060e8004769000
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060e8004769000

Check it out!  Its the same 50060482d52d2e76 we saw twice before with the WWN of the port we’re plugged into the fabric with.

 

Throw in a -s, and it’ll return all SCSI targets with their LUN Number.

(root)> fcinfo remote-port -p 210100e08ba0147a -s
Remote Port WWN: 50060482d52d2e76
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060482d52d2e76
        LUN: 0
          Vendor: EMC
          Product: SYMMETRIX
          OS Device Name: /dev/rdsk/c4t50060482D52D2E76d0s2
        LUN: 69
          Vendor: EMC
          Product: SYMMETRIX
          OS Device Name: /dev/rdsk/c20t60060480000190100665533030344539d0s2
        LUN: 70
          Vendor: EMC
          Product: SYMMETRIX
          OS Device Name: /dev/rdsk/c20t60060480000190100665533030344639d0s2
        LUN: 71
          Vendor: EMC
          Product: SYMMETRIX
          OS Device Name: /dev/rdsk/c20t60060480000190100665533030353039d0s2
        LUN: 72
          Vendor: EMC
          Product: SYMMETRIX
          OS Device Name: /dev/rdsk/c20t60060480000190100665533030353139d0s2
        LUN: 73
          Vendor: EMC
          Product: SYMMETRIX
          OS Device Name: /dev/rdsk/c20t60060480000190100665533030353239d0s2

 
Want to force a specific Fiber HBA to reinitialized and re-login into the Fabric?

(root)> luxadm -e forcelip <device>

i.e.

(root)> luxadm -e forcelip /devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl

A number of opensolaris communities have asked their members for feedback, and the list of technologies they would like to see added in the future. The storage community received a ton of feedback when they asked the community for the list of features they would like to have added to opensolaris, and this feedback was recently posted to the genunix wiki. If there are features you are interested in that don’t appear on the list, I would highly recommend adding them. There are so many cool things underway (Comstar, CIFS client and server, NPIV support, pNFS, better remote replication, etc.) in the storage community, and it’s awesome to see the storage project leaders reaching out to the community to get their ideas!

A while back I wrote an article titled dynamically growing a Clariion LUN with Solaris. In the article I described how to update the VTOC on a UN that was resized on the storage array. One of my colleagues came to me a few weeks back and told me the procedure was not working, and he couldn’t assign a new size (200GB in this case) to the device in format.

This made me curious, so I started poking around. To see what Solaris thought the actual size of the LUN was, I ran luxadm with the “display” option:

$ luxadm display /dev/rdsk/c5t600601602B9318008CD071B0FF7BDB11d0s2

DEVICE PROPERTIES for disk: /dev/rdsk/c5t600601602B9318008CD071B0FF7BDB11d0s2
  Vendor:               DGC
  Product ID:           RAID 5
  Revision:             0322
  Serial Num:           APM000612076
  Unformatted capacity: 204800.000 MBytes
  Read Cache:           Enabled
    Minimum prefetch:   0x0
    Maximum prefetch:   0x0
  Device Type:          Disk device
  Path(s):

  /dev/rdsk/c5t600601602B9318008CD071B0FF7BDB11d0s2
  /devices/scsi_vhci/disk@g600601602b9318008cd071b0ff7bdb11:c,raw
   Controller           /dev/cfg/c3
    Device Address              5006016039a00e91,1
    Host controller port WWN    10000000c9563354
    Class                       secondary
    State                       STANDBY
   Controller           /dev/cfg/c3
    Device Address              5006016939a00e91,1
    Host controller port WWN    10000000c9563354
    Class                       primary
    State                       ONLINE
   Controller           /dev/cfg/c4
    Device Address              5006016139a00e91,1
    Host controller port WWN    10000000c9563355
    Class                       secondary
    State                       STANDBY
   Controller           /dev/cfg/c4
    Device Address              5006016839a00e91,1
    Host controller port WWN    10000000c9563355
    Class                       primary
    State                       ONLINE

On the host we were having issues with, Solaris was able to see all 200GB, but for some reason the format “Auto Configure” option still thought the LUN was 50GB. When I chose the format type “DEFAULT,” the LUN showed up with a capacity of 200GB:

$ format -e

format> type
AVAILABLE DRIVE TYPES:
        0. Auto configure
        1. DEFAULT
        2. DEFAULT
        3. DEFAULT
        4. other
Specify disk type (enter its number)[2]: 1
selecting c5t600601602B9318008CD071B0FF7BDB11d0
[disk formatted]

partition> p
Current partition table (original):
Total disk cylinders available: 6524 + 2 (reserved cylinders)

Part      Tag    Flag     Cylinders         Size            Blocks
  0 unassigned    wm       1 - 26104      199.97GB    (26104/0/0) 419360760
  1 unassigned    wm       0                0         (0/0/0)             0
  2     backup    wu       0 - 26104      199.97GB    (26105/0/0) 419376825
  3 unassigned    wm       0                0         (0/0/0)             0
  4 unassigned    wm       0                0         (0/0/0)             0
  5 unassigned    wm       0                0         (0/0/0)             0
  6 unassigned    wm       0                0         (0/0/0)             0
  7 unassigned    wm       0                0         (0/0/0)             0
  8       boot    wu       0 -     0        7.84MB    (1/0/0)         16065
  9 unassigned    wm       0                0         (0/0/0)             0

After chatting with some qualified Sun folks, this madness started to make sense. When you select “Auto Configure,” format will choose an entry from the format.dat file instead of determing the drive’s capacity from the SCSI READ CAPACITY command. The DEFAULT option causes this command to be sent, and therefore the new capacity is available. For future reference, running format with the “-e” option and choosing “DEFAULT” seems to work.

I came across the sdparm utility while surfing the web last weekend. This super useful utility can be used to display and modify SCSI device parameters, and is the best tool I’ve found for dumping SCSI mode and VPD pages. I wish sdparm would have been around when I was reading through the SBC documentation on the T11 website. That would have been swell!

While reviewing the system logs on one of my SAN attached servers last week, I noticed hundreds of entries similar to the following:

Aug 28 13:10:14 foo scsi: [ID 243001 kern.warning] WARNING: /scsi_vhci (scsi_vhci0):
Aug 28 13:10:14 foo        /scsi_vhci/ssd@g600a0b80001fcb370000010646d3d207 (ssd21):
 Command Timeout on path /pci@9,600000/lpfc@1/fp@0,0 (fp3)
Aug 28 13:10:14 foo scsi: [ID 107833 kern.warning]
 WARNING: /scsi_vhci/ssd@g600a0b80001fcb370000010646d3d207 (ssd21):
Aug 28 13:10:14 foo        SCSI transport failed: reason 'timeout': retrying command

Since the errors were retryable, it looked like MPxIO was doing it’s job and retrying requests on one of the other paths. To see if all of the paths were up and operational (the host has four paths to disk), I ran the mpathadm utility with the “list” command and the “LU” option:

$ mpathadm list LU

        /dev/rdsk/c2t600A0B80001FCB370000010446D3D0C1d0s2
                Total Path Count: 4
                Operational Path Count: 4
        /dev/rdsk/c2t600A0B8000216462000000C746D3DA48d0s2
                Total Path Count: 4
                Operational Path Count: 4
        /dev/rdsk/c2t600A0B8000216462000000CD46D3DC1Cd0s2
                Total Path Count: 4
                Operational Path Count: 4
        /dev/rdsk/c2t600A0B80001FCB370000010646D3D207d0s2
                Total Path Count: 4
                Operational Path Count: 4
        /dev/rdsk/c2t600A0B8000216462000000CA46D3DB88d0s2
                Total Path Count: 4
                Operational Path Count: 4

Since all of the paths were available, I started to wonder if a cable was faulty. After running fcinfo on each of the four HBA ports, I came across the following:

$ fcinfo hba-port -l 10000000c94708f2

HBA Port WWN: 10000000c94708f2
        OS Device Name: /dev/cfg/c6
        Manufacturer: Emulex
        Model: LP9002L
        Firmware Version: 3.93a0
        FCode/BIOS Version: 1.41a4
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb
        Current Speed: 2Gb
        Node WWN: 20000000c94708f2
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 14
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 198724
                Invalid CRC Count: 63412

Bingo! The CRC errors were continuosly increasing, so I knew that either the HBA or fibre channel cable were faulty (as a side note, I can’t wait for the FMA project to harden the emlxs and qlc drivers!). During one of my storage training courses, the instructor mentioned that CRC errors are typically associated with bad cables. Once I swapped out the cable that was connected to the port with the errors, the CRC error counts no longer increased, and the scsi_vhci errors stopped! Niiiiiiiiiiiiiiiiiiice!

The /devices and /dev directories on one of my Solaris 9 hosts got majorly borked a few weeks back, and the trusy old `devfsadm -Cv’ command wasn’t able to fix our problem. To clean up the device tree, I booted from CDROM into single user mode and manually cleaned up the device hierarchy. Here is what I did to fix my problems (WARNING: This fixed my problem, but there is no guarantee that this will work for you. Please test changes similar to this on non-production systems prior to adjusting production systems.):

Step 1: Boot from CDROM into single user mode

Step 2: Mount the “/” partition to your favorite place (if your boot devices are mirrored, you will need to perform the following operations on each half of the mirror):

$ mount /dev/dsk/c0t0d0s0 /a

Step 3: Move the existing path_to_inst aside:

$ mv /a/etc/path_to_inst /a/etc/08012007.path_to_inst.orig

Step 4: Clean out the /devices and /dev directories:

$ rm -rf /a/devices/*

$ rm -rf /a/dev/*

Step 5: Replicate the /devices and /dev directories that were created during boot:

$ cd /devices; find . | cpio -pmd /a/devices

$ cd /dev; find . | cpio -pmd /a/dev

Step 6: Adjust the vfstab to reflect any device changes

Step 7: Boot with the “-a”, “-s” and “-r” options to create a new path_to_inst (you can optionally use `devfsadm -C -r /a -p /a/etc/path_to_inst -v’ to create the path_to_inst from single user mode), and to add device entries that weren’t found while booted from single user mode

Step 8: Grab a soda and enjoy the fruits of your labor! :)