The wonderful world of Leadville

In a SAN environment when dealing with external storage concepts such as EMC BCV’s, you’ll often have a request to create volumes on two different machines that are identical so replication on the back-end can occur.

 

When you look at a LUN presented to Solaris, it’ll appear with a cryptic name like the following:

 103. c20t60060480000190100665533030393836d0 <EMC-SYMMETRIX-5771 cyl 36826 alt 2 hd 60 sec 128>
          /scsi_vhci/ssd@g60060480000190100665533030353339

The c20 relates to the HBA (Fiber, SCSI, iSCSI) that provides a path to the device.  The “middle” sequence 60060480000190100665533030393836 between the “t (target)” and “d (device” is the WWN of the LUN.

 

Now, say your SAN engineer approaches you with some information like the following….

PROD DEV: 936

PROD LUN: 47

STAGE DEV: 986

STAGE LUN: 68

 

Ok… so what does that mean to us?  Using luxadm, we can probe a target to find out some specifc information about its back-end LUN name.

(root)> luxadm display /dev/rdsk/c20t60060480000190100665533030393836d0s2
DEVICE PROPERTIES for disk: /dev/rdsk/c20t60060480000190100665533030393836d0s2
  Vendor:               EMC
  Product ID:           SYMMETRIX
  Revision:             5771
  Serial Num:           xxxxxxxxxxx
  Unformatted capacity: 138105.000 MBytes
  Read Cache:           Enabled
    Minimum prefetch:   0x0
    Maximum prefetch:   0xffff
  Device Type:          Disk device
  Path(s):

  /dev/rdsk/c20t60060480000190100665533030393836d0s2
  /devices/scsi_vhci/ssd@g60060480000190100665533030393836:c,raw
   Controller           /devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0
    Device Address              50060482d52d2e76,68
    Host controller port WWN    210100e08ba0147a
    Class                       primary
    State                       ONLINE
   Controller           /devices/pci@17c,600000/SUNW,qlc@1,1/fp@0,0
    Device Address              50060482d52d2e59,68
    Host controller port WWN    210100e08ba0da73
    Class                       primary
    State                       ONLINE

The Device Address is the field we are interested in.  This shows us that the WWN of the Port we are plugged into is 50060482d52d2e59 and the LUN number is 68.

Now that we have the LUN number, we know what this LUN maps to.  We can then find the cooresponding LUN on the other machine (LUN Number 47) which maps to the BCV pair.

 

There are some other useful leadville commands that you may be interested in…

To display all HBAs available for use:

(root)> luxadm -e port
/devices/pci@15c,600000/SUNW,qlc@1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl                 CONNECTED
/devices/pci@15d,600000/SUNW,qlc@1,1/fp@0,0:devctl                 NOT CONNECTED
/devices/pci@15d,600000/SUNW,qlc@1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@17c,600000/SUNW,qlc@1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@17c,600000/SUNW,qlc@1,1/fp@0,0:devctl                 CONNECTED
/devices/pci@17d,600000/SUNW,qlc@1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@17d,600000/SUNW,qlc@1,1/fp@0,0:devctl                 NOT CONNECTED
/devices/pci@19c,600000/SUNW,qlc@1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@19c,600000/SUNW,qlc@1,1/fp@0,0:devctl                 NOT CONNECTED
/devices/pci@19d,600000/SUNW,qlc@1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@19d,600000/SUNW,qlc@1,1/fp@0,0:devctl                 NOT CONNECTED

Now that you have the device name, you can map that back to what device it cooralates to under /dev.

In this case since i’m using a Fiber channel HBA…

(root)> ls -l /dev/fc | grep /devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl
lrwxrwxrwx   1 root     root          55 Sep 26  2007 fp1 -> ../../devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl

Sweet.  So one of my HBAs is fp1.

Want to see more detailed information about what that Fiber HBA is connected to?

umt1a-bio-stg1:~
(root)> luxadm -e dump_map /devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl
Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type
0    611813  0         50060e8004769000 50060e8004769000 0x0  (Disk device)
1    624613  0         50060482d52d2e76 50060482d52d2e76 0x0  (Disk device)
2    617c13  0         210100e08ba0147a 200100e08ba0147a 0x1f (Unknown Type,Host Bus Adapter)

Note that the Port WWN 50060482d52d2e76 is the same WWN we saw above when looking for the LUN number. 

 

Want a dump of all LUNs attached to a controller?

(root)> cfgadm -o show_FCP_dev -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c4::50060482d52d2e76,0         disk         connected    configured   unknown
c4::50060482d52d2e76,69        disk         connected    configured   unknown
c4::50060482d52d2e76,70        disk         connected    configured   unknown
c4::50060482d52d2e76,71        disk         connected    configured   unknown
c4::50060482d52d2e76,72        disk         connected    configured   unknown
c4::50060482d52d2e76,73        disk         connected    configured   unknown
c4::50060482d52d2e76,74        disk         connected    configured   unknown
c4::50060482d52d2e76,75        disk         connected    configured   unknown
c4::50060482d52d2e76,76        disk         connected    configured   unknown
c4::50060482d52d2e76,77        disk         connected    configured   unknown
c4::50060482d52d2e76,78        disk         connected    configured   unknown
c4::50060482d52d2e76,79        disk         connected    configured   unknown
c4::50060482d52d2e76,80        disk         connected    configured   unknown

<snip>

….

Another super useful utility is the new fcinfo command which was introduced into Solaris 10.  The -l (linkstat) shows some detailed statistics on the HBA.

(root)> fcinfo hba-port -l
HBA Port WWN: 210100e08ba0147a
        OS Device Name: /dev/cfg/c4
        Manufacturer: QLogic Corp.
        Model: 375-3108-xx
        Firmware Version: 3.3.26
        FCode/BIOS Version:  fcode: 1.13;
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb
        Current Speed: 2Gb
        Node WWN: 200100e08ba0147a
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 0
                Invalid CRC Count: 0

        

With the WWN of the HBA, we can query the remote port information…

(root)> fcinfo remote-port -p 210100e08ba0147a
Remote Port WWN: 50060482d52d2e76
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060482d52d2e76
Remote Port WWN: 50060e8004769000
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060e8004769000

Check it out!  Its the same 50060482d52d2e76 we saw twice before with the WWN of the port we’re plugged into the fabric with.

 

Throw in a -s, and it’ll return all SCSI targets with their LUN Number.

(root)> fcinfo remote-port -p 210100e08ba0147a -s
Remote Port WWN: 50060482d52d2e76
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060482d52d2e76
        LUN: 0
          Vendor: EMC
          Product: SYMMETRIX
          OS Device Name: /dev/rdsk/c4t50060482D52D2E76d0s2
        LUN: 69
          Vendor: EMC
          Product: SYMMETRIX
          OS Device Name: /dev/rdsk/c20t60060480000190100665533030344539d0s2
        LUN: 70
          Vendor: EMC
          Product: SYMMETRIX
          OS Device Name: /dev/rdsk/c20t60060480000190100665533030344639d0s2
        LUN: 71
          Vendor: EMC
          Product: SYMMETRIX
          OS Device Name: /dev/rdsk/c20t60060480000190100665533030353039d0s2
        LUN: 72
          Vendor: EMC
          Product: SYMMETRIX
          OS Device Name: /dev/rdsk/c20t60060480000190100665533030353139d0s2
        LUN: 73
          Vendor: EMC
          Product: SYMMETRIX
          OS Device Name: /dev/rdsk/c20t60060480000190100665533030353239d0s2

 
Want to force a specific Fiber HBA to reinitialized and re-login into the Fabric?

(root)> luxadm -e forcelip <device>

i.e.

(root)> luxadm -e forcelip /devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl

5 thoughts on “The wonderful world of Leadville”

  1. Very instructive post. I’m going to practice some of the command you show here and take some notes, so I learn a bit more about it. I’m always a bit confused about how to get that kind of HBA/wwn/disk relations and this post will sure help me a lot.

    Thank a lot. I really find your blog a must for learning more about Solaris things.

  2. I love this site.. Most of the times I have used the commands and it works..like this ones suggested about leadville..keep up the good work.

Leave a Reply

Your email address will not be published. Required fields are marked *