Managing COMSTAR logical units with sbdadm

With project COMSTAR, you can present logical units to one or more hosts connected via Ethernet or fibre channel. Logical units are managed with the sbdadm utility, which has options to add, delete, list, import and modify logical units. To create a logical unit using a ZFS volume, the path to the ZFS volume can be passed to the “create-lu” option:

$ zfs create -V 18g bits/testvol

$ sbdadm create-lu /dev/zvol/rdsk/bits/testvol

Created the following LU:

              GUID                    DATA SIZE           SOURCE
--------------------------------  -------------------  ----------------
600144f02d65840000004a04c6b90004      19327287296      /dev/zvol/rdsk/bits/testvol



If you would prefer to use a file instead of a volume, you can do that as well:

$ touch /foo

$ sbdadm create-lu -s 100g /foo

Created the following LU:

              GUID                    DATA SIZE           SOURCE
--------------------------------  -------------------  ----------------
600144f02d65840000004a117a790002      107374182400     /foo



In the example above, I used the COMSTAR thin provisioning support to create a 100GB logical unit using the file /tmp. After a logical unit is added, it can be listed with the sbdadm “list-lu” option:

$ sbdadm list-lu

Found 4 LU(s)

              GUID                    DATA SIZE           SOURCE
--------------------------------  -------------------  ----------------
600144f02d65840000004a117a790002      26843545600      /foo
600144f02d65840000004a04c6b90004      19327287296      /dev/zvol/rdsk/bits/testvol
600144f02d65840000004a049b880001      107374116864     /dev/zvol/rdsk/bits/nevada110disk1
600144f02d658400000049fba1bf0001      107374116864     /dev/zvol/rdsk/bits/centosnode1



If you decide that a logical unit is no longer needed, you can remove it with the sbdadm “delete-lu” option:

$ sbdadm delete-lu 600144f02d65840000004a04c6b90004

If you need to increase or decrease the size of a logical unit, you can use the sbdadm “modify-lu” option:

$ sbdadm modify-lu -s 25g 600144f02d65840000004a117a790002

LU modified Successfully.



There is also an “import-lu” option which can be used to import a volume or file into COMSTAR. Nice!

Debugging issues with the COMSTAR fibre channel target

I recently talked about how you can turn an opensolaris host into a fibre channel storage array. The STMF framework maintains an internal trace buffer with recent activity, which can be extremely useful for debugging problems. To view the contents of the buffer, you can send the string “*stmf_trace_buf” to mdb:

$ echo ‘*stmf_trace_buf/s’ |mdb -k | more

0xffffff02d66ca000:             emlxs0:0004363: iport is ffffff02e10a07d0
emlxs1:0004809: iport is ffffff02df3a2d90
emlxs0:0005963: port state change from 0 to e
emlxs0:0005969: Posting sol ELS 3 (PLOGI) rp_id=fffffd lp_id=11100
emlxs0:0005969: Processing sol ELS 3 (PLOGI) rp_id=fffffd
emlxs0:0005969: Sol ELS 3 (PLOGI) completed with status 0, did/fffffd
emlxs0:0005969: Posting sol ELS 62 (SCR) rp_id=fffffd lp_id=11100
emlxs0:0005969: Processing sol ELS 62 (SCR) rp_id=fffffd
.....



This will print a slew of debugging data to your screen, which can be extremely handy for solving issues.

Turning an opensolaris host into a fibre channel storage array

The COMSTAR (Common Multiprotocol SCSI Target) project was integrated into opensolaris (Nevada) earlier this year, and provides a framework to turn any opensolaris host into a full featured SCSI target. COMSTAR’s power comes from the fact that it supports several transport protocols, which currently includes iSCSI, FCOE and fibre channel. Having just purchased a Brocade 3200 and a couple of boxes for my lab, I thought I would test out COMSTAR to see how well it performs as as a fibre channel target.

To get my fibre channel target up and operational, I first read through the COMSTAR administration guide and the opensolaris mailing list archives (since COMSTAR is relatively new, the list contains current issues and links to bug fixes).

The configuration I tested with includes the following:

Target: Quad core AMD Opteron server w/ one dual port Emulex LP10002 adapter
Initiator: Dual core AMD Athlon X2 w/ one dual port QLogic 2342 FC adapter

To get the fibre channel ports to operate in target mode, I updated /kernel/drv/emlxs.conf and set “target-mode” to 1:

# Update the emlxs.conf

# target-mode:  Controls COMSTAR target mode support for an adapter port.
#
# 0 = Disables target mode support. Enables  initiator mode support.
# 1 = Enables  target mode support. Disables initiator mode support.
#
# Usage examples:
#     target-mode=1;          Sets global default for target mode
#     emlxs0-target-mode=0;   emlxs0 will be an initiator port
#     emlxs1-target-mode=1;   emlxs1 will be a target port
#
# Range:  Min:0  Max:1  Default:0
#
target-mode=1;



The target mode setting takes effect at driver initialization, so I had to reboot my host to get these settings to take effect. When the box came back up, I used the fcinfo utility to verify the ports were operating in target mode:

$ fcinfo hba-port

HBA Port WWN: 10000000c93e4cea
        Port Mode: Target
        Port ID: 11100
        OS Device Name: Not Applicable
        Manufacturer: Emulex
        Model: LP10000DC
        Firmware Version: 1.92a1 (T2D1.92A1)
        FCode/BIOS Version: none
        Serial Number: MS42085682
        Driver Name: emlxs
        Driver Version: 2.40f (2009.02.10.16.30)
        Type: F-port
        State: online
        Supported Speeds: 1Gb 2Gb 
        Current Speed: 2Gb 
        Node WWN: 20000000c93e4cea
HBA Port WWN: 10000000c93e4ceb
        Port Mode: Target
        Port ID: 11000
        OS Device Name: Not Applicable
        Manufacturer: Emulex
        Model: LP10000DC
        Firmware Version: 1.92a1 (T2D1.92A1)
        FCode/BIOS Version: none
        Serial Number: MS42085682
        Driver Name: emlxs
        Driver Version: 2.40f (2009.02.10.16.30)
        Type: F-port
        State: online
        Supported Speeds: 1Gb 2Gb 
        Current Speed: 2Gb 
        Node WWN: 20000000c93e4ceb



Once the ports were operating in target mode, I enabled the STMF framework service with svcadm:

$ svcadm enable stmf

To verify the STMF framework was functioning, and the ports I configured to operate in target mode were under STMF control, I ran the stmfadm utility with the “list-target” option:

$ stmfadm list-target -v

Target: wwn.10000000C93E4CEB
    Operational Status: Online
    Provider Name     : emlxs1
    Alias             : emlxs1
    Sessions          : 0
Target: wwn.10000000C93E4CEA
    Operational Status: Online
    Provider Name     : emlxs0
    Alias             : emlxs0
    Sessions          : 0



In the output above, we can see that both ports are listed in the online state, and there are currently no sessions established to these targets. We can fix that by adding a few ZFS volumes (zvols) to act as backing stores (you can also use files), and then present these volumes through the STMF framework to one or more initiators. To create the ZFS volumes, the zfs utility can be run with the create subcommand, the “-V” option (create volume), the size of the volume, and the name of the volume relative to the ZFS pool:

$ zfs create -V 100g bits/nevada110disk1

Now that we have a backing store set up, we can use the sbdadm utility to create an STMF logical unit from the ZFS volume:

$ sbdadm create-lu /dev/zvol/rdsk/bits/nevada110disk1

Created the following LU:

              GUID                    DATA SIZE           SOURCE
--------------------------------  -------------------  ----------------
600144f02d65840000004a049b880001      107374116864     /dev/zvol/rdsk/bits/nevada110disk1


$ stmfadm list-lu -v 600144f02d65840000004a049b880001

LU Name: 600144F02D65840000004A049B880001
    Operational Status: Online
    Provider Name     : sbd
    Alias             : /dev/zvol/rdsk/bits/nevada110disk1
    View Entry Count  : 1



Sweet, we now have a logical unit that we can present out to one or more initiators. To ensure that only one node can access the logical unit, we need to create a host group that contains the initiators that can access this logical unit. To create a host group, the stmfadm utility can be run with the “create-hg” option and a name to associate with the host group:

$ stmfadm create-hg nevada110-hg

Once we define a host group, we can add one or more initiators (initiators are specified by their WWNs, which can be located with the stmfadm “list-target” option) to this group:

$ stmfadm add-hg-member -g nevada110-hg wwn.210000E08B1DF903

$ stmfadm add-hg-member -g nevada110-hg wwn.210100E08B3DF903

In the example above, I added two initiators to the nevada110-hg host group. To view the host group configuration, you can run stmfadm with the “list-hg” option:

$ stmfadm list-hg -v

Host Group: centosnode1-hg
        Member: wwn.2100001B320486C3
        Member: wwn.2101001B322486C3
Host Group: nevada110-hg
        Member: wwn.210000E08B1DF903
        Member: wwn.210100E08B3DF903



After the host groups are defined, we can create a view to tie the host group and logical units together:

$ stmfadm add-view -h nevada110-hg 600144f02d65840000004a049b880001

To view the views (pun intended), you can use the stmfadm “list-view” option:

$ stmfadm list-view -l 600144f02d65840000004a049b880001

View Entry: 0
    Host group   : nevada110-hg
    Target group : All
    LUN          : 0



Once all of the steps listed above are complete, you should be able to access the logical units from the initiators:

$ format

Searching for disks...
done

AVAILABLE DISK SELECTIONS:
       0. c0t1d0 
          /pci@0,0/pci1043,8239@5/disk@1,0
       1. c3t600144F02D65840000004A049B8A0002d0 
          /scsi_vhci/disk@g600144f02d65840000004a049b8a0002
Specify disk (enter its number): Specify disk (enter its number): 



This is pretty sweet, and I think a lot of shops will be able to benefit from the awesomeness that is COMSTAR!

Updating Emulex boot code on Solaris hosts

I previously discussed updating Emulex adapter firmware, but didn’t touch on updating the adapter’s boot code. The boot code allows you to boot from a SAN device, and is a separate upgrade from the standard firmware. To upgrade the boot code, you can run the emlxadm download_boot command with the name of the boot code image:

emlxadm> download_boot /export/home/matty/TB202A2.PRG

Image Components: REL type size=50232
DWC file: BOOT: version=03812092, 2.02a2

Current: Boot: none
New: Boot: 2.02a2 50232 (0xc438) bytes

Are you sure you want to download this image? (y or n): y

Downloading…

Result: Operation successful.
Done.

emlxadm> get_boot_rev

Boot revision: LP10000DC 2.02a2



This post is a reference for myself, and I take ZERO responsibility for botched firmware updates. As with all information on this blog, use this information at your own risk (no warranties of any sort are provided).

SCSI Enclosure Services

Eric Schrock has done some really cool work with integrating disk (SMART) /platform monitoring (IPMI) information into Opensolaris.   Just recently, he has extended FMA with a new technology called SES (SCSI Enclosure Services) into build 93 of OpenSolaris.

This looks like some really cool stuff.  The following was taken directly from his blog on the examples of using the new fmtopo utility to map out an external storage array.

# /usr/lib/fm/fmd/fmtopo
...

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:serial=2029QTF0000000002:part=Storage-J4400:revision=3R13/ses-enclosure=0

hc://:product-id=SUN-Storage-J4400:chassis-id=22029QTF0809QCK012:server-id=:part=123-4567-01/ses-enclosure=0/psu=0

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=:part=123-4567-01/ses-enclosure=0/psu=1

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=/ses-enclosure=0/fan=0

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=/ses-enclosure=0/fan=1

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=/ses-enclosure=0/fan=2

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=/ses-enclosure=0/fan=3

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=:serial=2029QTF0811RM0386:part=375-3584-01/ses-enclosure=0/controller=0

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=:serial=2029QTF0811RM0074:part=375-3584-01/ses-enclosure=0/controller=1

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=/ses-enclosure=0/bay=0

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=:serial=5QD0PC3X:part=SEAGATE-ST37500NSSUN750G-0720A0PC3X:revision=3.AZK/ses-enclosure=0/bay=0/disk=0

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=/ses-enclosure=0/bay=1

...

# fmtopo -V '*/ses-enclosure=0/bay=0/disk=0'
TIME                 UUID
Jul 14 03:54:23 3e95d95f-ce49-4a1b-a8be-b8d94a805ec8

hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=:serial=5QD0PC3X:part=SEAGATE-ST37500NSSUN750G-0720A0PC3X:revision=3.AZK/ses-enclosure=0/bay=0/disk=0
  group: protocol                       version: 1   stability: Private/Private
    resource          fmri      hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=:serial=5QD0PC3X:part=SEAGATE-ST37500NSSUN750G-0720A0PC3X:revision=3.AZK/ses-enclosure=0/bay=0/disk=0
    ASRU              fmri      dev:///:devid=id1,sd@TATA_____SEAGATE_ST37500NSSUN750G_0720A0PC3X_____5QD0PC3X____________//scsi_vhci/disk@gATASEAGATEST37500NSSUN750G0720A0PC3X5QD0PC3X
    label             string    SCSI Device  0
    FRU               fmri      hc://:product-id=SUN-Storage-J4400:chassis-id=2029QTF0809QCK012:server-id=:serial=5QD0PC3X:part=SEAGATE-ST37500NSSUN750G-0720A0PC3X:revision=3.AZK/ses-enclosure=0/bay=0/disk=0
  group: authority                      version: 1   stability: Private/Private
    product-id        string    SUN-Storage-J4400
    chassis-id        string    2029QTF0809QCK012
    server-id         string
  group: io                             version: 1   stability: Private/Private
    devfs-path        string    /scsi_vhci/disk@gATASEAGATEST37500NSSUN750G0720A0PC3X5QD0PC3X
    devid             string    id1,sd@TATA_____SEAGATE_ST37500NSSUN750G_0720A0PC3X_____5QD0PC3X____________
    phys-path         string[]  [ /pci@0,0/pci10de,377@a/pci1000,3150@0/disk@1c,0 /pci@0,0/pci10de,375@f/pci1000,3150@0/disk@1c,0 ]
  group: storage                        version: 1   stability: Private/Private
    logical-disk      string    c0tATASEAGATEST37500NSSUN750G0720A0PC3X5QD0PC3Xd0
    manufacturer      string    SEAGATE
    model             string    ST37500NSSUN750G 0720A0PC3X
    serial-number     string    5QD0PC3X
    firmware-revision string       3.AZK
    capacity-in-bytes string    750156374016

OpenSolaris now supports dynamic LUN expansion

With the following putback:

PSARC 2006/373 Dynamic Lun Expansion
6241086 Format should allow label adjustment when disk/lun size changes
6430818 Solaris needs mechanism of dynamically increasing lun size

OpenSoaris now has the ability to detect and use space that is dynamically allocated to a LUN. This has been a bit of a stumbling block in the past, and the available solutions to expand storage weren’t for the faint of heart. Hopefully this feature works as advertised!!