Turning an opensolaris host into a fibre channel storage array

The COMSTAR (Common Multiprotocol SCSI Target) project was integrated into opensolaris (Nevada) earlier this year, and provides a framework to turn any opensolaris host into a full featured SCSI target. COMSTAR’s power comes from the fact that it supports several transport protocols, which currently includes iSCSI, FCOE and fibre channel. Having just purchased a Brocade 3200 and a couple of boxes for my lab, I thought I would test out COMSTAR to see how well it performs as as a fibre channel target.

To get my fibre channel target up and operational, I first read through the COMSTAR administration guide and the opensolaris mailing list archives (since COMSTAR is relatively new, the list contains current issues and links to bug fixes).

The configuration I tested with includes the following:

Target: Quad core AMD Opteron server w/ one dual port Emulex LP10002 adapter
Initiator: Dual core AMD Athlon X2 w/ one dual port QLogic 2342 FC adapter

To get the fibre channel ports to operate in target mode, I updated /kernel/drv/emlxs.conf and set “target-mode” to 1:

# Update the emlxs.conf

# target-mode:  Controls COMSTAR target mode support for an adapter port.
#
# 0 = Disables target mode support. Enables  initiator mode support.
# 1 = Enables  target mode support. Disables initiator mode support.
#
# Usage examples:
#     target-mode=1;          Sets global default for target mode
#     emlxs0-target-mode=0;   emlxs0 will be an initiator port
#     emlxs1-target-mode=1;   emlxs1 will be a target port
#
# Range:  Min:0  Max:1  Default:0
#
target-mode=1;



The target mode setting takes effect at driver initialization, so I had to reboot my host to get these settings to take effect. When the box came back up, I used the fcinfo utility to verify the ports were operating in target mode:

$ fcinfo hba-port

HBA Port WWN: 10000000c93e4cea
        Port Mode: Target
        Port ID: 11100
        OS Device Name: Not Applicable
        Manufacturer: Emulex
        Model: LP10000DC
        Firmware Version: 1.92a1 (T2D1.92A1)
        FCode/BIOS Version: none
        Serial Number: MS42085682
        Driver Name: emlxs
        Driver Version: 2.40f (2009.02.10.16.30)
        Type: F-port
        State: online
        Supported Speeds: 1Gb 2Gb 
        Current Speed: 2Gb 
        Node WWN: 20000000c93e4cea
HBA Port WWN: 10000000c93e4ceb
        Port Mode: Target
        Port ID: 11000
        OS Device Name: Not Applicable
        Manufacturer: Emulex
        Model: LP10000DC
        Firmware Version: 1.92a1 (T2D1.92A1)
        FCode/BIOS Version: none
        Serial Number: MS42085682
        Driver Name: emlxs
        Driver Version: 2.40f (2009.02.10.16.30)
        Type: F-port
        State: online
        Supported Speeds: 1Gb 2Gb 
        Current Speed: 2Gb 
        Node WWN: 20000000c93e4ceb



Once the ports were operating in target mode, I enabled the STMF framework service with svcadm:

$ svcadm enable stmf

To verify the STMF framework was functioning, and the ports I configured to operate in target mode were under STMF control, I ran the stmfadm utility with the “list-target” option:

$ stmfadm list-target -v

Target: wwn.10000000C93E4CEB
    Operational Status: Online
    Provider Name     : emlxs1
    Alias             : emlxs1
    Sessions          : 0
Target: wwn.10000000C93E4CEA
    Operational Status: Online
    Provider Name     : emlxs0
    Alias             : emlxs0
    Sessions          : 0



In the output above, we can see that both ports are listed in the online state, and there are currently no sessions established to these targets. We can fix that by adding a few ZFS volumes (zvols) to act as backing stores (you can also use files), and then present these volumes through the STMF framework to one or more initiators. To create the ZFS volumes, the zfs utility can be run with the create subcommand, the “-V” option (create volume), the size of the volume, and the name of the volume relative to the ZFS pool:

$ zfs create -V 100g bits/nevada110disk1

Now that we have a backing store set up, we can use the sbdadm utility to create an STMF logical unit from the ZFS volume:

$ sbdadm create-lu /dev/zvol/rdsk/bits/nevada110disk1

Created the following LU:

              GUID                    DATA SIZE           SOURCE
--------------------------------  -------------------  ----------------
600144f02d65840000004a049b880001      107374116864     /dev/zvol/rdsk/bits/nevada110disk1


$ stmfadm list-lu -v 600144f02d65840000004a049b880001

LU Name: 600144F02D65840000004A049B880001
    Operational Status: Online
    Provider Name     : sbd
    Alias             : /dev/zvol/rdsk/bits/nevada110disk1
    View Entry Count  : 1



Sweet, we now have a logical unit that we can present out to one or more initiators. To ensure that only one node can access the logical unit, we need to create a host group that contains the initiators that can access this logical unit. To create a host group, the stmfadm utility can be run with the “create-hg” option and a name to associate with the host group:

$ stmfadm create-hg nevada110-hg

Once we define a host group, we can add one or more initiators (initiators are specified by their WWNs, which can be located with the stmfadm “list-target” option) to this group:

$ stmfadm add-hg-member -g nevada110-hg wwn.210000E08B1DF903

$ stmfadm add-hg-member -g nevada110-hg wwn.210100E08B3DF903

In the example above, I added two initiators to the nevada110-hg host group. To view the host group configuration, you can run stmfadm with the “list-hg” option:

$ stmfadm list-hg -v

Host Group: centosnode1-hg
        Member: wwn.2100001B320486C3
        Member: wwn.2101001B322486C3
Host Group: nevada110-hg
        Member: wwn.210000E08B1DF903
        Member: wwn.210100E08B3DF903



After the host groups are defined, we can create a view to tie the host group and logical units together:

$ stmfadm add-view -h nevada110-hg 600144f02d65840000004a049b880001

To view the views (pun intended), you can use the stmfadm “list-view” option:

$ stmfadm list-view -l 600144f02d65840000004a049b880001

View Entry: 0
    Host group   : nevada110-hg
    Target group : All
    LUN          : 0



Once all of the steps listed above are complete, you should be able to access the logical units from the initiators:

$ format

Searching for disks...
done

AVAILABLE DISK SELECTIONS:
       0. c0t1d0 
          /pci@0,0/pci1043,8239@5/disk@1,0
       1. c3t600144F02D65840000004A049B8A0002d0 
          /scsi_vhci/disk@g600144f02d65840000004a049b8a0002
Specify disk (enter its number): Specify disk (enter its number): 



This is pretty sweet, and I think a lot of shops will be able to benefit from the awesomeness that is COMSTAR!

5 thoughts on “Turning an opensolaris host into a fibre channel storage array”

  1. Multipath? If a path is active with IO and you yank the cable how does your host deal with the issue? If you are using OpenSolars as a storage target for VMware does the VMware server with the windows machines on it crash?

  2. Wonderful post! Just to add for newbies like me…

    You need to enable expert mode:
    option expert_mode=1

    And the drop to a pure opensolaris/linux prompt by typing:
    !bash

    When you enter that, you will be prompted that you are about to go into dangerous territory. Just say “y” with batting an eye! ;)

  3. Hi! I’m experimenting with this on an OpenIndiana 151a host, but at the moment I’ve only QLA2342 HBA’s, and can’t get theese into target mode.
    I tried to use the /kernel/drv/qlc.conf file, and copied your conf update from emlxs.conf, but unortunatly no luck! Any idéas or suggestions?

    Rgrds Johan

  4. Johan, QLA2342 is too old for COMSTAR. You need QLA2460 (pretty cheap right now) if you want QLT (target driver for qlogic HBAs).

    It looks like old Emulex should work. But I’ve never played with them. Yet. :-D

Leave a Reply

Your email address will not be published. Required fields are marked *