The COMSTAR (Common Multiprotocol SCSI Target) project was integrated into opensolaris (Nevada) earlier this year, and provides a framework to turn any opensolaris host into a full featured SCSI target. COMSTAR’s power comes from the fact that it supports several transport protocols, which currently includes iSCSI, FCOE and fibre channel. Having just purchased a Brocade 3200 and a couple of boxes for my lab, I thought I would test out COMSTAR to see how well it performs as as a fibre channel target.
To get my fibre channel target up and operational, I first read through the COMSTAR administration guide and the opensolaris mailing list archives (since COMSTAR is relatively new, the list contains current issues and links to bug fixes).
The configuration I tested with includes the following:
Target: Quad core AMD Opteron server w/ one dual port Emulex LP10002
adapter
Initiator: Dual core AMD Athlon X2 w/ one dual port QLogic 2342 FC
adapter
To get the fibre channel ports to operate in target mode, I updated /kernel/drv/emlxs.conf and set “target-mode” to 1:
# Update the emlxs.conf
# target-mode: Controls COMSTAR target mode support for an adapter port.
#
# 0 = Disables target mode support. Enables initiator mode support.
# 1 = Enables target mode support. Disables initiator mode support.
#
# Usage examples:
# target-mode=1; Sets global default for target mode
# emlxs0-target-mode=0; emlxs0 will be an initiator port
# emlxs1-target-mode=1; emlxs1 will be a target port
#
# Range: Min:0 Max:1 Default:0
#
target-mode=1;
The target mode setting takes effect at driver initialization, so I had to reboot my host to get these settings to take effect. When the box came back up, I used the fcinfo utility to verify the ports were operating in target mode:
$ fcinfo hba-port
HBA Port WWN: 10000000c93e4cea
Port Mode: Target
Port ID: 11100
OS Device Name: Not Applicable
Manufacturer: Emulex
Model: LP10000DC
Firmware Version: 1.92a1 (T2D1.92A1)
FCode/BIOS Version: none
Serial Number: MS42085682
Driver Name: emlxs
Driver Version: 2.40f (2009.02.10.16.30)
Type: F-port
State: online
Supported Speeds: 1Gb 2Gb
Current Speed: 2Gb
Node WWN: 20000000c93e4cea
HBA Port WWN: 10000000c93e4ceb
Port Mode: Target
Port ID: 11000
OS Device Name: Not Applicable
Manufacturer: Emulex
Model: LP10000DC
Firmware Version: 1.92a1 (T2D1.92A1)
FCode/BIOS Version: none
Serial Number: MS42085682
Driver Name: emlxs
Driver Version: 2.40f (2009.02.10.16.30)
Type: F-port
State: online
Supported Speeds: 1Gb 2Gb
Current Speed: 2Gb
Node WWN: 20000000c93e4ceb
Once the ports were operating in target mode, I enabled the STMF framework service with svcadm:
$ svcadm enable stmf
To verify the STMF framework was functioning, and the ports I configured to operate in target mode were under STMF control, I ran the stmfadm utility with the “list-target” option:
$ stmfadm list-target -v
Target: wwn.10000000C93E4CEB
Operational Status: Online
Provider Name : emlxs1
Alias : emlxs1
Sessions : 0
Target: wwn.10000000C93E4CEA
Operational Status: Online
Provider Name : emlxs0
Alias : emlxs0
Sessions : 0
In the output above, we can see that both ports are listed in the online state, and there are currently no sessions established to these targets. We can fix that by adding a few ZFS volumes (zvols) to act as backing stores (you can also use files), and then present these volumes through the STMF framework to one or more initiators. To create the ZFS volumes, the zfs utility can be run with the create subcommand, the “-V” option (create volume), the size of the volume, and the name of the volume relative to the ZFS pool:
$ zfs create -V 100g bits/nevada110disk1
Now that we have a backing store set up, we can use the sbdadm utility to create an STMF logical unit from the ZFS volume:
$ sbdadm create-lu /dev/zvol/rdsk/bits/nevada110disk1
Created the following LU:
GUID DATA SIZE SOURCE
-------------------------------- ------------------- ----------------
600144f02d65840000004a049b880001 107374116864 /dev/zvol/rdsk/bits/nevada110disk1
$ stmfadm list-lu -v 600144f02d65840000004a049b880001
LU Name: 600144F02D65840000004A049B880001
Operational Status: Online
Provider Name : sbd
Alias : /dev/zvol/rdsk/bits/nevada110disk1
View Entry Count : 1
Sweet, we now have a logical unit that we can present out to one or more initiators. To ensure that only one node can access the logical unit, we need to create a host group that contains the initiators that can access this logical unit. To create a host group, the stmfadm utility can be run with the “create-hg” option and a name to associate with the host group:
$ stmfadm create-hg nevada110-hg
Once we define a host group, we can add one or more initiators (initiators are specified by their WWNs, which can be located with the stmfadm “list-target” option) to this group:
$ stmfadm add-hg-member -g nevada110-hg wwn.210000E08B1DF903
$ stmfadm add-hg-member -g nevada110-hg wwn.210100E08B3DF903
In the example above, I added two initiators to the nevada110-hg host group. To view the host group configuration, you can run stmfadm with the “list-hg” option:
$ stmfadm list-hg -v
Host Group: centosnode1-hg
Member: wwn.2100001B320486C3
Member: wwn.2101001B322486C3
Host Group: nevada110-hg
Member: wwn.210000E08B1DF903
Member: wwn.210100E08B3DF903
After the host groups are defined, we can create a view to tie the host group and logical units together:
$ stmfadm add-view -h nevada110-hg 600144f02d65840000004a049b880001
To view the views (pun intended), you can use the stmfadm “list-view” option:
$ stmfadm list-view -l 600144f02d65840000004a049b880001
View Entry: 0
Host group : nevada110-hg
Target group : All
LUN : 0
Once all of the steps listed above are complete, you should be able to access the logical units from the initiators:
$ format
Searching for disks...
done
AVAILABLE DISK SELECTIONS:
0. c0t1d0
/pci@0,0/pci1043,8239@5/disk@1,0
1. c3t600144F02D65840000004A049B8A0002d0
/scsi_vhci/disk@g600144f02d65840000004a049b8a0002
Specify disk (enter its number): Specify disk (enter its number):
This is pretty sweet, and I think a lot of shops will be able to benefit from the awesomeness that is COMSTAR!