Populating /dev/sg when Leadville drivers are in use

For the past day and a half, I have been trying to create device entries for several STK 9940B drives in the /dev/sg directory on a Solaris 10 server that is using the emlx driver. This should be straight forward, since Netbackup provides the sg.build shell script to assist with populating the sg.conf and devlink.tab files. Due to some issues with the way the sg.build script parses the output from luxadm, the script isn’t able to locate the WWPN of the emlx controlled HBAs in our servers. Once I found the problem, I was able to get things working by hand editing the results from the sg.build prior to updating the system files. Below is an overview of what I did, which will hopefully assist others who are encountering this problem.

To get your Solaris 10 server to properly populate /dev/rmt/* and /dev/sg/* with your tape devices, you will first need to locate the WWPN of each tape drive that you want your server to use. This information can be retrieved two ways. The first way is with the fcinfo “remote-port” option:

$ fcinfo remote-port -slp 10000000c92a9043

Remote Port WWN: 500104f0005f029d
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 500104f0005f027c
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 21
                Invalid CRC Count: 0
        LUN: 0
          Vendor: STK     
          Product: T9940B          
          OS Device Name: /dev/rmt/7n

This information can also be retrieved by first running luxadm to get the physical device paths for each HBA:

$ luxadm -e port

/devices/pci@8,600000/SUNW,qlc@2/fp@0,0:devctl                     CONNECTED
/devices/pci@8,700000/lpfc@4/fp@0,0:devctl                         CONNECTED
/devices/pci@8,700000/lpfc@5/fp@0,0:devctl                         CONNECTED
/devices/pci@9,600000/lpfc@1/fp@0,0:devctl                         CONNECTED
/devices/pci@9,600000/lpfc@2/fp@0,0:devctl                         CONNECTED

After you retrieve the device paths, you can run luxadm with the “-e” and “dump_map” options to display the targets and LUNs that are present on each path:

$ luxadm -e dump_map /devices/pci@8,700000/lpfc@4/fp@0,0:devctl

Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type
0    21855   55        500104f0005f027c 500104f0005f027b 0x1  (Tape device)
1    21955   55        500104f0005f0285 500104f0005f0284 0x1  (Tape device)
2    21b55   55        500104f0005f026d 500104f0005f026c 0x1  (Tape device)
3    21a00   0         10000000c936560c 20000000c936560c 0x1f (Unknown Type,Host Bus Adap
ter)

Once you identify the WWPNs associated with your tape devices, you will need to add one entry per tape drive to the /kernel/drv/sg.conf configuration file. Here are two sample sg.conf entries:

name="sg" parent="fp" target=0 lun=0 fc-port-wwn="500104f0005f027c";
name="sg" parent="fp" target=0 lun=0 fc-port-wwn="500104f0005f0285";

After the sg.conf file is updated, you will need to add one entry per tape drive to the /etc/devlink.tab file. These entries will then be used by devfsadmd to populate the /dev/sg directory with the pertinent symbolic links. Here are two sample entries from my devlink.tab file:

type=ddi_pseudo;name=sg;addr=w500104f0005f027c,0;       sg/c\N0t\A1l0
type=ddi_pseudo;name=sg;addr=w500104f0005f0285,0;       sg/c\N0t\A1l0

In addition to defining the device links for the /dev/sg directory in the devlink.tab file, you can also add one or more entries to bind your tape drives to specific /dev/rmt/[0-9]+ numbers. Here are two sample entries that map tape drives to specific /dev/rmt/ entries:

type=ddi_byte:tape;addr=w500104f0005f027c,0;    rmt/0\M0
type=ddi_byte:tape;addr=w500104f0005f0285,0;    rmt/1\M0

While I can’t easily represent tabs in my blog post, you should note that there is a tab between the “;” and the “sg” and “rmt” strings. After you update and verify all of the configuration files listed above, you should clear out /dev/rmt/* and /dev/sg/* to ensure that the device entries are created cleanly (if you don’t clean out the old entries, you will get new entries tacked on to the old ones):

$ rm -f /dev/rmt/*

$ rm -f /dev/sg/*

If you removed the device entries, you will then need to perform a reconfiguration reboot to rebuild your device links (for some reason, the /dev/sg entries don’t get populated until a reconfiguration reboot occurs). When the box comes back up, you should have a nice clean set of links in /dev/rmt/* and /dev/sg/*:

$ ls -la /dev/rmt/* | head -1

lrwxrwxrwx   1 root     root          64 Jun 25 16:40 /dev/rmt/0 -> ../../devices/pci@8,700000/lpfc@4/fp@0,0/st@w500104f0005f027c,0:

$ ls -la /dev/sg/* | head -1

lrwxrwxrwx   1 root     root          67 Jun 25 16:44 /dev/sg/c0tw500104f0005f026dl0 -> ../../devices/pci@8,700000/lpfc@4/fp@0,0/sg@w500104f0005f026d,0:raw

Now that my tape drives are showing up correctly, it’s time to start doing some backups!

Preparing for the Sun cluster 3.2 certification

I recently passed the Sun Certified System Administrator For Sun Cluster 3.2 exam, and thought I would share my thoughts on the exam in case others are interested in taking it. When I decided to take the exam, I had limited experience with Sun Cluster 3.X, and didn’t have money for a training course. To prepare for the certification exam, I first read the following documents cover to cover (I didn’t read the entire developer’s guide, just parts of it to better undertand how data services actually work):

Sun cluster 3.2 concepts guide

Sun cluster 3.2 system administration guide

Sun quorum server reference manual

Sun cluster 3.2 data service for Oracle guide

Sun cluster 3.2 data service for Oracle RAC guide

Sun cluster 3.2 data service developer’s guide

In addition to the documents listed above, it is also important to understand how Solaris Volume Manager and Veritas Volume Manager work in clustered environments. While I can’t go into test specifics due to the agreement I signed when I took the exam, you should fair pretty well if you read through the documents listed above, and understand each of the items listed in the Sun cluster certification objectives. I made sure I knew the objectives by creating a two column table and placing each objective in it’s own cell on the left side of the table, and then filling in the cell on the right side of the table as I came across information relating to the objective. I then used this information to cram the week of the exam, and now have it available as a reference. If you decide to take the exam, good luck!!! Let me know what you thought of it!!

Using the Emulex storage tools to manage Leadville controlled Emulex adapters

With the introduction of Solaris 10, the Sun leadville storage stack was integrated into Solaris. This means that you no longer need to download and install the SFS stack or third party drivers, and the native Solaris tools (e.g., iostat, mpathadm, fcinfo, luxadm, etc.) can be used to manage specific aspects of storage management. I would like to emphasize the word “specific,” since there are a number of management tasks that can’t be performed (or performed easily) with the built-in tools.

If you happen to be using Emulex host bus adapters with Solaris 10, you are in luck! Emulex provides the SFS drivers and emlxadm management tool, which can be used to supplement the tools provided with Solaris. The Emulex tools and drivers are provided by the EMLXemlxu package, which you can download from Emulex’s website. The pkgadd utility can be used to install the package, and it will place the management tools under the /opt/EMLXemlxu directory. To begin using the Emulex storage management tools, you can run the emlxadm utility without any options:

$ /opt/EMLXemlxu/bin/emlxadm

Available Emulex HBA's:

1.  /devices/pci@8,700000/lpfc@5/fp@0,0 (CONNECTED)
2.  /devices/pci@8,700000/lpfc@4/fp@0,0 (CONNECTED)
3.  /devices/pci@9,600000/lpfc@1/fp@0,0 (CONNECTED)
4.  /devices/pci@9,600000/lpfc@2/fp@0,0 (CONNECTED)

Enter an HBA number or zero to exit: 1

This will display the list of adapters that are installed on the system, and will allow you to select a specific adapter to manage. Once you select a specific HBA from the list, you can issue commands through the emlxadm shell to view or configure HBAs. To view the fcode, boot code and firmware versions installed on the detected HBAs, you can use the “get_fw_rev,” “get_fcode_rev” and “get_boot_rev” commands:

emlxadm> get_fw_rev

Firmware revision: LP9002L 3.93a0

emlxadm> get_fcode_rev

FCODE revision: LP9002L 1.41a4

emlxadm> get_boot_rev

Boot revision: LP9002L 5.00a5

To view HBA specific attributes such as the link speed, HBA state and the WWPN assigned to the card, you can run the “get_host_params” command:

emlxadm> get_host_params

Host:
          Dtype: 0
FC4_type[proto]: 0x00000120, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 0x00000000, 
0x00000000, 0x00000000
          State: Online
      Linkspeed: 2Gb
           D_id: 11b00
           LILP: 0
      Hard Addr: 0
           WWPN: 10000000c936560c
           WWNN: 20000000c936560c

To list the number of targets visible through a specific HBA, the “get_num_devs” command can be used:

emlxadm> get_num_devs

There are 3 devices reported on this port.

Since the fabric name server contains a slew of useful information, emlxadm provides a “ns” command to dump it’s contents:

emlxadm> ns

Nameserver: 
-----------------------------------------------------------------------------------
     TYPE: Lport
      PID: 011855 
     WWPN: 500104f0005f0276
PORT_NAME: (STK     T9940B          1.35)
     WWNN: 500104f0005f0275
NODE_NAME: (null)
      IPA: ffffffffffffffff
  IP_ADDR: 0.0.0.0
    CLASS: Class3
FC4_TYPES: 00000100,00000000,00000000,00000000,00000000,00000000,00000000,00000000 

-----------------------------------------------------------------------------------
     TYPE: Nport
      PID: 011900 
     WWPN: 500104f0005f027f
PORT_NAME: (STK     T9940B          1.35)
     WWNN: 500104f0005f027e
NODE_NAME: (null)
      IPA: ffffffffffffffff
  IP_ADDR: 0.0.0.0
    CLASS: Class3
FC4_TYPES: 00000100,00000000,00000000,00000000,00000000,00000000,00000000,00000000 

-----------------------------------------------------------------------------------
     TYPE: Lport
      PID: 011A55 
     WWPN: 500104f0005f0273
PORT_NAME: (STK     T9940B          1.35)
     WWNN: 500104f0005f0272
NODE_NAME: (null)
      IPA: ffffffffffffffff
  IP_ADDR: 0.0.0.0
    CLASS: Class3
FC4_TYPES: 00000100,00000000,00000000,00000000,00000000,00000000,00000000,00000000 

The emlxadm utility also provides “download_<fcode|fw|boot>” commands to download firmware images to each HBA, and has several “reset_<link|hard|hard_core>” commands to reset HBAs. These are two areas where the native Solaris tools don’t shine so well, so the emlxadm utility is a welcome addition to the Solaris storage stack! Viva la Emulex!!

Observing I/O behavior with the DTraceToolkit article

I just posted my article observing I/O behavior with the DTraceToolkit to my website. If your interested in learning more about how to analyze I/O behavior on Solaris 10 and Nevada hosts, you might be interested in the article. I would like to thank Brendan Gregg for proofreading the article for me.