Blog O’ Matty

I have written several times about the Solaris Leadvilel storage stack. While debugging a problem on a Solaris 9 host that wasn’t using the Leadville stack, I would wait and wait and wait for various commands to complete. Here is one such example:

$ timex /usr/openv/volmgr/bin/scan >/dev/null

real        2:25.09
user           0.10
sys            0.45

After migrating the same host to Solaris 10 and the Leadville stack, I no longer had to wait for my commands to complete. Here is an example:

$ timex /usr/openv/volmgr/bin/scan >/dev/null

real           0.11
user           0.05
sys            0.03

After doing a bit of debugging, I noticed that the Solaris 9 fibre channel stack (which was using a vendor supplied HBA driver) was enumerating each path to see if devices were present. In the Solaris 10 case (which was using a driver in the Leadville stack), the requests where satisfied from device nodes that were cached in the device tree. I wish I could prove this without a doubt, but unfortunately I can’t run Chris’ awesome scsi.d DTrace script on my Solaris 9 host. If one of the Sun storage engineers happens to come across this blog entry, please leave me a comment with your thoughts.

I have been repairing our backup environment for the past few weeks, and have encountered several nifty tools in the Netbackup volumen management bin directory. Once of these tools is the scan utility, which displays the robots and tape devices visible to a system:

$ /usr/openv/volmgr/bin/scan |more

************************************************************
*********************** SDT_TAPE    ************************
*********************** SDT_CHANGER ************************
*********************** SDT_OPTICAL ************************
************************************************************
------------------------------------------------------------
Device Name  : "/dev/rmt/0cbn"
Passthru Name: "/dev/sg/c0tw500104f0005f027cl0"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "STK     T9940B          1.35"
Vendor ID  : "STK     "
Product ID : "T9940B          "
Product Rev: "1.35"
Serial Number: "479000037011"
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 10
Removable      : Yes
Device Supports: SCSI-3
Flags : 0x4
Reason: 0x0

                      <.....>

This utility is extremely useful for getting the device paths for a specific tape device, and for viewing the information returned from a SCSI INQUIRY command. Viva la scan!

I ventured out recently to see the The English beat, Matisyahu and 311. Based on some logistical issues, I didn’t get to see The English Beat or Matisyahu, but I was able to to see the entire 311 show (311 is one of my favorite bands, so I was stoked I got to see their entire set). The boys from Omaha put on a great show, and Doug and Nick sounded awesome! If you haven’t listened to 311 before, their music can best be described as a mix of hip-hop, funk and rock and roll.

The set list for the evening was packed with a number of tunes from transister, and contained classic hits like “Beautiful Disaster,” “Amber,” “All Mixed Up,” “Down,” “Don’t Tread on Me,” “Hive,” “Feels So Good,” “Prisoner,” “Homebrew” and “Beyond the Gray Sky.” All in all the show was fantastic, and I was fortunate to have some incredible folks with me to cheer on the band! It was also cool to see the vast majority of the crowd singing along, since audience participation can make or break a show. Niiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiice!

While perusing my mailing lists this morning, I came across a comment from Ellard Roush on running Oracle RAC inside a Solaris zone:

“There is an active project to support Oracle RAC running in a zone environment. This new feature will be a “Zone Cluster”, which is a virtual cluster where each virtual node is a Zone on a different physical node. The Zone Cluster will be able to support cluster applications in general. From the perspective of the application, the Zone Cluster will appear to be a cluster dedicated to that application.”

I noticed that the “cluster” branded zone type was putback in build 67, and the fact that Sun is going to implement virtual nodes rocks! This will allow you to completley abstract the cluster nodes from the hardware, and will offer a TON of additional flexibility. I can’t wait to play with this goodness!

As a conscientious Solaris administrator, I make every attempt possible to protect my servers from malicious users. This includes disabling all unneeded services, enabling strong password policies, configuring system auditing, enabling strong network defaults, applying system patches and configuring system logging. When I configure system logging, I like to configure the syslogd daemon to log everything to a centralized location. This is typically accomplished by adding an entry similar to the following to /etc/syslog.conf:

*.debug @logserver.prefetch.net

Additionally, I like to log each time a user logs into my systems, as well as all attempts to su to another user. To log all su attempts, the file /var/adm/sulog can be created (in recent releases of Solaris, this file is created by default):

$ touch /var/adm/sulog

To log all successful and unsuccessful logins, you will first need to set the variable SYSLOG_FAILED_LOGINS in /etc/default/login to the value 0. Once the variable is adjusted, you will need to create a log file to store the login attempts:

$ touch /var/adm/loginlog

After the log file is created, the auth priority needs to be added to /etc/syslog.conf:

auth.debug /var/adm/loginlog

With the loginlog and sulog files in place, it is relativley easy to see who accessed a given system at time X, and who tried to become the super user.

I manage a number of Brocade switches, and periodically they encounter hardware problems (bad SPFs, faulty cables, etc.). To ensure that I am notified when these problems occur, I like to configure the switches to log errors to a centralized syslog server. Configuring a Brocade switch to use syslog is as simple as running the syslogdIpAdd command with the IP address of the syslog server:

switch1:admin> syslogdIpAdd “192.168.23.138″
Committing configuration…done.

switch1:admin> syslogdIpAdd “192.168.23.139″
Committing configuration…done.

Once one or more syslog servers are configured, the syslogdIpShow command can be used to verify that the servers were added:

switch1:admin> syslogdIpShow

syslog.IP.address.1: 192.168.23.138
syslog.IP.address.2: 192.168.23.139

If everything went smoothly, you should see entries similar to the following on the syslog server:

Jul 15 11:16:14 switch1 0 0x102d7800 (tShell): Jul 15 10:29:57
Jul 15 11:16:14 switch1      INFO SYS-LOGCLRD, 4, Error log cleared
Jul 15 11:16:14 switch1

The Brocade Silkworm family of switches maintain a number of important performance and reliability counters for each switch port. Performance and reliability counters for specific switch ports can be viewed with the porterrshow, portshow and portStatsShow commands. To view system wide throughput statistics, the portPerfShow command can be run with an optional interval:

switch1:admin> portPerfShow 5

      0    1    2    3    4    5    6    7    8    9   10   11   12   13   14   15  Total
    -------------------------------------------------------------------------------------
      0    0   21m  28m  31m   0  8.4m   0   28m  21m  31m   0  8.4m   0    0    0  178m
      0    0   20m  29m  31m   0   10m   0   29m  20m  31m   0   10m   0    0    0  182m
      0    0   18m  36m  31m   0   14m   0   36m  18m  31m   0   14m   0    0    0  201m
      0    0   17m  34m  30m   0  7.0m   0   34m  17m  31m   0  7.0m   0    0    0  179m

The individual switch ports are listed above the dashes, and the numbers below the dashes contain the number of bytes transmitted and received per second. This is a super useful tool for getting a high level overview of what your switch is doing.

Brocade switches have become one of the most widely deployed componets in most Storage Area Networks (SANs). One thing that has led to Brocade’s success is their robust CLI, which allow you to view and modify almost every aspect of their switch. This includes zoning configurations, SNMP attributes, domain ids, switch names and network addresses, etc. All of this configuration information is necessary for the switch to function properly, and should be periodically backed up to allow speedy recovery when disaster hits.

Each Brocade switch comes with the “configUpload” and “configDownload” commands to back up a switch configuration to a remote system, or to restore a configuration from a remote system. ConfigUplaod has two modes of oepration: interactive mode and automatic mode. To use the interactive mode to upload a config from a switch named switch1 to an ftp server with the IP address 1.2.3.4, configUpload can be run to walk you through backing up the configuration:

switch1:admin> configupload
Server Name or IP Address [host]: 1.2.3.4
User Name [user]: matty
File Name [config.txt]: switch1_config.txt
Protocol (RSHD or FTP) [rshd]: ftp
Password:
upload complete

After the configuration is uploaded, you will have a text file with you switches configuration on the remove server:

$ ls -l sw*

-rw-r--r--   1 matty    other       7342 Jul  7 09:15 switch1_config.txt

To restore a configuration, you can use the configDownload command. Both of these commands allow the paramters to be passed as arguments to the script, so they are ideal for automation (there is a backup script on the Brocade support site that can be used to automate configuration backups). In case others find it useful, I placed my Brocade cheat sheet on my website.