The value of an all-in-one storage stack

I have written several times about the Solaris Leadvilel storage stack. While debugging a problem on a Solaris 9 host that wasn’t using the Leadville stack, I would wait and wait and wait for various commands to complete. Here is one such example:

$ timex /usr/openv/volmgr/bin/scan >/dev/null

real        2:25.09
user           0.10
sys            0.45

After migrating the same host to Solaris 10 and the Leadville stack, I no longer had to wait for my commands to complete. Here is an example:

$ timex /usr/openv/volmgr/bin/scan >/dev/null

real           0.11
user           0.05
sys            0.03

After doing a bit of debugging, I noticed that the Solaris 9 fibre channel stack (which was using a vendor supplied HBA driver) was enumerating each path to see if devices were present. In the Solaris 10 case (which was using a driver in the Leadville stack), the requests where satisfied from device nodes that were cached in the device tree. I wish I could prove this without a doubt, but unfortunately I can’t run Chris’ awesome scsi.d DTrace script on my Solaris 9 host. If one of the Sun storage engineers happens to come across this blog entry, please leave me a comment with your thoughts.

Viewing tape devices on Solaris hosts

I have been repairing our backup environment for the past few weeks, and have encountered several nifty tools in the Netbackup volumen management bin directory. Once of these tools is the scan utility, which displays the robots and tape devices visible to a system:

$ /usr/openv/volmgr/bin/scan |more

************************************************************
*********************** SDT_TAPE    ************************
*********************** SDT_CHANGER ************************
*********************** SDT_OPTICAL ************************
************************************************************
------------------------------------------------------------
Device Name  : "/dev/rmt/0cbn"
Passthru Name: "/dev/sg/c0tw500104f0005f027cl0"
Volume Header: ""
Port: -1; Bus: -1; Target: -1; LUN: -1
Inquiry    : "STK     T9940B          1.35"
Vendor ID  : "STK     "
Product ID : "T9940B          "
Product Rev: "1.35"
Serial Number: "479000037011"
WWN          : ""
WWN Id Type  : 0
Device Identifier: ""
Device Type    : SDT_TAPE
NetBackup Drive Type: 10
Removable      : Yes
Device Supports: SCSI-3
Flags : 0x4
Reason: 0x0

                      <.....>

This utility is extremely useful for getting the device paths for a specific tape device, and for viewing the information returned from a SCSI INQUIRY command. Viva la scan!

Concert review: 311

I ventured out recently to see the The English beat, Matisyahu and 311. Based on some logistical issues, I didn’t get to see The English Beat or Matisyahu, but I was able to to see the entire 311 show (311 is one of my favorite bands, so I was stoked I got to see their entire set). The boys from Omaha put on a great show, and Doug and Nick sounded awesome! If you haven’t listened to 311 before, their music can best be described as a mix of hip-hop, funk and rock and roll.

The set list for the evening was packed with a number of tunes from transister, and contained classic hits like “Beautiful Disaster,” “Amber,” “All Mixed Up,” “Down,” “Don’t Tread on Me,” “Hive,” “Feels So Good,” “Prisoner,” “Homebrew” and “Beyond the Gray Sky.” All in all the show was fantastic, and I was fortunate to have some incredible folks with me to cheer on the band! It was also cool to see the vast majority of the crowd singing along, since audience participation can make or break a show. Niiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiice!

Running Oracle RAC inside Solaris zones

While perusing my mailing lists this morning, I came across a comment from Ellard Roush on running Oracle RAC inside a Solaris zone:

“There is an active project to support Oracle RAC running in a zone environment. This new feature will be a “Zone Cluster”, which is a virtual cluster where each virtual node is a Zone on a different physical node. The Zone Cluster will be able to support cluster applications in general. From the perspective of the application, the Zone Cluster will appear to be a cluster dedicated to that application.”

I noticed that the “cluster” branded zone type was putback in build 67, and the fact that Sun is going to implement virtual nodes rocks! This will allow you to completley abstract the cluster nodes from the hardware, and will offer a TON of additional flexibility. I can’t wait to play with this goodness!

Logging su attempts and failed logins

As a conscientious Solaris administrator, I make every attempt possible to protect my servers from malicious users. This includes disabling all unneeded services, enabling strong password policies, configuring system auditing, enabling strong network defaults, applying system patches and configuring system logging. When I configure system logging, I like to configure the syslogd daemon to log everything to a centralized location. This is typically accomplished by adding an entry similar to the following to /etc/syslog.conf:

*.debug @logserver.prefetch.net

Additionally, I like to log each time a user logs into my systems, as well as all attempts to su to another user. To log all su attempts, the file /var/adm/sulog can be created (in recent releases of Solaris, this file is created by default):

$ touch /var/adm/sulog

To log all successful and unsuccessful logins, you will first need to set the variable SYSLOG_FAILED_LOGINS in /etc/default/login to the value 0. Once the variable is adjusted, you will need to create a log file to store the login attempts:

$ touch /var/adm/loginlog

After the log file is created, the auth priority needs to be added to /etc/syslog.conf:

auth.debug /var/adm/loginlog

With the loginlog and sulog files in place, it is relativley easy to see who accessed a given system at time X, and who tried to become the super user.

Configuring Brocade switches to log to syslog

I manage a number of Brocade switches, and periodically they encounter hardware problems (bad SPFs, faulty cables, etc.). To ensure that I am notified when these problems occur, I like to configure the switches to log errors to a centralized syslog server. Configuring a Brocade switch to use syslog is as simple as running the syslogdIpAdd command with the IP address of the syslog server:

switch1:admin> syslogdIpAdd “192.168.23.138”
Committing configuration…done.

switch1:admin> syslogdIpAdd “192.168.23.139”
Committing configuration…done.

Once one or more syslog servers are configured, the syslogdIpShow command can be used to verify that the servers were added:

switch1:admin> syslogdIpShow

syslog.IP.address.1: 192.168.23.138
syslog.IP.address.2: 192.168.23.139

If everything went smoothly, you should see entries similar to the following on the syslog server:

Jul 15 11:16:14 switch1 0 0x102d7800 (tShell): Jul 15 10:29:57
Jul 15 11:16:14 switch1      INFO SYS-LOGCLRD, 4, Error log cleared
Jul 15 11:16:14 switch1