The wonderful world of Leadville

In a SAN environment when dealing with external storage concepts such as EMC BCV’s, you’ll often have a request to create volumes on two different machines that are identical so replication on the back-end can occur.


When you look at a LUN presented to Solaris, it’ll appear with a cryptic name like the following:

 103. c20t60060480000190100665533030393836d0 <EMC-SYMMETRIX-5771 cyl 36826 alt 2 hd 60 sec 128>

The c20 relates to the HBA (Fiber, SCSI, iSCSI) that provides a path to the device.  The “middle” sequence 60060480000190100665533030393836 between the “t (target)” and “d (device” is the WWN of the LUN.


Now, say your SAN engineer approaches you with some information like the following….






Ok… so what does that mean to us?  Using luxadm, we can probe a target to find out some specifc information about its back-end LUN name.

(root)> luxadm display /dev/rdsk/c20t60060480000190100665533030393836d0s2
DEVICE PROPERTIES for disk: /dev/rdsk/c20t60060480000190100665533030393836d0s2
  Vendor:               EMC
  Product ID:           SYMMETRIX
  Revision:             5771
  Serial Num:           xxxxxxxxxxx
  Unformatted capacity: 138105.000 MBytes
  Read Cache:           Enabled
    Minimum prefetch:   0x0
    Maximum prefetch:   0xffff
  Device Type:          Disk device

   Controller           /devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0
    Device Address              50060482d52d2e76,68
    Host controller port WWN    210100e08ba0147a
    Class                       primary
    State                       ONLINE
   Controller           /devices/pci@17c,600000/SUNW,qlc@1,1/fp@0,0
    Device Address              50060482d52d2e59,68
    Host controller port WWN    210100e08ba0da73
    Class                       primary
    State                       ONLINE

The Device Address is the field we are interested in.  This shows us that the WWN of the Port we are plugged into is 50060482d52d2e59 and the LUN number is 68.

Now that we have the LUN number, we know what this LUN maps to.  We can then find the cooresponding LUN on the other machine (LUN Number 47) which maps to the BCV pair.


There are some other useful leadville commands that you may be interested in…

To display all HBAs available for use:

(root)> luxadm -e port
/devices/pci@15c,600000/SUNW,qlc@1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl                 CONNECTED
/devices/pci@15d,600000/SUNW,qlc@1,1/fp@0,0:devctl                 NOT CONNECTED
/devices/pci@15d,600000/SUNW,qlc@1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@17c,600000/SUNW,qlc@1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@17c,600000/SUNW,qlc@1,1/fp@0,0:devctl                 CONNECTED
/devices/pci@17d,600000/SUNW,qlc@1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@17d,600000/SUNW,qlc@1,1/fp@0,0:devctl                 NOT CONNECTED
/devices/pci@19c,600000/SUNW,qlc@1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@19c,600000/SUNW,qlc@1,1/fp@0,0:devctl                 NOT CONNECTED
/devices/pci@19d,600000/SUNW,qlc@1/fp@0,0:devctl                   NOT CONNECTED
/devices/pci@19d,600000/SUNW,qlc@1,1/fp@0,0:devctl                 NOT CONNECTED

Now that you have the device name, you can map that back to what device it cooralates to under /dev.

In this case since i’m using a Fiber channel HBA…

(root)> ls -l /dev/fc | grep /devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl
lrwxrwxrwx   1 root     root          55 Sep 26  2007 fp1 -> ../../devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl

Sweet.  So one of my HBAs is fp1.

Want to see more detailed information about what that Fiber HBA is connected to?

(root)> luxadm -e dump_map /devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl
Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type
0    611813  0         50060e8004769000 50060e8004769000 0x0  (Disk device)
1    624613  0         50060482d52d2e76 50060482d52d2e76 0x0  (Disk device)
2    617c13  0         210100e08ba0147a 200100e08ba0147a 0x1f (Unknown Type,Host Bus Adapter)

Note that the Port WWN 50060482d52d2e76 is the same WWN we saw above when looking for the LUN number. 


Want a dump of all LUNs attached to a controller?

(root)> cfgadm -o show_FCP_dev -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c4::50060482d52d2e76,0         disk         connected    configured   unknown
c4::50060482d52d2e76,69        disk         connected    configured   unknown
c4::50060482d52d2e76,70        disk         connected    configured   unknown
c4::50060482d52d2e76,71        disk         connected    configured   unknown
c4::50060482d52d2e76,72        disk         connected    configured   unknown
c4::50060482d52d2e76,73        disk         connected    configured   unknown
c4::50060482d52d2e76,74        disk         connected    configured   unknown
c4::50060482d52d2e76,75        disk         connected    configured   unknown
c4::50060482d52d2e76,76        disk         connected    configured   unknown
c4::50060482d52d2e76,77        disk         connected    configured   unknown
c4::50060482d52d2e76,78        disk         connected    configured   unknown
c4::50060482d52d2e76,79        disk         connected    configured   unknown
c4::50060482d52d2e76,80        disk         connected    configured   unknown



Another super useful utility is the new fcinfo command which was introduced into Solaris 10.  The -l (linkstat) shows some detailed statistics on the HBA.

(root)> fcinfo hba-port -l
HBA Port WWN: 210100e08ba0147a
        OS Device Name: /dev/cfg/c4
        Manufacturer: QLogic Corp.
        Model: 375-3108-xx
        Firmware Version: 3.3.26
        FCode/BIOS Version:  fcode: 1.13;
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb
        Current Speed: 2Gb
        Node WWN: 200100e08ba0147a
        Link Error Statistics:
                Link Failure Count: 0
                Loss of Sync Count: 0
                Loss of Signal Count: 0
                Primitive Seq Protocol Error Count: 0
                Invalid Tx Word Count: 0
                Invalid CRC Count: 0


With the WWN of the HBA, we can query the remote port information…

(root)> fcinfo remote-port -p 210100e08ba0147a
Remote Port WWN: 50060482d52d2e76
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060482d52d2e76
Remote Port WWN: 50060e8004769000
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060e8004769000

Check it out!  Its the same 50060482d52d2e76 we saw twice before with the WWN of the port we’re plugged into the fabric with.


Throw in a -s, and it’ll return all SCSI targets with their LUN Number.

(root)> fcinfo remote-port -p 210100e08ba0147a -s
Remote Port WWN: 50060482d52d2e76
        Active FC4 Types: SCSI
        SCSI Target: yes
        Node WWN: 50060482d52d2e76
        LUN: 0
          Vendor: EMC
          Product: SYMMETRIX
          OS Device Name: /dev/rdsk/c4t50060482D52D2E76d0s2
        LUN: 69
          Vendor: EMC
          Product: SYMMETRIX
          OS Device Name: /dev/rdsk/c20t60060480000190100665533030344539d0s2
        LUN: 70
          Vendor: EMC
          Product: SYMMETRIX
          OS Device Name: /dev/rdsk/c20t60060480000190100665533030344639d0s2
        LUN: 71
          Vendor: EMC
          Product: SYMMETRIX
          OS Device Name: /dev/rdsk/c20t60060480000190100665533030353039d0s2
        LUN: 72
          Vendor: EMC
          Product: SYMMETRIX
          OS Device Name: /dev/rdsk/c20t60060480000190100665533030353139d0s2
        LUN: 73
          Vendor: EMC
          Product: SYMMETRIX
          OS Device Name: /dev/rdsk/c20t60060480000190100665533030353239d0s2

Want to force a specific Fiber HBA to reinitialized and re-login into the Fabric?

(root)> luxadm -e forcelip <device>


(root)> luxadm -e forcelip /devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl

ZFS boot support for SPARC / x86

The flag day for ZFS boot support was just announced which will allow for root file systems (/, /var, /usr) to be bootable from both SPARC and x86 platforms.  It looks like this functionality is going to come into OpenSolaris at build 88.   The install support (selecting ZFS file systems from a jumpstart profile) or from optical media looks like it’ll make its way into build 89.

A lot of people have been waiting to play with this on SPARC platforms.   =)  Its exciting stuff.

If you would like to find out some more information about what changes were required to the boot process to allow for ZFS root, check out the ZFS-Boot project’s website on OpenSolaris found here.

Respect my ~/.Xauthority !#@$!

South Park is a hilarious show, and I think that Cartman is the best character.  One of Cartman’s classic lines is “YOU WILL RESPECT MY AUTHORITAH!#!”

So Cartman wasn’t a unix geek and wasn’t talking about X11 Forwarding / SSH, but maybe there is a moral to the story.

You have to execute some sort of GUI program on a remote host and it requires root access in order to execute (or you have to change to a different user to execute the GUI with correct permissions)… 

At first, when you logged into the machine for the first time without X11 forwarding enabled, your ~/.Xauthority file doesn’t exist…

cartman@locutus:~$ ls -l ~/.Xauthority
ls: /home/cartman/.Xauthority: No such file or directory

So you log back out, and when you ssh back into the remote machine, you remember to forward X11 by issuing..

$ ssh -X <user>@<remote box>


$ ssh -X cartman@locutus
Linux locutus 2.6.22-14-generic #1 SMP Sun Oct 14 23:05:12 GMT 2007 i686
cartman@locutus:~$ xclock

Sure enough, you fire up /usr/bin/xclock (or /usr/openwin/bin/xclock) and verify that the GUI program displays back on your local desktop.  

cartman@locutus:~$ echo $DISPLAY

Sweet.  Next, when you change users..

cartman@locutus:~$ su –

root@locutus:~# id
uid=0(root) gid=0(root) groups=0(root)

root@locutus:~# xclock
Error: Can’t open display:


you loose your X11 forwarding.  DOH!

So whats the solution here?  You can’t log in directly to the box as the root user (this should always be disabled.  Its really bad practice if it isn’t) — and you don’t really want to throw a SSH key into /root/.ssh/authorized_keys for obvious reasons — so what’s there to do?

When you SSH into a machine with X11 forwarding, it opens a TCP port, tunnels it through SSHD, and stores this information into a MIT cookie file in your home directory called ~/.Xauthority

All we have to do is “move” this information along with us when we change users.  We can use the xauth command to manipulate this for us.  First, lets display what the value of our cookie is.  Note the :10 matching up to our $DISPLAY variable…

cartman@locutus:~$ xauth list
locutus/unix:10  MIT-MAGIC-COOKIE-1  e2cba22d040f0e75dcbd203ee40736de

Now lets change users..

cartman@locutus:~$ su –

root@locutus:~# ls -l /root/.Xauthority
ls: /root/.Xauthority: No such file or directory

root@locutus:~# xauth list

So no MIT cookies currently exist… That makes sense because we didn’t X11 port forward into the root account.. Lets add one.  Don’t forget the “/unix” after the FQDN..
root@locutus:~# xauth add locutus/unix:10 MIT-MAGIC-COOKIE-1

xauth:  creating new authority file /root/.Xauthority

root@locutus:~# xauth list
localhost/unix:10  MIT-MAGIC-COOKIE-1  e2cba22d040f0e75dcbd203ee40736de

We’re not done yet… The last thing we have to do is to set our $DISPLAY variable to the same display as above..  Right now it may be set to null…

root@locutus:~# echo $DISPLAY

root@locutus:~# xclock
Error: Can’t open display:

So lets set it to localhost:10.0

root@locutus:~# export DISPLAY=localhost:10
root@locutus:~# xclock

Sure enough, we get xclock to display.  We didn’t have to be the root user in this example.  Any local user could perform the same function.

Alternativly, xauth also has a “merge” function to where you can read an existing ~/<user/.Xauthority file to merge with another.  This really is only going to work for the root user (unless you chmod) because the permissions on this file is octal 600…

root@locutus:~# ls -l /home/cartman/.Xauthority
-rw——- 1 cartman cartman 53 2008-04-05 00:22 /home/cartman/.Xauthority

Lets remove the previous Xauthority we had in place…

root@locutus:~# xauth list
locutus/unix:10  MIT-MAGIC-COOKIE-1  e2cba22d040f0e75dcbd203ee40736de

root@locutus:~# xauth remove locutus/unix:10


And then we’ll use the merge function pointing at a specific .Xauthority file…
root@locutus:~# xauth merge /home/cartman/.Xauthority

Sure enough, it imported correctly..

root@locutus:~# xauth list
locutus/unix:10  MIT-MAGIC-COOKIE-1  e2cba22d040f0e75dcbd203ee40736de

Our DISPLAY variable matches the display above and xclock starts up without any errors.
root@locutus:~# echo $DISPLAY
root@locutus:~# xclock


When xauth is invoked without any options, it brings up a menu based configuration utility thats pretty neat… Here’s “xauth info” in action…

root@locutus:~# xauth
Using authority file /root/.Xauthority
xauth> help
    add dpyname protoname hexkey   add entry
    exit                           save changes and exit program
    extract filename dpyname…    extract entries into file
    help [topic]                   print help
    info                           print information about entries
    list [dpyname…]              list entries
    merge filename…              merge entries from files
    nextract filename dpyname…   numerically extract entries
    nlist [dpyname…]             numerically list entries
    nmerge filename…             numerically merge entries
    quit                           abort changes and exit program
    remove dpyname…              remove entries
    source filename                read commands from file
    ?                              list available commands
    generate dpyname protoname [options]  use server to generate entry
    options are:
      timeout n    authorization expiration time in seconds
      trusted      clients using this entry are trusted
      untrusted    clients using this entry are untrusted
      group n      clients using this entry belong to application group n
      data hexkey  auth protocol specific data needed to generate the entry

xauth> info
Authority file:       /root/.Xauthority
File new:             no
File locked:          no
Number of entries:    1
Changes honored:      yes
Changes made:         no
Current input:        (stdin):2

There are also all sorts of security implecations surrounding ~/.Xauthority where the root user or administrator could hijack X11 sessions.  This article is a great read and I suggest taking a look at it when you have a chance.  It also goes into better detail on the steps of how the X11 forward occurs and security hazards surrounding it.


Beer to be integrated into Nevada!!

I just came across the beer fastrack, which proposes to integrate beer(1) into opensolaris (I am disappointed that wine(1) is not in scope):

This case describes the proper usage model for the common zymological
beverage, beer(1).  Though the beverage 'wine(1)' has similar processes
and has similar results, the density and concentrations are different,
so this case will not include 'wine(1)'.  This case will also not
include common compression techniques (warming and cooling), but will
reference them.

 In ancient times, the Mesopotamians  discovered that grape juice left
out for some time changed into a fluid that, when consumed, cause
people to act giddy and talk loudly, but otherwise felt very happy.  A
curious group, they also experimented with other materials, and found
that using grains (rice, wheat, and barley), they could get a fluid
with a little less impact, and more of it could be used.  Grains were
also more abundant, so it was also more economical.  This became beer(1).

 The ancient Egyptians documented their procedures for constructing
beer(1), and has been reproduced in modern times.  Though the quality
of the reproduced product isn't as good as modern beer(1), much of the
effects were as they are today.

 In the 16th century, the French, being taxed on the volume of the
liquid, determined that the fluid can be compressed by applying heat
and cold to get the essence of the liquid.  The compressed liquid was
usable by itself, but decompression was never viable.

 The British also discovered that if the flower of the hop plant was
added, the stability of beer would improve.  A lot of hops allowed for
beer to remain stable for the long journey to India (hence, the "India
Pale Ale").

 Beer was also popular with the American settlers, where the first
commercial production facility was built in Hoboken New Jersey in the
late 16th century.  Perhaps it was the impact of this fluid that caused
brewers such as George Washington, Samuel Adams, Paul Revere, and
enthusiast Ben Franklin to dislike the British.  It might also be the
reason the Bostonians put tea in Boston harbor, and not in hot water.

 In 1857, Louis Pasteur made the connection of yeast with fermentation.
But it was the German chemist, Eduard Buchner, winner of the 1907 Nobel
Prize, that determined that it was the yeast enzyme, zymase, that
catalyzes glycolysis, and leads to the characteristics of beer(1).

 In 1985, the need for standards caused the certification group BJCP to
define standards and certify individuals that are qualified to validate
beer.  This standard is included in the case materials.

Technical Details
 The detailed formula for creation is:

    C6H12O6 + O2 +  => 2 CH3CH2OH + 2 C02 + 2ATP


   Sugar + Oxygen +  => Alcohol + Carbon Dioxide + energy

This is an aerobic reaction as it consumes oxygen, and the catalyst is
the zymase of yeast.

 Considering that the common sugars used have significantly different
characteristics, and the process of turning these sugars into compounds
that can easily be converted to beer, the end product will be dependent
on the actual starting components.

 For the correct product, it is important that the fungi species
Saccharomyces Cerevisiae be used, as other catalysts will impact the
formula and generate a completely different and undesirable product.

 Some of the detailed starting components or procedures are considered
proprietary and cannot be described in this case, but the common
process allows for and encourages this individuality, and actually
improves the usage.

The most common usage is:

     cd  /pub
    more beer

though recent usage has been:

    find /brewer -name beer -exec drink {} \;

Note that usage has the same results as discovered by the Mesopotamians,
and can lead to the usage of date(1) and/or join(1), and often leads to
the misuse of chat(1m).  Excessive usage can cause garbled verbage, user
data corruption, unexpected reduced power, decreased vertical stability,
or unintentional overflow.  Normal usage will cause a slight to moderate
ammonia buildup that will require regular flushing.

beer(1) is accessible to all provided that security constraints (below)
are followed.  Even people with disabilities can use and appreciate the
value of beer(1).

beer(1) already is internationalized, and is well understood by many in
all languages.  Usage and callouts vary in various languages, but general
charactierstics remain constant.  It also has a built in mechanism for
people of different languages to understand each other, though it can
lead to disagreements during sporting events (especially with Zebras).

beer(1) has the following authorizations:               Having this authorization allows
                                       the user to acquire beer(1).              Having this authorization allows
                                       the user to to pass audits.        Having this authorization denies
                                       the user access to beer(1) but
                                       allows the user to continue to
                                       use auto(1).

These should be considered weak authorizations, as any college
student can easily circumvent the auth.  And many
ignore the and
auths.  However, auditing is handled by federal agencies, and failure
to have these authorizations while using beer(1) (especially the use of
auto(1) while using beer(1)) can result in reduced monetary resources
and/or severely restricted access.

Imported Interfaces
     Name              Stability
     Beer              Volatile
     Sake              Volatile
     Barley Wine       Extremely Volatile

Exporting Interfaces
     Name              Stability
     Urethra           Standard (and important after using beer(1))

     Bottle            (7 to 22 US fluid ounces)
     Growler           (Half-gallon jug - but jek3 knows this)
     pony keg          (3 to 7.5 US gallons)
     Standard Keg      (15.5 US gallons a.k.a. half-barrel)
     Barrel (standard unit)  (31 US gallons)

I need to thank Randy Fishel for proposing this on the ARC mailing list, and for making me laugh on April fools day!