If you’re a frequent user of the bash shell, I would suggest taking a peek at the GNU reference guide next time you have a chance. There are a lot of cool built in functions/commands within bash that are pretty neat. To get an idea of what these built in commands are:
$ ps
PID TTY TIME CMD
15997 pts/2 00:00:00 bash
23625 pts/2 00:00:00 ps
$ which enable
/usr/bin/which: no enable in (/bin:/usr/bin:/usr/sbin:/sbin)
$ enable
enable .
enable :
enable [
enable alias
enable bg
enable bind
enable break
enable builtin
enable caller
enable cd
enable command
enable compgen
enable complete
enable continue
enable declare
enable dirs
enable disown
enable echo
enable enable
enable eval
enable exec
enable exit
enable export
enable false
enable fc
enable fg
enable getopts
enable hash
enable help
enable history
enable jobs
enable kill
enable let
enable local
enable logout
enable popd
enable printf
enable pushd
enable pwd
enable read
enable readonly
enable return
enable set
enable shift
enable shopt
enable source
enable suspend
enable test
enable times
enable trap
enable true
enable type
enable typeset
enable ulimit
enable umask
enable unalias
enable unset
enable wait
Some of these do have binaries within the /usr or /bin namespace, while others do not. Bash’s internal built in definition of these commands is what actually gets executed…
$ which cd
/usr/bin/which: no cd in (/bin:/usr/bin:/usr/sbin:/sbin)
$ which echo
/bin/echo
$ which eval
/usr/bin/which: no eval in (/bin:/usr/bin:/usr/sbin:/sbin)
$ which exit
/usr/bin/which: no exit in (/bin:/usr/bin:/usr/sbin:/sbin)
$ which kill
/bin/kill
Some of these are pretty intuitive why they should be built-in commands (cd, exit, etc.) Looking through the GNU reference guide describes what all of these do. There is also a built-in called “help” which describes some of this without going to the GNU reference guide.
$ which help
/usr/bin/which: no help in (/bin:/usr/bin:/usr/sbin:/sbin)
$ help cd
cd: cd [-L|-P] [dir]
Change the current directory to DIR. The variable $HOME is the
default DIR. The variable CDPATH defines the search path for
the directory containing DIR. Alternative directory names in CDPATH
are separated by a colon (:). A null directory name is the same as
the current directory, i.e. `.'. If DIR begins with a slash (/),
then CDPATH is not used. If the directory is not found, and the
shell option `cdable_vars' is set, then try the word as a variable
name. If that variable has a value, then cd to the value of that
variable. The -P option says to use the physical directory structure
instead of following symbolic links; the -L option forces symbolic
links
to be followed.
I found the “bind” built in to be super useful. Bash has shortcuts that allow you to move the cursor around, search history, etc. and I always find myself forgetting what the keyboard shortcut / sequence is. When this happens, I just use the “bind” (not DNS!) built in to find what I’m looking for…
$ bind -p
Displays all “bindings” of keyboard shortcuts to variables. Here, I want to see what keyboard shortcut I can use to search through my command history to find a command…
$ bind -p | grep reverse-search-history
“\C-r”: reverse-search-history
Ah, so its Ctrl-r.. Sure enough, hitting ctrl-r at a command prompt brings up the reverse search prompt through my history and I can look for any command… Looking for the command “cat” seems to have found me looking through /etc/passwd.
$ ``
(reverse-i-search)`cat’: cat /etc/passwd
Hitting ctrl-r multiple times once command is found will continue to search backwards through the history.
(reverse-i-search)`cat’: cat /proc/kallsyms | more
Reverse history search is pretty useful, but there are so many other neat features of bash that I find invaluable. What if I have created some super long command with multiple pipes and I want to move the cursor to the beginning of the line?
$ bind -p | grep beginning-of-line
“\C-a”: beginning-of-line
“\eOH”: beginning-of-line
“\e[1~”: beginning-of-line
“\e[H”: beginning-of-line
Well, it looks like there are a few defined shortcuts, but ctrl-a seems to be a winner. Sure enough..
$ cat /etc/passwd | awk -F: '{print 7}' | sed 's,bin,mike,'[]
$ [c]at /etc/passwd | awk -F: '{print 7}' | sed 's,bin,mike,'
Take a peek! You might find yourself saving some keystrokes.
$ help bind
bind: bind [-lpvsPVS] [-m keymap] [-f filename] [-q name] [-u name] [-r
keyseq] [-x keyseq:shell-command] [keyseq:readline-function or
readline-command]
Bind a key sequence to a Readline function or a macro, or set
a Readline variable. The non-option argument syntax is equivalent
to that found in ~/.inputrc, but must be passed as a single argument:
bind '"\C-x\C-r": re-read-init-file'.
bind accepts the following options:
-m keymap Use `keymap' as the keymap for the duration of
this
command. Acceptable keymap names are emacs,
emacs-standard, emacs-meta, emacs-ctlx, vi, vi-move,
vi-command, and vi-insert.
-l List names of functions.
-P List function names and bindings.
-p List functions and bindings in a form that can be
reused as input.
-r keyseq Remove the binding for KEYSEQ.
-x keyseq:shell-command Cause SHELL-COMMAND to be executed when
KEYSEQ is entered.
-f filename Read key bindings from FILENAME.
-q function-name Query about which keys invoke the named function.
-u function-name Unbind all keys which are bound to the named
function.
-V List variable names and values
-v List variable names and values in a form that can
be reused as input.
-S List key sequences that invoke macros and their
values
-s List key sequences that invoke macros and their
values
in a form that can be reused as input.
Want to see if your current shell has history enabled, or some other key bash features? The GNU Bash Reference guide has details on what all of these variables are.. Like checkwinsize.
$ shopt
cdable_vars off
cdspell off
checkhash off
checkwinsize on
cmdhist on
dotglob off
execfail off
expand_aliases on
extdebug off
extglob off
extquote on
failglob off
force_fignore on
gnu_errfmt off
histappend off
histreedit off
histverify off
hostcomplete on
huponexit off
interactive_comments on
lithist off
login_shell on
mailwarn off
no_empty_cmd_completion off
nocaseglob off
nocasematch off
nullglob off
progcomp on
promptvars on
restricted_shell off
shift_verbose off
sourcepath on
xpg_echo off
$ help shopt
shopt: shopt [-pqsu] [-o long-option] optname [optname...]
Toggle the values of variables controlling optional behavior.
The -s flag means to enable (set) each OPTNAME; the -u flag
unsets each OPTNAME. The -q flag suppresses output; the exit
status indicates whether each OPTNAME is set or unset. The -o
option restricts the OPTNAMEs to those defined for use with
`set -o'. With no options, or with the -p option, a list of all
settable options is displayed, with an indication of whether or
not each is set.
checkwinsize
If set, Bash checks the window size after each command and, if
necessary, updates the values of LINES and COLUMNS.
Ah nice. My shell wont send a SIGHUP to jobs while i’m logging out of an interactive shell. =)
While attempting to install a Sun package this week, I encountered the following error:
$ pkgadd -d . MYpackage
## Waiting for up to <300> seconds for package administration
commands to become available (another user is administering packages on
zone <zoneA>)
^C
1 package was not processed!
After a bit of truss’ing, I noticed that the pkgadd commands were checking for the existence of files with the name .ai.pkg.zone.lock.<DYNAMICALLY_GENERATED_STRING> in /tmp. Based on a cursory inspection of the package utility source code, it appears these files are used as lock files to prevent multiple package commands from running at the same time. Since this was the only package installation running on the system, I logged into the zone and removed the stale lock file:
$ zlogin zoneA
$ rm /tmp/.ai.pkg.zone.lock-afdb66cf-1dd1-11b2-a049-000d560ddc3e
Once I removed this file, the package installed like a champ! Nice!
Most SSH clients have the ability to perform local and remote port forwarding. This is a pretty neat use of SSH if you haven’t ever seen it before. OpenSSH can take it one step further and provide a full VPN solution encrypting all network traffic on all ports between two machines. This is pretty powerful stuff. This is useful for a quick-and-dirty way to encrypt all traffic between two machines. For a longer term solution, you might want to check out how to configure IPsec or use OpenVPN. All three solutions have some really cool features and benefits.
OpenSSH is the most widely deployed open source SSH client / server solution today. Most Linux/BSD hosts I have encountered will use this as the client/server by default. Sun’s ssh packages are based off of the OpenSSH distribution with some tweaks and modifications, but its pretty close to OpenSSH’s implementation.
Anyways, to create a VPN tunnel between two machines, two variables in sshd_config need to be tweaked as well as the presence of the tun/tap kernel module. This kernel module is available on most Linux/BSD distributions. It may have to be compiled and inserted into the Solaris kernel, or you can download it here.
[root@locutus ~]# uname -a Linux locutus 2.6.25-14.fc9.i686 #1 SMP Thu May 1 06:28:41 EDT 2008 i686 i686 i386 GNU/Linux [root@locutus ~]# lsmod | grep tun tun 11776 2 [root@locutus ~]# modinfo tun filename: /lib/modules/2.6.25-14.fc9.i686/kernel/drivers/net/tun.ko alias: char-major-10-200 license: GPL author: (C) 1999-2004 Max Krasnyansky maxk@qualcomm.com description: Universal TUN/TAP device driver srcversion: 12C02361DF16200902CDE64 depends: vermagic: 2.6.25-14.fc9.i686 SMP mod_unload 686 4KSTACKS So, the tun module has already been inserted into my running kernel. Next, set these two variables in sshd_config and have sshd re-read its configuration files…
[root@locutus ~]# egrep ‘PermitTunnel|PermitRootLogin’ /etc/ssh/sshd_config PermitRootLogin yes PermitTunnel yes
[root@locutus ~]# service sshd reload Reloading sshd: [ OK ] [root@locutus ~]#
Make sure you don’t mess up sshd_config and reload the daemon as your only way to access the machine! Console access is always a good thing.
Now all we need to do is to open the VPN tunnel itself. Here, I open a VPN tunnel to localhost (not really useful) but you can get the idea…
[root@locutus ~]# ssh -w any:any root@localhost
The any:any defines the local:remote “tun” device. We could have put 0:1 here, (tun0 as the local, tun1 as the remote) but any:any takes care of it for us in case there are any pre-existing tun devices in use.) or 0:0 if we were accessing a real remote machine. (I can’t define two “tun0” devices on localhost)
So, after my SSH session connects, sure enough the tun devices exist..
[root@locutus ~]# ifconfig tun0 tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 POINTOPOINT NOARP MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
[root@locutus ~]# ifconfig tun1 tun1 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 POINTOPOINT NOARP MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
Now all we have to do is assign them some network addresses and let them know that its a point-to-point connection between the two…
[root@locutus ~]# ifconfig tun0 10.0.0.10 pointopoint 10.0.0.11 [root@locutus ~]# ifconfig tun1 10.0.0.11 pointopoint 10.0.0.10
[root@locutus ~]# ifconfig tun0 tun0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 inet addr:10.0.0.10 P-t-P:10.0.0.11 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
[root@locutus ~]# ifconfig tun1 tun1 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 inet addr:10.0.0.11 P-t-P:10.0.0.10 Mask:255.255.255.255 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) Nice! Now all network traffic between the machines using those newly created addresses will be tunneled through SSH! Could this be a solution for NFSv3 and firewalls? ;-)
If there are multiple machines on the “other side” of the VPN that you would want to connect to, you will also need to add a route..
[root@locutus ~]# route add -net 10.0.0.0 netmask 255.255.255.0 gw 10.0.0.10 tun0
[root@locutus ~]# netstat -rn Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 10.0.0.11 0.0.0.0 255.255.255.255 UH 0 0 0 tun0 10.0.0.10 0.0.0.0 255.255.255.255 UH 0 0 0 tun1 10.0.0.0 10.0.0.10 255.255.255.0 UG 0 0 0 tun0
So any traffic destined for 10.0.0.0/24 is gonna go out tun0 to be routed onwards and upwards.
We’ve also got to set up an arp entry..
[root@locutus ~]# arp -sD 10.0.0.11 eth0 pub
[root@locutus ~]# arp -an ? (10.0.0.11) at PERM PUP on eth0
Ryan McGuire wrote a pretty cool blog entryabout not only this feature, but some other really neat things with OpenSSH. I based a lot of this article after learning them from his site. Check out his python script that will automate a lot of this for you. Thanks Ryan!
In a SAN environment when dealing with external storage concepts such as EMC BCV’s, you’ll often have a request to create volumes on two different machines that are identical so replication on the back-end can occur.
When you look at a LUN presented to Solaris, it’ll appear with a cryptic name like the following:
103. c20t60060480000190100665533030393836d0 <EMC-SYMMETRIX-5771 cyl
36826 alt 2 hd 60 sec 128>
<a>/scsi_vhci/ssd@g60060480000190100665533030353339</a>
The c20 relates to the HBA (Fiber, SCSI, iSCSI) that provides a path to the device. The “middle” sequence 60060480000190100665533030393836 between the “t (target)” and “d (device” is the WWN of the LUN.
Now, say your SAN engineer approaches you with some information like the following….
PROD DEV: 936
PROD LUN: 47
STAGE DEV: 986
STAGE LUN: 68
Ok… so what does that mean to us? Using luxadm, we can probe a target to find out some specifc information about its back-end LUN name.
(root)> luxadm display
/dev/rdsk/c20t60060480000190100665533030393836d0s2
DEVICE PROPERTIES for disk:
/dev/rdsk/c20t60060480000190100665533030393836d0s2
Vendor: EMC
Product ID: SYMMETRIX
Revision: 5771
Serial Num: xxxxxxxxxxx
Unformatted capacity: 138105.000 MBytes
Read Cache: Enabled
Minimum prefetch: 0x0
Maximum prefetch: 0xffff
Device Type: Disk device
Path(s):
/dev/rdsk/c20t60060480000190100665533030393836d0s2
<a>/devices/scsi_vhci/ssd@g60060480000190100665533030393836:c,raw</a>
Controller
<a>/devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0</a>
** Device Address 50060482d52d2e76,68**
Host controller port WWN 210100e08ba0147a
Class primary
State ONLINE
Controller
<a>/devices/pci@17c,600000/SUNW,qlc@1,1/fp@0,0</a>
** Device Address 50060482d52d2e59,68
** Host controller port WWN 210100e08ba0da73
Class primary
State ONLINE
The Device Address is the field we are interested in. This shows us that the WWN of the Port we are plugged into is 50060482d52d2e59and the LUN number is 68.
Now that we have the LUN number, we know what this LUN maps to. We can then find the cooresponding LUN on the other machine (LUN Number 47) which maps to the BCV pair.
There are some other useful leadville commands that you may be interested in…
To display all HBAs available for use:
(root)> luxadm -e port
<a>/devices/pci@15c,600000/SUNW,qlc@1/fp@0,0:devctl</a>
NOT CONNECTED
<a>/devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl</a>
CONNECTED
<a>/devices/pci@15d,600000/SUNW,qlc@1,1/fp@0,0:devctl</a>
NOT CONNECTED
<a>/devices/pci@15d,600000/SUNW,qlc@1/fp@0,0:devctl</a>
NOT CONNECTED
<a>/devices/pci@17c,600000/SUNW,qlc@1/fp@0,0:devctl</a>
NOT CONNECTED
<a>/devices/pci@17c,600000/SUNW,qlc@1,1/fp@0,0:devctl</a>
CONNECTED
<a>/devices/pci@17d,600000/SUNW,qlc@1/fp@0,0:devctl</a>
NOT CONNECTED
<a>/devices/pci@17d,600000/SUNW,qlc@1,1/fp@0,0:devctl</a>
NOT CONNECTED
<a>/devices/pci@19c,600000/SUNW,qlc@1/fp@0,0:devctl</a>
NOT CONNECTED
<a>/devices/pci@19c,600000/SUNW,qlc@1,1/fp@0,0:devctl</a>
NOT CONNECTED
<a>/devices/pci@19d,600000/SUNW,qlc@1/fp@0,0:devctl</a>
NOT CONNECTED
<a>/devices/pci@19d,600000/SUNW,qlc@1,1/fp@0,0:devctl</a>
NOT CONNECTED
Now that you have the device name, you can map that back to what device it cooralates to under /dev.
In this case since i’m using a Fiber channel HBA…
(root)> ls -l /dev/fc | grep
<a>/devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl</a>
lrwxrwxrwx 1 root root 55 Sep 26 2007 **fp1 ->**
<../../devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl>
Sweet. So one of my HBAs is fp1.
Want to see more detailed information about what that Fiber HBA is connected to?
umt1a-bio-stg1:~
(root)> luxadm -e dump_map
<a>/devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl</a>
Pos Port_ID Hard_Addr Port WWN Node WWN Type
0 611813 0 50060e8004769000 50060e8004769000 0x0 (Disk
device)
1 624613 0 50060482d52d2e76 50060482d52d2e76 0x0 (Disk
device)
2 617c13 0 210100e08ba0147a 200100e08ba0147a 0x1f (Unknown
Type,Host Bus Adapter)
Note that the Port WWN 50060482d52d2e76 is the same WWN we saw above when looking for the LUN number.
Want a dump of all LUNs attached to a controller?
(root)> cfgadm -o show_FCP_dev -al
Ap_Id Type Receptacle Occupant
Condition
c4::50060482d52d2e76,0 disk connected configured
unknown
c4::50060482d52d2e76,69 disk connected configured
unknown
c4::50060482d52d2e76,70 disk connected configured
unknown
c4::50060482d52d2e76,71 disk connected configured
unknown
c4::50060482d52d2e76,72 disk connected configured
unknown
c4::50060482d52d2e76,73 disk connected configured
unknown
c4::50060482d52d2e76,74 disk connected configured
unknown
c4::50060482d52d2e76,75 disk connected configured
unknown
c4::50060482d52d2e76,76 disk connected configured
unknown
c4::50060482d52d2e76,77 disk connected configured
unknown
c4::50060482d52d2e76,78 disk connected configured
unknown
c4::50060482d52d2e76,79 disk connected configured
unknown
c4::50060482d52d2e76,80 disk connected configured
unknown
<snip>
....
Another super useful utility is the new fcinfo command which was introduced into Solaris 10. The -l (linkstat) shows some detailed statistics on the HBA.
(root)> fcinfo hba-port -l
HBA Port WWN: 210100e08ba0147a
OS Device Name: /dev/cfg/c4
Manufacturer: QLogic Corp.
Model: 375-3108-xx
Firmware Version: 3.3.26
FCode/BIOS Version: fcode: 1.13;
Type: N-port
State: online
Supported Speeds: 1Gb 2Gb
Current Speed: 2Gb
Node WWN: 200100e08ba0147a
Link Error Statistics:
Link Failure Count: 0
Loss of Sync Count: 0
Loss of Signal Count: 0
Primitive Seq Protocol Error Count: 0
Invalid Tx Word Count: 0
Invalid CRC Count: 0
With the WWN of the HBA, we can query the remote port information…
(root)> fcinfo remote-port -p 210100e08ba0147a
Remote Port WWN: 50060482d52d2e76
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 50060482d52d2e76
Remote Port WWN: 50060e8004769000
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 50060e8004769000
Check it out! Its the same 50060482d52d2e76 we saw twice before with the WWN of the port we’re plugged into the fabric with.
Throw in a -s, and it’ll return all SCSI targets with their LUN Number.
(root)> fcinfo remote-port -p 210100e08ba0147a -s
Remote Port WWN: 50060482d52d2e76
Active FC4 Types: SCSI
SCSI Target: yes
Node WWN: 50060482d52d2e76
LUN: 0
Vendor: EMC
Product: SYMMETRIX
OS Device Name: /dev/rdsk/c4t50060482D52D2E76d0s2
LUN: 69
Vendor: EMC
Product: SYMMETRIX
OS Device Name:
/dev/rdsk/c20t60060480000190100665533030344539d0s2
LUN: 70
Vendor: EMC
Product: SYMMETRIX
OS Device Name:
/dev/rdsk/c20t60060480000190100665533030344639d0s2
LUN: 71
Vendor: EMC
Product: SYMMETRIX
OS Device Name:
/dev/rdsk/c20t60060480000190100665533030353039d0s2
LUN: 72
Vendor: EMC
Product: SYMMETRIX
OS Device Name:
/dev/rdsk/c20t60060480000190100665533030353139d0s2
LUN: 73
Vendor: EMC
Product: SYMMETRIX
OS Device Name:
/dev/rdsk/c20t60060480000190100665533030353239d0s2
Want to force a specific Fiber HBA to reinitialized and re-login into the Fabric?
(root)> luxadm -e forcelip <device>
i.e.
(root)> luxadm -e forcelip
<a>/devices/pci@15c,600000/SUNW,qlc@1,1/fp@0,0:devctl</a>
The flag day for ZFS boot supportwas just announced which will allow for root file systems (/, /var, /usr) to be bootable from both SPARC and x86 platforms. It looks like this functionality is going to come into OpenSolaris at build 88. The install support (selecting ZFS file systems from a jumpstart profile) or from optical media looks like it’ll make its way into build 89.
A lot of people have been waiting to play with this on SPARC platforms. =) Its exciting stuff.
If you would like to find out some more information about what changes were required to the boot process to allow for ZFS root, check out the ZFS-Boot project’s website on OpenSolaris found here.