I use mpt-status to monitor LSI Logic RAID controllers in my Linux hosts. While attempting to build a package for CentOS 5.3, I ran into the following error:
$ gmake
KERNEL_PATH=/usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18.x86_64**
gcc -Iincl -Wall -W -O2
-I/usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18.x86_64/drivers/message/fusion
-I/lib/modules/`uname -r`/build/drivers/message/fusion
-I/lib/modules/`uname -r`/source/drivers/message/fusion -o mpt-status
mpt-status.c
In file included from mpt-status.c:38:
mpt-status.h:6:28: error: linux/compiler.h: No such file or directory
It appears that the compiler.h header doesn’t come with the CentOS kernel-headers package (and according to a couple of posts I read, it is no longer a publicly consumable header file), so I removed it from mpt-status.c. This allowed mpt-status.c to build and install correctly:
$ gmake
KERNEL_PATH=/usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18.x86_64**
gcc -Iincl -Wall -W -O2
-I/usr/src/redhat/BUILD/kernel-2.6.18/linux-2.6.18.x86_64/drivers/message/fusion
-I/lib/modules/`uname -r`/build/drivers/message/fusion
-I/lib/modules/`uname -r`/source/driver
s/message/fusion -o mpt-status mpt-status.c
If you are attempting to build mpt-status.c for the first time, make sure you properly install the CentOS Linux kernel source. I received a couple of e-mails from individuals who were trying to get mpt-status working, and in each case the Linux kernel source was not installed properly.
While testing out the latest net-snmp bits, I noticed that various TCP and UDP mibII OIDs would continuosly display zeros on my Solaris 10 update 6 host:
$ snmpwalk -c ... -v 2c localhost udpInDatagrams.0
UDP-MIB::udpInDatagrams.0 = Counter32: 0
Generate lots of UDP traffic.
$ snmpwalk -c ... -v 2c localhost udpInDatagrams.0
UDP-MIB::udpInDatagrams.0 = Counter32: 0
This turns out to be caused by a net-snmp bug, which is fixed by applying the patch attached to the bug report. I am posting this here to help others who may bump into this issue.
I spent a fair amount of time debugging a bizarre net-snmp issue yesterday, and learned a TON about how net-snmp is implemented. While reading through the net-snmp code, I came across a number of macros similar to the following:
DEBUGMSGTL(("mibII/udpScalar", "Initialising UDP scalar group "));
The first argument to the DEBUGMSGCTL() macros contains a token name, which can be passed to the snmpd daemon’s “-D” option to get verbose debugging data:
$ snmpd -c /etc/snmpd.conf -p /var/run/snmpd.pid -Lo -q -V -f -D
kernel_sunos5,mibII/udpScalar
Received SNMP packet(s) from UDP: [0.0.0.0]->[1.2.3.4.102]:-14599
GETNEXT message
-- UDP-MIB::udpInDatagrams
kernel_sunos5: getMibstat (10, *, 40, 0, *, *)
kernel_sunos5: ... cache_valid 0 time 0 ttl 30 now 1239840358
kernel_sunos5: ...... getmib (263, 0, ...)
kernel_sunos5:dlpi: calling getentry aac: req_type: 0, buf: 24, entrysize: 40
kernel_sunos5: bad cache length 24 - not multiple of entry size 40
kernel_sunos5: ...... getmib returns 2
kernel_sunos5: ... result 1 rc 2
kernel_sunos5: ... getMibstat returns 1
mibII/udpScalar: getMibstat call from udp_load : MIB_UDP 10, sizeof(mib2_udp_t): 40
mibII/udpScalar: Loaded UDP scalar Group (solaris)
mibII/udpScalar: Handler - mode GETmibII/udpScalar: oid: UDP-MIB::udpInDatagrams.0Received SNMP packet(s) from UDP: [0.0.0.0]->[1.2.3.4]:-14599
To find the tokens that are available, you can bust out the trusty find utility:
$ cd net-snmp-5.4.2.1
$ find . -type f | xargs grep DEBUGMSG |more
./agent/agent_handler.c: DEBUGMSGTL(("handler::register", "Registering %s (", reginfo->handlerName));
./agent/agent_handler.c: DEBUGMSG(("handler::register", "::%s", handler->handler_name));
./agent/agent_handler.c: DEBUGMSG(("handler::register", ") at "));
......
While the debugging output is a bit primitive, it is extremely useful when you can compare it side-by-side with the net-snmp source code. This helped me locate and fix an annoying bug (data is incorrect on Solaris 10 u4+ hosts), which allowed me to roll out the new version of the code to various hosts (the new version fixes a couple of bugs that lead to the daemon hanging after a period of time). Debugging is a bunch of fun, and there is nothing better than finding a solution to an issue!
My lab environment at home consists of numerous hosts running Linux, Solaris, HP-UX and Windows. To allow me to rebuild hosts with little impact to the applications they run, I typically assign one or more iSCSI LUNs to each host. Configuring the Solaris iSCSI initiator is pretty straight forward, and I documented it a while back for SysAdmin magazine. The Linux Open iSCSI initiator is just as easy to configure, and I thought I would document the process here for future reference.
To configure the Linux initiator to use one or more iSCSI targets, you will first need to install the iscsi-initiator packages. If you are using CentOS Linux, this is as easy as running:
$ yum install iscsi-initiator-utils
As part of the installation process, the postinstall script will run iscsi-iname to gnerate a unique IQN for the host (If you are new to iSCSI, you can learn more about it here). This IQN is persistently stored in the file /etc/iscsi/initiatorname.iscsi:
$ cat initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:377d9b3267d0
With the IQN in hand, you can add LUNs to the target, configure ACLs, and optionally set up CHAP authentication. Once that is complete, you will need to modify /etc/iscsi/iscsid.conf to suit your needs. For a simple home set up, the only thing you need to adjust is your CHAP authentication credentials (for heavy I/O workloads, there are a few additional options that should be adjusted to achieve maximum performance):
#To enable CHAP authentication set node.session.auth.authmethod
#to CHAP. The default is None.
node.session.auth.authmethod = CHAP
# To set a CHAP username and password for initiator
# authentication by the target(s), uncomment the following lines:
node.session.auth.username = centosnode1
node.session.auth.password = SuperSecretPWD
# To enable CHAP authentication for a discovery session to the target
# set discovery.sendtargets.auth.authmethod to CHAP. The default is None.
discovery.sendtargets.auth.authmethod = CHAP
# To set a discovery session CHAP username and password for the initiator
# authentication by the target(s), uncomment the following lines:
discovery.sendtargets.auth.username = centosnode1
discovery.sendtargets.auth.password = SuperSecretPWD
Once the configuration file is modified, you will need to start (or restart) the iscsid daemon and ensure it comes up at system initialization:
$ service iscsid start
$ chkconfig iscsid on
Next up, you will need to use the iscsiadm utility to discover the targets you added to your iSCSI target (for larger installations, you probably want to use the iSNS discovery method):
$ iscsiadm -m discovery -t sendtargets -p 192.168.1.80:3260 -d8
$ iscsiadm -m discovery -t sendtargets -p 192.168.1.81:3260 -d8
In the example above, iscsiadm will connect to the portals passed to the “-p” option, and will use the sendtargets discovery method to locate new targets. As part of the discovery process, iscsiadm will populate the iSCSI database (typically located in /var/lib/iscsi/) with the portals and targets it discovers. Since issues can periodically surface during discovery, I like to add the “-d8” flag to get verbose output. This is extremely handy for seeing what is going on, and also for debugging problems.
Assuming you didn’t encounter any issues, you can run iscsiadm to view iSCSI status. To review the discovered targets, iscsiadm can be run with the “node” option:
$ iscsiadm -m node
192.168.1.81:3260,1 iqn.1986-03.com.sun:02:505908f0-ec05-e598-aeb4-985dc2e7df4b.centosnode2-tgt2
192.168.1.80:3260,1 iqn.1986-03.com.sun:02:505908f0-ec05-e598-aeb4-985dc2e7df4b.centosnode2-tgt2
192.168.1.81:3260,1 iqn.1986-03.com.sun:02:7d244055-cb21-47f2-e6e4-a58a84e6cbb7.centosnode2-tgt1
192.168.1.80:3260,1 iqn.1986-03.com.sun:02:7d244055-cb21-47f2-e6e4-a58a84e6cbb7.centosnode2-tgt1
192.168.1.81:3260,1 iqn.1986-03.com.sun:02:8bd4d404-bf4c-eeb0-9514-986f1e634dc7.centosnode2-tgt3
192.168.1.80:3260,1 iqn.1986-03.com.sun:02:8bd4d404-bf4c-eeb0-9514-986f1e634dc7.centosnode2-tgt3
To see active sessions, iscsiadm can be run with the “session” option:
$ iscsiadm -m session
tcp: [1] 192.168.1.81:3260,1 iqn.1986-03.com.sun:02:505908f0-ec05-e598-aeb4-985dc2e7df4b.centosnode2-tgt2
tcp: [2] 192.168.1.80:3260,1 iqn.1986-03.com.sun:02:505908f0-ec05-e598-aeb4-985dc2e7df4b.centosnode2-tgt2
tcp: [3] 192.168.1.81:3260,1 iqn.1986-03.com.sun:02:7d244055-cb21-47f2-e6e4-a58a84e6cbb7.centosnode2-tgt1
tcp: [4] 192.168.1.80:3260,1 iqn.1986-03.com.sun:02:7d244055-cb21-47f2-e6e4-a58a84e6cbb7.centosnode2-tgt1
tcp: [5] 192.168.1.81:3260,1 iqn.1986-03.com.sun:02:8bd4d404-bf4c-eeb0-9514-986f1e634dc7.centosnode2-tgt3
tcp: [6] 192.168.1.80:3260,1 iqn.1986-03.com.sun:02:8bd4d404-bf4c-eeb0-9514-986f1e634dc7.centosnode2-tgt3
You can also view the portals in use, and the targets that are visible through each portal:
$ iscsiadm -m node -P 1
Target: iqn.1986-03.com.sun:02:df912764-3b31-4c9e-cd3d-bcf4010d83f8.centosnode1-tgt1
Portal: 192.168.1.81:3260,1
Iface Name: default
Portal: 192.168.1.80:3260,1
Iface Name: default
Target: iqn.1986-03.com.sun:02:a6bda08a-8686-e8b4-8999-fe1bd0861033.centosnode1-tgt3
Portal: 192.168.1.81:3260,1
Iface Name: default
Portal: 192.168.1.80:3260,1
Iface Name: default
Target: iqn.1986-03.com.sun:02:5876b107-b756-e40b-d9d6-cd0aa30d7144.centosnode1-tgt2
Portal: 192.168.1.81:3260,1
Iface Name: default
Portal: 192.168.1.80:3260,1
Iface Name: default
If you configured your host to access multiple portals on the target, you should configured /etc/multipath.conf and enable the multipathd daemon:
$ service multipathd start
$ chkconfig multipathd on
Once multipathd is running, you can run the multipath utility to view the paths assigned to each target (typically you would see the WWN or a friendly mpath[0-9]* name for each multipathed device, but I have configured aliases to simplify my life):
$ multipath -ll
iscsidisk3 (3600144f049e0360f00000c29877ac600) dm-1 SUN,SOLARIS
[size=9.0G][features=0][hwhandler=0]
_ round-robin 0 [prio=1][active]
_ 2:0:0:0 sdg 8:96 [active][ready]
_ round-robin 0 [prio=1][enabled]
_ 3:0:0:0 sdf 8:80 [active][ready]
iscsidisk2 (3600144f049e0361100000c29877ac600) dm-0 SUN,SOLARIS
[size=9.0G][features=0][hwhandler=0]
_ round-robin 0 [prio=1][active]
_ 4:0:0:0 sdi 8:128 [active][ready]
_ round-robin 0 [prio=1][enabled]
_ 5:0:0:0 sdh 8:112 [active][ready]
iscsidisk1 (3600144f049e0361000000c29877ac600) dm-2 SUN,SOLARIS
[size=9.0G][features=0][hwhandler=0]
_ round-robin 0 [prio=1][active]
_ 7:0:0:0 sdj 8:144 [active][ready]
_ round-robin 0 [prio=1][enabled]
_ 6:0:0:0 sde 8:64 [active][ready]
If everything went smoothly up to this point, you should see several new devices in the ‘fdisk -l’ output. Each device will need to be labeled, and have a file system laid on top of it. If you have multiple paths to a device (you can tell this by looking at the multipath utilities output), you will need to use the /dev/mapper device instead of the /dv/sd[a-z] devices. The Open iSCSI project is pretty awesome, and you can read through their README to learn more about the technology.
I needed to add a second file system to one of my Solaris 10 zones this morning, and needed to do so without rebooting the zone. Since the global zone uses loopback mounts to present file systems to zones, adding a new file system was as easy as loopback mounting the file system into the zone’s file system:
$ mount -F lofs /filesystems/zone1oracle03 /zones/zone1/root/ora03
Once the file system was mounted, I added it to the zone configuration and then verified it was mounted:
$ mount | grep ora03
/filesystems/zone1oracle03 on filesystems/zone1oracle0 read/write/setuid/devices/nonbmand/exec/xattr/atime/dev=2d9000b on Sun Apr 12 10:43:19 2009
/zones/zone1/root/ora03 on /filesystems/zone1oracle03 read/write/setuid/devices/dev=2d9000b on Sun Apr 12 10:44:07 2009
Now to update my ASM disk group to use the storage.