Debugging problems with the open-iscsi initiator

I talked about the open-iscsi initiator in a previous post, and briefly touched on debugging problems. If you want to get additional debugging data for the iscsid daemon, you can add the “-d 8” option to the iscsi initialization file:

$ less /etc/init.d/iscsid

        echo -n $"Turning off network shutdown. "
        # we do not want iscsi or network to run during system shutdown
        # incase there are RAID or multipath devices using
        # iscsi disks
        chkconfig --level 06 network off
        rm /etc/rc0.d/*network
        rm /etc/rc6.d/*network

        echo -n $"Starting iSCSI daemon: "
        modprobe -q iscsi_tcp
        modprobe -q ib_iser
        daemon iscsid -d 8
        [ $RETVAL -eq 0 ] || return

        touch /var/lock/subsys/iscsid


This will cause the daemon to write a bunch of debugging data to the syslog logs, which you can then use to isolate problems.

Configuring the open-iscsi initiator on CentOS Linux hosts

My lab environment at home consists of numerous hosts running Linux, Solaris, HP-UX and Windows. To allow me to rebuild hosts with little impact to the applications they run, I typically assign one or more iSCSI LUNs to each host. Configuring the Solaris iSCSI initiator is pretty straight forward, and I documented it a while back for SysAdmin magazine. The Linux Open iSCSI initiator is just as easy to configure, and I thought I would document the process here for future reference.

To configure the Linux initiator to use one or more iSCSI targets, you will first need to install the iscsi-initiator packages. If you are using CentOS Linux, this is as easy as running:

$ yum install iscsi-initiator-utils

As part of the installation process, the postinstall script will run iscsi-iname to gnerate a unique IQN for the host (If you are new to iSCSI, you can learn more about it here). This IQN is persistently stored in the file /etc/iscsi/initiatorname.iscsi:

$ cat initiatorname.iscsi

With the IQN in hand, you can add LUNs to the target, configure ACLs, and optionally set up CHAP authentication. Once that is complete, you will need to modify /etc/iscsi/iscsid.conf to suit your needs. For a simple home set up, the only thing you need to adjust is your CHAP authentication credentials (for heavy I/O workloads, there are a few additional options that should be adjusted to achieve maximum performance):

# To enable CHAP authentication set node.session.auth.authmethod
# to CHAP. The default is None.
node.session.auth.authmethod = CHAP

# To set a CHAP username and password for initiator
# authentication by the target(s), uncomment the following lines:
node.session.auth.username = centosnode1
node.session.auth.password = SuperSecretPWD

# To enable CHAP authentication for a discovery session to the target
# set discovery.sendtargets.auth.authmethod to CHAP. The default is None.
discovery.sendtargets.auth.authmethod = CHAP

# To set a discovery session CHAP username and password for the initiator
# authentication by the target(s), uncomment the following lines:
discovery.sendtargets.auth.username = centosnode1
discovery.sendtargets.auth.password = SuperSecretPWD

Once the configuration file is modified, you will need to start (or restart) the iscsid daemon and ensure it comes up at system initialization:

$ service iscsid start
$ chkconfig iscsid on

Next up, you will need to use the iscsiadm utility to discover the targets you added to your iSCSI target (for larger installations, you probably want to use the iSNS discovery method):

$ iscsiadm -m discovery -t sendtargets -p -d8
$ iscsiadm -m discovery -t sendtargets -p -d8

In the example above, iscsiadm will connect to the portals passed to the “-p” option, and will use the sendtargets discovery method to locate new targets. As part of the discovery process, iscsiadm will populate the iSCSI database (typically located in /var/lib/iscsi/) with the portals and targets it discovers. Since issues can periodically surface during discovery, I like to add the “-d8” flag to get verbose output. This is extremely handy for seeing what is going on, and also for debugging problems.

Assuming you didn’t encounter any issues, you can run iscsiadm to view iSCSI status. To review the discovered targets, iscsiadm can be run with the “node” option:

$ iscsiadm -m node,1,1,1,1,1,1

To see active sessions, iscsiadm can be run with the “session” option:

$ iscsiadm -m session

tcp: [1],1
tcp: [2],1
tcp: [3],1
tcp: [4],1
tcp: [5],1
tcp: [6],1

You can also view the portals in use, and the targets that are visible through each portal:

$ iscsiadm -m node -P 1

		Iface Name: default
		Iface Name: default
		Iface Name: default
		Iface Name: default
		Iface Name: default
		Iface Name: default

If you configured your host to access multiple portals on the target, you should configured /etc/multipath.conf and enable the multipathd daemon:

$ service multipathd start
$ chkconfig multipathd on

Once multipathd is running, you can run the multipath utility to view the paths assigned to each target (typically you would see the WWN or a friendly mpath[0-9]* name for each multipathed device, but I have configured aliases to simplify my life):

$ multipath -ll

iscsidisk3 (3600144f049e0360f00000c29877ac600) dm-1 SUN,SOLARIS
\_ round-robin 0 [prio=1][active]
 \_ 2:0:0:0 sdg 8:96  [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 3:0:0:0 sdf 8:80  [active][ready]
iscsidisk2 (3600144f049e0361100000c29877ac600) dm-0 SUN,SOLARIS
\_ round-robin 0 [prio=1][active]
 \_ 4:0:0:0 sdi 8:128 [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 5:0:0:0 sdh 8:112 [active][ready]
iscsidisk1 (3600144f049e0361000000c29877ac600) dm-2 SUN,SOLARIS
\_ round-robin 0 [prio=1][active]
 \_ 7:0:0:0 sdj 8:144 [active][ready]
\_ round-robin 0 [prio=1][enabled]
 \_ 6:0:0:0 sde 8:64  [active][ready]

If everything went smoothly up to this point, you should see several new devices in the ‘fdisk -l’ output. Each device will need to be labeled, and have a file system laid on top of it. If you have multiple paths to a device (you can tell this by looking at the multipath utilities output), you will need to use the /dev/mapper device instead of the /dv/sd[a-z] devices. The Open iSCSI project is pretty awesome, and you can read through their README to learn more about the technology.