Tuning network and vm settings on CentOS Linux servers with ktune

While poking around the CentOS package repository, I came across the ktune package. Ktune comes with a set of kernel tunables that are useful for network and disk intensive workloads, and provides the ktune service to apply these settings during system startup. Ktune includes settings for TCP/IP buffers, setting the deadline scheduler as the default I/O scheduler, and entries to adjust the swappiness, dirty_ratio and pagecache settings. The full list of tunables can be viewed by paging through the following two configuration files:

$ less /etc/sysctl.ktune

$ less /etc/sysconfig/ktune

To activate the settings, you can enable the ktune service with the chkconfig and service utilities:

$ chkconfig ktune on

$ service ktune start

Saving current sysctl settings:                            [  OK  ]
Applying ktune sysctl settings from /etc/sysctl.ktune:     [  OK  ]
Applying sysctl settings from /etc/sysctl.conf:            [  OK  ]
Applying deadline elevator: sda                            [  OK  ]



This is an awesome package, and I definitely plan to use the network settings on all of my CentOS hosts.

Viewing CPU frequency scaling data on CentOS Linux hosts

I just built a new quad core AMD Opteron host, and was curious what frequency steppings were available for my processor. After a bit of poking around, I came across the cpufreq-utils package, which contains utilities to view and set processor frequency settings. To view the settings for the processors on a system, cpufreq-info can be run without any options (this assumes you installed the cpufreq-utils package):

$ cpufreq-info

cpufrequtils 005: cpufreq-info (C) Dominik Brodowski 2004-2006
Report errors and bugs to cpufreq@vger.kernel.org, please.
analyzing CPU 0:
  driver: powernow-k8
  CPUs which need to switch frequency at the same time: 0
  hardware limits: 1.10 GHz - 2.20 GHz
  available frequency steps: 2.20 GHz, 2.00 GHz, 1.70 GHz, 1.40 GHz, 1.10 GHz
  available cpufreq governors: ondemand, userspace, performance
  current policy: frequency should be within 1.10 GHz and 2.20 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 1.10 GHz (asserted by call to hardware).
analyzing CPU 1:
  driver: powernow-k8
  CPUs which need to switch frequency at the same time: 1
  hardware limits: 1.10 GHz - 2.20 GHz
  available frequency steps: 2.20 GHz, 2.00 GHz, 1.70 GHz, 1.40 GHz, 1.10 GHz
  available cpufreq governors: ondemand, userspace, performance
  current policy: frequency should be within 1.10 GHz and 2.20 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 1.10 GHz (asserted by call to hardware).
analyzing CPU 2:
  driver: powernow-k8
  CPUs which need to switch frequency at the same time: 2
  hardware limits: 1.10 GHz - 2.20 GHz
  available frequency steps: 2.20 GHz, 2.00 GHz, 1.70 GHz, 1.40 GHz, 1.10 GHz
  available cpufreq governors: ondemand, userspace, performance
  current policy: frequency should be within 1.10 GHz and 2.20 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 1.10 GHz (asserted by call to hardware).
analyzing CPU 3:
  driver: powernow-k8
  CPUs which need to switch frequency at the same time: 3
  hardware limits: 1.10 GHz - 2.20 GHz
  available frequency steps: 2.20 GHz, 2.00 GHz, 1.70 GHz, 1.40 GHz, 1.10 GHz
  available cpufreq governors: ondemand, userspace, performance
  current policy: frequency should be within 1.10 GHz and 2.20 GHz.
                  The governor "ondemand" may decide which speed to use
                  within this range.
  current CPU frequency is 1.10 GHz (asserted by call to hardware).



The output above contains the list of frequency steppings:

available frequency steps: 2.20 GHz, 2.00 GHz, 1.70 GHz, 1.40 GHz, 1.10 GHz

As well as the current operating frequency of each core:

current CPU frequency is 1.10 GHz (asserted by call to hardware).

If you want to change the frequency profile for a processor, you can use the cpufreq-set utility. I should use this as an opportunity to pick up a kilowatt, and see how much energy is being saved by running the CPUs at 1.1GHZ instead of 2.2GHZ.

Debugging problems with the open-iscsi initiator

I talked about the open-iscsi initiator in a previous post, and briefly touched on debugging problems. If you want to get additional debugging data for the iscsid daemon, you can add the “-d 8” option to the iscsi initialization file:

$ less /etc/init.d/iscsid

start()
{
        echo -n $"Turning off network shutdown. "
        # we do not want iscsi or network to run during system shutdown
        # incase there are RAID or multipath devices using
        # iscsi disks
        chkconfig --level 06 network off
        rm /etc/rc0.d/*network
        rm /etc/rc6.d/*network

        echo -n $"Starting iSCSI daemon: "
        modprobe -q iscsi_tcp
        modprobe -q ib_iser
        daemon iscsid -d 8
        RETVAL=$?
        echo
        [ $RETVAL -eq 0 ] || return

        touch /var/lock/subsys/iscsid

        success
        echo
}



This will cause the daemon to write a bunch of debugging data to the syslog logs, which you can then use to isolate problems.

Best I/O scheduler to use with virtualized Linux hosts

While reading through the Redhat Oracle 10G best practices document, I came across this gem:

“In virtualized environments, it is often detrimental to schedule I/O at both the host and guest layers. If multiple guests access storage on a file system or block devices managed by the host operating system, the host may be able to schedule I/O more efficiently because it alone is aware of requests from all guests and knows the physical layout of storage, which may not map linearly to the guests’ virtual storage. Red Hat Enterprise Linux 4 and Red Hat Enterprise Linux 5 guests can use the noop I/O scheduler to allow the host to optimize I/O requests.”

This makes complete sense, and I am going to have to test out the noop I/O scheduler in my lab this weekend. I’m curious how many folks run the default I/O scheduler in their Xen or KVM guests, and are actually hindering I/O performance by doing so.

Updating Emulex firmware on Solaris hosts

I’ve written about the Emulex emlxadm utility in the past, and how it can be used to view and manage adapters. In addition to viewing stats and configuration data, you can also use emlxadm to view and update firmware. To view the current firmware version, you can type “get_fw_rev” into the emlxadm shell:

emlxadm> get_fw_rev

Firmware revision: LP10000DC 1.81a1

If the firmware needs to be updated, you can run download_fcode with the path to the firmware image:

emlxadm> download_fcode /export/home/matty/emulex/firmware/1.92a1/td192a1.all

Image Components: NOP type
AWC file: KERN: version=ff801416, 1.40a6
DWC file: SLI2: version=07831991, 1.92a1
DWC prog: TEST: version=00f51051, 1.01a1
DWC prog: STUB: version=02881991, 1.92a1
DWC prog: SLI1: version=06831991, 1.92a1
DWC prog: SLI2: version=07831991, 1.92a1

Current: Fcode: none
New: Fcode: none 376064 (0x5bd00) bytes

Are you sure you want to download this image? (y or n): y

Downloading…

Result: Operation successful.
Done.

*NOTE: The new FCode version will not be activated until the host
is rebooted or a DR operation is performed on the adapter.

emlxadm> get_fw_rev

Firmware revision: LP10000DC 1.92a1

This post is a reference for myself, and I take ZERO responsibility for botched firmware updates. As with all information on this blog, use this information at your own risk (no warranties of any sort are provided).

Changing the IP address on a Brocade 3200

I purchased a used Brocade 3200 for my home lab, and needed to update the network information to allow myself to login remotely. To view the existing network configuration, I serial consoled in and ran the ipAddrShow command:

Switch:admin> ipAddrShow
Ethernet IP Address: 10.1.1.48
Ethernet Subnetmask: 255.255.255.0
Fibre Channel IP Address: none
Fibre Channel Subnetmask: none
Gateway Address: 10.1.1.1

Once I checked the existing settings, I ran the ipAddrSet command to update the network settings:

Switch:admin> ipAddrSet
Ethernet IP Address [10.192.40.148]: 192.168.1.2
Ethernet Subnetmask [255.255.240.0]: 255.255.255.0
Fibre Channel IP Address [none]:
Fibre Channel Subnetmask [none]:
Gateway Address [10.192.47.254]: 192.168.1.1

I really dig Brocade, and their command set is simple and easy to use.