Viewing NIC throughput with nicstat

While analzying some throughput issues this weekend, it came to my attention that I have never written about my favorite network analysis tools. I have 10 – 12 utilities that I cannot live without, and plan to cover each utility over the next few weeks (stay tuned!). Tonight, you are in luck! I present to you nicstat:

$ nicstat 5

    Time   Int   rKb/s   wKb/s   rPk/s   wPk/s    rAvs    wAvs   %Util     Sat
00:27:34  hme0    1.50   70.79   21.96   57.29   69.93 1265.28    0.59    0.00
00:27:39  hme0    1.38   68.24   20.15   55.46   70.28 1259.90    0.57    0.00
00:27:44  hme0    1.33   72.09   19.37   58.51   70.45 1261.69    0.60    0.00
00:27:49  hme0    1.57   70.89   22.36   58.29   71.73 1245.39    0.59    0.00
00:27:54  hme0    1.40   69.54   20.36   56.48   70.24 1260.69    0.58    0.00
00:27:59  hme0    1.32   69.52   18.96   56.29   71.05 1264.75    0.58    0.00
00:28:04  hme0    1.51   70.82   22.16   57.09   69.89 1270.31    0.59    0.00

This nifty utility reports utilization metrics for each Ethernet interface in a Solaris server. nicstat is written in C and Perl, and the program header contains the following descriptions of each field that is displayed:

Int            Interface
rKb/s          read Kbytes/s
wKb/s          write Kbytes/s
rPk/s          read Packets/s
wPk/s          write Packets/s
rAvs read      Average size, bytes
wAvs write     Average size, bytes
%Util          %Utilisation (r+w/ifspeed)
Sat            Saturation (defer, nocanput, norecvbuf, noxmtbuf)

Brendan Gregg creates some amazing tools, and I highly recommend sending him lots of beer and chocolate (or thank you emails if your cheap)!

Making Solaris DHCP server IP addresses usable

While working through a few issues with our x86 jumpstart setup, I saw the following error messages when one of the clients was attempting to PXE boot:

429f62d0: Datagram received on network device: ge0(limited broadcast)
429f62d0: (01000D609C8888,1.2.3.4) currently marked as unusable.

To make the IP address usable again, I used the Solaris pntadm utility:

$ pntadm -M 1.2.3.4 -f PERMANENT 1.2.3.0

The “-M” option instructs pntadm to modify the entry, and the “-f PERMANENT” option tells pntadm to make this entry permanent. The final option is the macro associated with the network which the IP address/client is assigned to. To verify the command worked, we can use pntadm’s “-P” (print) option:

$ pntadm -P 1.2.3.0

Client ID       Flags  Client IP    Server IP    Lease  Expiration Macro    Comment
01000D609C8888  01     1.2.3.4      1.2.3.200    Forever           1.2.3.0  me

Once I adjusted the flags, everything started working. Jeah!

Viewing TCP connection data

The netstat utility provides a command line interface to retrieve system routing tables, connection states, and network statistics. Two of the available statisticss, “tcpPassiveOpens” and “tcpActiveOpens,” provide data on the number of new connections to a server (tcpPassiveOpens), and the number of connections initiated by the server (tcpActiveOpens). Both statistics counters can be retrieved with netstat’s “-s” option, and a simple while loop allows a SysAdmin to get a high level view of TCP connections to and from a server:

$ while :
> do
>        netstat -s | egrep '(tcpPassiveOpens|tcpActiveOpens)'
>       sleep 10
> done
        tcpActiveOpens      =  6228     tcpPassiveOpens     =    75
        tcpActiveOpens      =  6228     tcpPassiveOpens     =    75
        tcpActiveOpens      =  6228     tcpPassiveOpens     =    76
        tcpActiveOpens      =  6228     tcpPassiveOpens     =    85
        tcpActiveOpens      =  6228     tcpPassiveOpens     =   140
        tcpActiveOpens      =  6228     tcpPassiveOpens     =   197
        tcpActiveOpens      =  6228     tcpPassiveOpens     =   255

This is kinda fun to run on super busy web servers!!!