Greping through binary data with zgrep and gzcat

I periodically need to look for a given string in one or more compressed log files. Taking the time (and resources) to decompress each file on the file system takes time, especially when I don’t plan to leave the file uncompressed. When these situations arise, I turn to my good friends gzcat and zgrep.

The zgrep utility allows you to look for a pattern in one or more compressed files. If you want to find the string “target port down” in a set of logs named sanlogs*.gz, you can do the following:

$ zgrep “target port down” sanlogs*.gz

If zgrep finds the string, it will print it. Alternatively, you can use the gzcat utility to decompress the contents of a file prior to sending it to another tool for processing:

$ gzcat services.gz | tail -2
# 48557-49150 Unassigned
# 49151 IANA Reserved

Both tools are incredibly useful, and should be in every SysAdmins tool bag.

Solaris SNMP agent updated to net-snmp version 5.4.1

I just saw that the opensolaris net-snmp software was updated to version 5.4.1:

Author: Vijay HN
Repository: /hg/onnv/onnv-gate
Latest revision: 739325d040fb9f332a2ae9e196de38e06f1ec81b
Total changesets: 1
Log message:
LSARC 2008/355 System Management Agent (SMA1.0) migration to Net-SNMP 5.4.1
6848323 ON package changes required for SMA to net-snmp migration

This is a welcome addition, and will provide admins with a number of new capabilities (e.g., disk monitoring) that aren’t available in the net-snmp 5.0.9 build that is currently part of Solaris!

Dealing with cron bad user messages on Solaris hosts

While reviewing the cron logs on one of my Solaris hosts, I noticed a number of entries similar to the following:

CMD: /opt/software/bin/arrecord -backup
> arr 20359 c Thu Sep 3 23:45:00 2009
! bad user (arr) Thu Sep 3 23:45:00 2009
< arr 20359 c Thu Sep 3 23:45:00 2009 rc=1 These errors are typically generated when the account the job run as doesn't exist, or when the user's shadow entry is locked (locked accounts have a *LK* in the /etc/shadow password field). In this specific case a password or NP entry (the account doesn't have a password, and logins are denied) wasn't assigned to the arr user, so the account was still listed in the locked state. Setting a strong password fixed the issue, and everything is working swimmingly!

Viewing PCI bus throughput on Solaris systems

The PCI and PCI express interconnect technologies have become the defacto standard for connecting peripherals to most motherboards. 64-bit 66 MHZ PCI interconnects run with speeds up to 528MB/s, and share the available bandwidth between devices on the PCI bus. PCI express x32 runs with speeds up to 8 GB/s, and provides dedicated “lanes” to connect each peripheral directly to the motherboards chipsets. This allows PCI express devices to utilize all of the available bandwidth, and maximizes throughput since PCI express devices do not need to compete with other devices on the bus. On systems that utilize gigabit Ethernet and 4 GB/s HBAs, it is possible to saturate a 33 or 66 MHZ PCI bus during peak loads. If you happen to be using a PCI interconnect and the Solaris Operating System, you can easily measure the current PCI bus utilization with the busstat(1m) command:

$ busstat -w pcis0,pic0=dvma_cycles,pic1=dvma_wd_xfr -w pcis1=pic0=dvma_cycles,pic1=dvma_wd_xfr 5

time dev    event0               pic0        event1               pic1
5    pcis0  dvma_cycles          482928      dvma_wd_xfr          115184
5    pcis1  dvma_cycles          20217638    dvma_wd_xfr          32815051
10   pcis0  dvma_cycles          482778      dvma_wd_xfr          115046
10   pcis1  dvma_cycles          20989675    dvma_wd_xfr          34068291
15   pcis0  dvma_cycles          482643      dvma_wd_xfr          115069
15   pcis1  dvma_cycles          20834960    dvma_wd_xfr          33817211

bustat’s “-w (instrument PIC to collect statistics) option will program the PCI interconnects PICs (programmable interrupt controller) to collect the statistic passed as an option. These statistics including the number of PIO mode reads and writes, bus throughput, stream transfers, interrupts, and the number of DMA and DVMA operations performed. These counters are documented in the Systems Performance and Tuning book, and can also be viewed by sifting through writing Solaris PCI device drivers for Sun SPARC platforms.

Building ISC DHCP 4.X on Solaris hosts

I attempted to upgrade my ISC DHCP installation to dhcp-4.1.1b1 this past weekend, and ran into the following configure error:

$ ./configure –prefix=/bits/software/dhcp-4.1.1b1

checking for a BSD-compatible install... /bin/ginstall -c
checking whether build environment is sane... yes
checking for gawk... gawk
checking whether make sets $(MAKE)... no
checking for gcc... gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... unsupported
checking for style of include used by make... none
checking dependency style of gcc... none
checking how to run the C preprocessor... /lib/cpp
configure: error: C preprocessor "/lib/cpp" fails sanity check
See `config.log' for more details.

Th config.log had a number of errors similar to the following:

conftest.c:11:19: stdio.h: No such file or directory
conftest.c:12:23: sys/types.h: No such file or directory
conftest.c:13:22: sys/stat.h: No such file or directory

Which are due to missing system headers. I reviewed the list of packages that were installed, and sure enough SUNWhea (this package contains the various header files) was missing. I installed this package as well as a number of others:

$ pkgadd -d . SUNWhea SUNWbinutils SUNWarc SUNWgcc SUNWgccruntime

$ pkgadd -d . SUNWlibsigsegv SUNWgm4 SUNWgnu-automake-110 SUNWaconf

Any everything compiled and installed perfectly.

Using the CPU power management features in Solaris

I mentioned previously that I built out some new hardware. When I was spec’ing out the hardware, I made sure to get “green” components that supported advanced power management features. Solaris is able to take advantage of the CPU power states, and can lower the processor operating frequency when a server is idle. Power managementis handled by the the Solaris kernel, and configured through the /etc/power.conf configuration file. By default, the file will contain a CPU power management entry similar to the following:

cpupm                   enable
cpu-threshold           1s

The cpupm directive enables CPU power management, and the cpu-threshold directive indicates how often the processor needs to be idle before the CPU frequency is lowered. To check the current operating frequency of a CPU, you can check the current_clock_Hz value for each CPU:

$ kstat -m cpu_info -i 0 -s current_clock_Hz

module: cpu_info                        instance: 0     
name:   cpu_info0                       class:    misc
        current_clock_Hz                1100000000

This is awesome stuff, and when the disk power management project is integrated, servers will hopefully be able to reduce their power consumption dramatically. Viva la Solaris!