Generating E-mail when Solaris security patches are available

I wrote about pca a few weeks back, and absolutely love the capabilities it brings to the table. To keep my servers up date with the latest security patches, I added the following cron job to extract security patches from the pca output, and E-mail them to my account:

0 0 * * * pca -l | awk '$5 ~ /S/ { print $0}'  | mailx -s "Security updates for $HOSTNAME" root

This works pretty well, and I now get E-mailed when security patches are available.

Testing postfix restrictions

The postfix mail delivery system allows one or more restrictions to be placed on incoming messages. These restrictions allow you to block messages when clients don’t use the SMTP handshake as outlined in RFC 821 during delivery, if the name supplied in the HELO is not fully qualified or fails to resolve, or if the connecting host resides on one or more blacklists. Several restrictions come enabled by default, but others need to be manually enabled by adding additional directives to the smtpd_helo_restrictions, smtpd_sender_restrictions and smtpd_recipient_restrictions variables. Since new restrictions can have unexpected results (e.g., lost E-mail), it is beneficial to test these rules prior to enforcing them.

The soft_bounce directive is one way to test out new rules. When soft_bounce is set in the main.cf, a message that would be blocked by a restriction is rejected with a soft reject error code (typically something in the 4XX range). The soft reject will force the sending host to queue the message, and attempt redelivery at a later date. Each time a soft bounce occurs, postfix will log a message to the logging facility. Since the server that sent the message will attempt to redeliver the message at a later date, you can tweak the configuration prior to the next delivery attempt if you want to accept mail from the host that matched the restriction. This method does have one major drawback. Numerous MTAs on the Internet use low TTLs on messages in their mail queues, so using soft bounces can result in messages getting returned to the sender if the restriction isn’t adjusted in a timely manner.

An alternative method to soft bounces is the warn_if_reject qualifier. Instead of soft bouncing the message if the sender matches a restriction, the warn_if_reject qualifier will cause postfix to deliver the message, and log a warning when a message matches a restriction. This ensures that mail is delivered if the restriction has an unknown side effect, and allows the mail admin to test restrictions in a non-intrusive way. Warn_if_reject can be applied to all of the postfix restrictions ( I have yet to find a restriction that warn_if_reject doesn’t apply to, but there may be some), and is set by prepending the string “warn_if_reject” to the restriction:

smtpd_helo_restrictions         = permit_mynetworks,
                                  reject_non_fqdn_hostname,
                                  warn_if_reject reject_invalid_hostname,
                                  warn_if_reject reject_unknown_hostname

To get a daily report with messages that matched the restriction, you can use the awesome pflogsum Perl script. Using warn_if_reject has saved me a fair amount of grief, and is a great way to make sure a restriction does what you want it to.

Implementing backoff timers in RSS readers

I have recently been on a quest to find a new RSS reader for OS X. NetNewsWire looks to be the leading candidate, since LifeRea doesn’t have a native port to OS X. One thing I noticed in the clients I tested, is that they have fixed times when they will check ALL feeds for new content. This time increment can be in minutes, hours, days, weeks or months. Since several of my feeds get updated 1 – 2 times a month, the fixed time metric doesn’t really work all that well with those sources (especially since the time applies to all feeds). While it may take a bit of additional coding, it would be neat to have syndication clients learn how often feed are updated, and base their syndication checks around that. Hopefully the RSS reader developers will add backoff timers to their clients, which should save bandwidth, and reduce the amount of work each web server needs to do.

Understanding MySQL performance data with mysqlreport

MySQL maintains numerous operational metrics (e.g., connections, questions, etc), which can be accessed by running ‘show status’ or one of it’s variants from the mysql client. The mysqlreport Perl script can be used to summarize this data into a nicely formatted report with several useful performance metrics:

$ mysqlreport –user privuser –password password -all

MySQL 5.0.24             uptime 0 10:20:0       Tue Aug  8 01:05:17 2006

__ Key _________________________________________________________________
Buffer usage   77.00k of  64.00M  %Used:   0.12
Write ratio      0.04
Read ratio       0.02

__ Questions ___________________________________________________________
Total           3.55k    0.10/s
  QC Hits       2.11k    0.06/s  %Total:  59.38
  DMS             964    0.03/s           27.15
  Com_            240    0.01/s            6.76
  COM_QUIT        238    0.01/s            6.70
Slow                0    0.00/s            0.00  %DMS:   0.00
DMS               964    0.03/s           27.15
  SELECT          935    0.03/s           26.34         96.99
  UPDATE           17    0.00/s            0.48          1.76
  INSERT           12    0.00/s            0.34          1.24
  REPLACE           0    0.00/s            0.00          0.00
  DELETE            0    0.00/s            0.00          0.00
Com_              240    0.01/s            6.76
  change_db       238    0.01/s            6.70
  show_variab       1    0.00/s            0.03
  show_status       1    0.00/s            0.03

__ SELECT and Sort _____________________________________________________
Scan              566    0.02/s %SELECT:  60.53
Range              44    0.00/s            4.71
Full join           0    0.00/s            0.00
Range check         0    0.00/s            0.00
Full rng join       0    0.00/s            0.00
Sort scan         558    0.01/s
Sort range         49    0.00/s
Sort mrg pass       0    0.00/s

__ Query Cache _________________________________________________________
Memory usage    3.06M of  16.00M  %Used:  19.09
Block Fragmnt   0.06%
Hits            2.11k    0.06/s
Inserts           932    0.03/s
Prunes              1    0.00/s
Insrt:Prune     932:1    0.03/s
Hit:Insert     2.26:1

__ Table Locks _________________________________________________________
Waited              0    0.00/s  %Total:   0.00
Immediate       1.22k    0.03/s

__ Tables ______________________________________________________________
Open               19 of  128    %Cache:  14.84
Opened             25    0.00/s

__ Connections _________________________________________________________
Max used            2 of  128      %Max:   1.56
Total             240    0.01/s

__ Created Temp ________________________________________________________
Disk table        159    0.00/s
Table             399    0.01/s
File                0    0.00/s

The output from mysqlreport includes metrics on key cache and query cache utilization, the number of operations (SELECT, UPDATE, etc.) performed, thread utilization, connection volumes, table locks, and the types of temporary tables used by sorts. The folks over at hackmysql provide a nice writeup on what each report means, and the typical value ranges you should see in each report. Running mysqlreport is a great way to get a high-level understanding of how a database is performing, and can greatly assist with identifying the areas that are worth reviewing in greater detail.

Extracting system and network statistics with net-snmp

The net-snmp software suite implements the SNMPv1, SNMPv2 and SNMPv3 protocols, and comes with several utilities to remotely retrieve data from servers and network devices. One such utility is snmpstatus, which allows you to retrieve the overall status of a given device:

$ snmpstatus -v 2c -c yikes 10.10.1.1

[10.10.1.7]=>[Content Switch SW Version 07.90 with SNMPv1/v2c Agent] Up: 152 days, 10:47:55.03
Interfaces: 6, Recv/Trans packets: -1275435709/-1122090667 | IP: 58510352/75120061
2 interfaces are down!

Another nifty tool that comes with net-snmp is snmpnetstat, which allows you to get a “netstat”-like view of any device that supports SNMP. The following example show how to retrieve the active and passive UDP and TCP ports from a device:

$ snmpnetstat -v 2c -c yikes -a 10.10.1.1

Active Internet (tcp) Connections (including servers)
Proto Local Address                Foreign Address              (state)
tcp    *.ftp                        *.*                          LISTEN
tcp    *.ssh                        *.*                          LISTEN
tcp    *.telnet                     *.*                          LISTEN
tcp    *.80                         *.*                          LISTEN
tcp    *.443                        *.*                          LISTEN
tcp    10.10.1.1.4189             www.youch.net.8080  ESTABLISHED
tcp    10.10.1.1.4190             www.youch.net.8080  ESTABLISHED
Active Internet (udp) Connections
Proto Local Address
udp    *.ntp
udp    *.snmpd
udp    *.1024
udp    *.5002
udp    *.47806

Snmpstat also allows you to retrieve network statistics with the “-s” option:

$ snmpnetstat -v 2c -c yikes -s 10.10.1.1

ip:
        58509607 total datagrams received
        0 datagrams with header errors
        6068304 datagrams with an invalid destination address
        0 datagrams forwarded
        0 datagrams with unknown protocol
        0 datagrams discarded
        52441309 datagrams delivered
        75119154 output datagram requests
        0 output datagrams discarded
        45 datagrams with no route
        0 fragments received
        0 datagrams reassembled
        0 reassembly failures
        0 datagrams fragmented
        0 fragmentation failures
        0 fragments created
icmp:
        2634607 total messages received
        0 messages dropped due to errors
        584 ouput message requests
        270 output messages discarded
        Output Histogram:
                Destination unreachable: 270
                Echo Reply: 314
        Input Histogram:
                Echo Request: 314
                Echo Reply: 2634293
tcp:
        2476943 active opens
        713945482 passive opens
        322859 failed attempts
        2073710 resets of established connections
        694522544 current established connections
        -1065832368 segments received
        1831139460 segments sent
        18 segments retransmitted
tcp:
        2476943 active opens
        713945482 passive opens
        322859 failed attempts
        2073710 resets of established connections
        694522544 current established connections
        -1065832365 segments received
        1831139460 segments sent
        18 segments retransmitted
udp:
        1244436728 total datagrams received
        0 datagrams to invalid port
        29924 datagrams dropped due to errors
        1244436695 output datagram requests
udp:
        1244436728 total datagrams received
        0 datagrams to invalid port
        29924 datagrams dropped due to errors
        1244436695 output datagram requests

SNMP is a useful protocol, and I am looking forward to seeing more vendors support SNMPv3.