Monitoring PF Firewalls For Health And Performance


The PF (packet filter) firewall package was introduced in OpenBSD 3.0, and has since been ported to the FreeBSD and NetBSD Operating Systems. PF contains a stateful packet inspection engine, the ability to replicate state information to a backup firewall, a flexible self optimizing rule engine, QOS support, and the ability to collect performance metrics. These metrics can be useful for gauging the performance of a firewall platform, and provide a way to trend firewall performance over time. This article will describe several utilities that can be used to monitor the health and performance of a PF firewall.

Viewing performance counters with pfctl

The pfctl utility provides a command line interface to monitor and control the PF firewall module. pfctl contains options to display firewall configuration information and a variety of performance and status counters. To display the status and performance counters, the “info” parameter can be passed to pfctl’s “-s” (show values) option:

$ sudo pfctl -s info``

Status: Enabled for 2 days 10:33:04           Debug: Urgent

Hostid: 0xdf1a2b73

Interface Stats for tun0              IPv4             IPv6
  Bytes In                       577197106                0
  Bytes Out                       30295580                0
  Packets In
    Passed                          564503                0
    Blocked                           1494                0
  Packets Out
    Passed                          386562                0
    Blocked                              0                0

State Table                          Total             Rate
  current entries                        1
  searches                         2164707           10.3/s
  inserts                            23601            0.1/s
  removals                           23600            0.1/s
Counters
  match                            1226533            5.8/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              0            0.0/s
  memory                                 0            0.0/s

In addition to displaying the counters listed above, the show values option can be used to display firewall rulesets, NAT table entries, state table entries, and ALTQ queues. To get detailed usage data for each show values option, pfctl can be invoked with the “-v” (verbose) option. The following example shows how to get detailed usage data for each firewall rule:

$ sudo pfctl -v -s rules``

@0 scrub in all fragment reassemble
  [ Evaluations: 3884714   Packets: 1917431   Bytes: 0           States: 0     ]

@1 block drop in log on tun0 all
  [ Evaluations: 1200737   Packets: 1368      Bytes: 170289      States: 0     ]

The pfctl(8) man page contains the full list of values that can be be passed to the show values option, and covers pfctl’s advanced options.

Viewing active connections with pftop

The pftop utility provides a “top” like view of the PF state table. pftop displays source and destination IP addresses, TCP and UDP port numbers, packets and bytes transmitted, the age of a connection, and the time left until a connection will be removed from the state table. pftop contains several options to control how data is displayed, and is invoked by running the “pftop” executable from the command line:

$ sudo pftop``

pftop can be installed from the OpenBSD ports collection, or downloaded from the pftop website.

Monitoring logged packets with tcpdump

PF can record network packet headers and data when the log key word is used with a rule. Log files can be valuable for debugging rules, understanding traffic flows, and finding performance bottlenecks. When a packet matches a rule with the log key word, the headers and packet body are sent to the pflog pseudo-device. Once a packet is logged to the pflog pseudo-device, The tcpdump utility can be used to print the packet’s contents in real time:

$ sudo tcpdump -i pflog0 -o -ttt -vv -e -n``

Nov 26 00:58:32.001036 rule 4/0(match): block in on tun0: 1.2.3.4.4087 > 5.6.7.8.www: S [tcp sum ok] (src OS: Windows XP SP1, Windows 2000 SP2+) 3615243359:3615243359(0) win 16384 <mss 1440,nop,nop,sackOK> (DF) (ttl 119, id 7135)
        ........

This example prints protocol headers, timestamps, and an OS fingerprint (using the signatures available in /etc/pf.os) for each packet logged to the pflog pseudo-device. tcpdump’s “-r” option can be used to process PF log files written to the file system by the pflogd daemon:

$ sudo tcpdump -r /var/log/pflog -o -ttt -vv -e -n dst port 80``

Nov 26 00:58:32.001036 rule 4/0(match): block in on tun0: 1.2.3.4.4087 > 5.6.7.8.www: S [tcp sum ok] (src OS: Windows XP SP
1, Windows 2000 SP2+) 3615243359:3615243359(0) win 16384 <mss 1440,nop,nop,sackOK> (DF) (ttl 119, id 7135)
        ........

This example uses tcpdump’s filtering capabilities to limit the results to connections with a destination port of 80. tcpdump also allows connections to be filtered by IP address, hostname, Ethernet address, TCP flags, etc.

Summarizing log files with fwanalog

fwanalog is a firewall log file analysis tool that can be used to parse and summarize the log files from several firewall packages (e.g., PF, Checkpoint, ipfilter, iptables, Cisco PIX ). fwanalog uses the analog log file analysis program, and a simple straight forward configuration file to generate reports. The software is available as source code from the fwanalog website.

The fwanalog package contains several sample configuration files, each used to represent a different firewall platform. The “fwanalog.opts.openbsd3” configuration file contains the definitions required to process PF log files on OpenBSD 3.X platforms. When the fwanalog.sh script is invoked, it will process the contents of the configuration file named “fwanalog.opts” by default. The fwanalog documentation recommends creating a symbolic link to point to the platform specific configuration file:

$ ln -s fwanalog.opts.openbsd3 fwanalog.opts``

The fwanalog.opts configuration file contains a series of variables to define the log file locations, the format of each log file, and the location to store the reports. The customizable variables are summarized below:

outdir           - Location to store reports
logformat        - Format of the log file to analyze
inputfiles_mask  - The log file name (supports wildcards)
inputfiles_dir   - The directory with the log files
inputfiles_mtime - The maximum age of the log files to parse

The following configuration file can be used to parse PF log files on the OpenBSD platform:

#!/bin/sh

###########################################################################
#
#       User-changeable options for fwanalog.sh
#
#       $Id: fwanalog.opts.openbsd,v 1.19 2003/11/25 17:11:31 bb Exp $
#
###########################################################################
outdir="/usr/local/fwanalog/reports"
# The directory where the output goes to, without / at the end. You need write
# permissions, of course, and should secure this directory with permissions,
# minefields, guard dogs etc. It will be created if you don`t have it yet.

logformat="pf_30"
# What log format your firewall writes.
# Currently available options:
#       iptables        Linux 2.4 iptables              (probably in /var/log/messages)
#       ipchains        Linux 2.2 ipchains              (probably in /var/log/messages)
#       ipf                     BSD/Solaris ipfilter    (probably in /var/log/ipflog)
#       openbsd         this was the same as ipf until OpenBSD 2.9; this also 
#                                       seems to work on NetBSD
#       freebsd         FreeBSD`s output format (probably in /var/log/ipflog)
#       solarisipf      Solaris 8.0 Intel ipf 3.4.20 (using ipmon -sn &)
#       pf_30           OpenBSD 3.0 pf binary log format
#                                       fwanalog *must* run on OpenBSD 3.0 for this to work 
#                                       (because of the special tcpdump of OpenBSD)
#       zynos           ZyNOS (ZyXEL, Netgear) logfile
#       pix                     Cisco Pix (tested with version 6.22/IOS)
#       watchguard      Watchguard Firebox
#       fw1                     Checkpoint Firewall-One (not fw-1 NG!)

# Feel free to program a parser for your firewall if it is not supported.
# See the comments in iptables() and ipf()
#
# The officially maintained formats are pf_30 and iptables.

inputfiles_mask="pflog*gz"      # The name of your logfiles, with a wildcard if you want
inputfiles_dir="/var/log"       # The directory where your logfiles are in,
                                                        #       e.g. /var/log
inputfiles_mtime="30"           # How old the logfiles can be
# You can change this to your log rotate interval + 1 day (so you never miss a logfile entry)
inputfiles=`find $inputfiles_dir -maxdepth 1 -name "$inputfiles_mask" -mtime -$inputfiles_mtime | sort -r`
# This should find the names of the logfiles you want to parse
# It MUST return the names in reverse order (chronologically) or you 
# will have LOTS of duplicate lines in your log.

onehost=dynip
# Available options: false true dynip

# Default: false

# Set to true if this firewall runs on one machine only and you want to see  
# the source hosts (not the protected target hosts) in the Blocked Packet 
# Report. This is suggested if you protect one server, but loses information
# if you protect a network.

# Set to "dynip" if your firewall has a dynamic IP address.

# After changing onehost, you must delete everything in $outdir!

sep_hosts=false
# Set to true if you want fwanalog to create a separate, additional report for
# each attacking host IP.
# WARNING: this can run for hours using 100 % CPU and consume lots of hard
# disk space (up to 25 kB per host) so you can easily fill up your server if
# too many packets from different hosts were blocked.
# Also, this makes only limited sense with onehost mode set to true.
# If you set this option after having used fwanalog, some hosts won`t be
# linked in the report. You can create a report for a host with the 
# "-a <IP-address>" command line option.

sep_packets=false
# Like sep_hosts, but for blocked packets.
# The corresponding command line option is "-p <packet>"

# Program invocations - add path if needed

analog="analog"
# Full pathname if you need, or "nice analog" if you want to de-priorize it

date="date"             # should be GNU date or one which can print the timezone.
                                # see "timezone" below
grep="grep"     # should be GNU grep
egrep="egrep"   # should be GNU egrep
zegrep="zegrep" # this is just a shellscript on most systems. If you don`t 
                                # have it, copy it from another Unix-lookalike.
gzcat="gzcat"   # needed only on OpenBSD 3.x
sed="sed"
perl="perl"
tcpdump="tcpdump"       # needed only on OpenBSD 3.x

timezone=`$date +%z`
# Which timezone the server is in. Correct if the server fwanalog runs on
# is not in the timezone the firewall is in.
# The %z option of date is supported on GNU/Linux and OpenBSD, 
# but apparently NOT on FreeBSD so you will have to insert your
# timezone difference (e.g. -0500) yourself or use GNU date.

Once the configuration file is modified to match your desired layout and platform fwanalog.sh can be executed to generate a report:

$ fwanalog.sh``

This will place the reports in the directory assigned to the “outdir” variable. The reports can be viewed with a web browser, or emailed as part of a routine reporting process.

Graphing PF Performance Data with pfstat

The pfstat utility can be used to collect and graph statistics exported through the /dev/pf pseudo-device. pfstat can be installed from the OpenBSD ports collection, or downloaded from the pfstat website.

pfstat requires the PF “loginterface” global configuration directive to be set in the pf.conf configuration file. This directive enables statistics collection for one of the physical interfaces in the firewall. The following pf.conf entry will collect statistics on the hme0 interface:

set loginterface hme0

Once statistics collecting is enabled, the pfstat utility can be invoked with the “-q” option. This will query the current value of each statistics counter, and printed the result to standard out:

$ pfstat -q``

1101400143 1101219586 483226347 25637411 0 0 496899 3866 325988 0 0 0 0 0 6 1692642 17030 17024 879499 0 2 0 0 0

pfstat uses this data to generate historical utilization graphs, so the data should be collected at periodic intervals if graphs are desired. The following cron job will collect statistics every five minutes, and write the results to “/var/log/pfstat/pfstat”:

*/5 * * * * /usr/local/bin/pfstat -q >> /var/log/pfstat/pfstat

To graph the data that is collected, a pfstat configuration file needs to be created. This file describes the graphs to generate, how to display the data, and where to store the output. The following example shows the pfstat configuration required to graph state table data:

image "/home/matty/pfstat/images/state_table.jpg" {
     from 3 months to now
     width 800 height 300
     left
        graph states_entries  label "state table entries"   color 0 255 0,
        graph states_searches label "state table searches"  color 255 0 0,
        graph states_inserts  label "state table insertions"   color 0 0 255,
        graph states_removals label "state table removals"  color 0 0 0
}

The pfstat configuration file contains one or more “image” directives. Each image directive is followed by the file name of the image to generate, and a set of curly braces to control the attributes of the image. The “from” and “to” keywords select the time interval to graph. The value that follows the “from” keyword contains an integer value and a time frame (minutes, hours, days, weeks, months, years) to control how far back pfstat will go when processing data. The “to” keyword controls how pfstat processes new data elements, and the special key word “now” indicates the current time.

The “height” and “width” directives set the size, in pixels, for the height and width of the image to be output. Two directives determine the horizontal alignment of text descriptions: “left” aligns text on the left side of the graph and “right” aligns text on the right side. The “graph” statements control which data is graphed, the label assigned to the graph, and the colors used to create the entries on the graph. As of pfstat version 1.7, pfstat can graph packets, bytes, state table information, and several miscellaneous packet counters. The full list of options that can be passed to the “graph” directive are described in the pfstat(8) man page.

Once the configuration file is created, we can execute pfstat, and pass the configuration and data file as arguments to the “-c” and “-d” options:

$ pfstat -c /etc/pfstat/pfstat.conf -d /var/log/pfstat/pfstat >/dev/null``

Here is a pfstat.conf to graph IPv4, IPv6, and state table information:

image "/var/www/htdocs/pfstat/ipv4_bytes.jpg" {
        from 3 months to now
        width 800 height 300
        left
                graph bytes_v4_in       label "incoming" color 0 255 0,
                graph bytes_v4_out      label "outgoing" color 255 0 0
}

image "/var/www/htdocs/pfstat/ipv6_bytes.jpg" {
        from 3 months to now
        width 800 height 300
        left
                graph bytes_v6_in       label "incoming" color 0 255 0,
                graph bytes_v6_out      label "outgoing" color 255 0 0
}

image "/var/www/htdocs/pfstat/ipv4_connections.jpg" {
        from 3 months to now
        width 800 height 300
        left
             graph packets_v4_in_pass  label "passed in v4"   color 0 255 0,
             graph packets_v4_out_pass label "passed out v4"  color 255 0 0,
             graph packets_v4_in_drop  label "dropped in v4"   color 0 0 255,
             graph packets_v4_out_drop label "dropped out v4"  color 255 0 255
}

image "/var/www/htdocs/pfstat/ipv6_connections.jpg" {
        from 3 months to now
        width 800 height 300
        left
             graph packets_v6_in_pass  label "passed in v4"   color 0 255 0,
             graph packets_v6_out_pass label "passed out v4"  color 255 0 0,
             graph packets_v6_in_drop  label "dropped in v4"   color 0 0 255,
             graph packets_v6_out_drop label "dropped out v4"  color 0 0 0 
}

image "/var/www/htdocs/pfstat/state_table.jpg" {
        from 3 months to now
        width 800 height 300
        left
             graph states_entries  label "state table entries"   color 0 255 0,
             graph states_searches label "state table searches"  color 255 0 0,
             graph states_inserts  label "state table insertions"   color 0 0 255,
             graph states_removals label "state table removals"  color 0 0 0
}

Conclusion

This article provided an introduction to several tools that can be used to monitor a PF firewall. The PF FAQ and application manual pages contain additional details on advanced topic, and additional options that can be used to customize each application.

References

The following references were used while writing this article:

Acknowledgements

Ryan would like to thank the OpenBSD, PF, pftop, pfstat, tcpdump, analog and fwanalog developers for their awesome work!

Originally published in the September ‘05 issue of SysAdmin Magazine