Top Button Left Second Button Left Third Button Left Fourth Button Left Fifth Button Left Sixth Button left Seventh Button left Eight Button Left

Monitoring OpenLDAP Server Performance


LDAP has become the Internet standard directory access protocol, and is used to access everything from DNS zone files to user account information. As companies and software vendors rely more heavily on LDAP directory servers, the need to measure server throughput and performance becomes imperative. This article will cover several tools that can be used to monitor the health and performance of an LDAP directory server, and will explain how ORCA can be used to trend directory server performance over time.

Logging


When investigating the performance characteristics of an LDAP server, the first place to start is usually the logfiles. The OpenLDAP server provides a flexible logging subsystem, and defines several log levels to control the verbosity of the logfiles:

Level Description
-1    enable all debugging
0     no debugging
1     trace function calls
2     debug packet handling
4     heavy trace debugging
8     connection management
16    print out packets sent and received
32    search filter processing
64    configuration file processing
128   access control list processing
256   stats log connections/operations/results
512   stats log entries sent
1024  print communication with shell back-ends
2048  print entry parsing debugging

each loglevel is additive, and can be configured through the slapd.conf "loglevel" directive, or passed as an argument to slapd's "-d" option. The following example shows how to log detailed information on ACL and Search filter processing:

$ slapd -4 -f /etc/slapd.conf -u openldap -g openldap \
  -h "ldap://ldap.prefetch.net  ldaps://ldap.prefetch.net" -d 160

OpenLDAP will log all information by default to syslog's LOCAL4 facility. If you want to use a different facility, you can pass the facility name to slapd's "-l" option.

Monitoring Operation Times


When LDAP clients and servers are separated by routers and firewalls, sporadic network behavior (e.g., lost TCP segments or bad CRCs) can cause unexpected application behavior. To help measure the latency between an LDAP client and server, we developed ldap-ping.pl. ldap-ping.pl is written in Perl, and relies on the Time::HiRes, Getopt::Std, Net::LDAP and Net::LDAPS modules.

ldap-ping.pl works by opening a TCP connection to a directory server, issuing an anonymous bind, searching the RootDSE, and unbinding from the server. Each of these operations is measured with the Perl hi resolution timers, and displayed in "ping" style format:

$ ldap-ping.pl -s ldap.prefetch.net -p 389 -d 10
Querying LDAP server ldap.prefetch.net:389 every 10 seconds (Ctrl-C to stop):
Fri Nov 12 16:42:14 2004: new=0.025s, = bind=0.008s, search=0.067s, unbind=0.003s [local port=50377] [Normal Delay]
Fri Nov 12 16:42:25 2004: new=0.011s, = bind=0.001s, search=0.015s, unbind=0.001s [local port=50378] [Normal Delay]
Fri Nov 12 16:42:35 2004: new=0.010s, = bind=0.002s, search=0.015s, unbind=0.001s [local port=50379] [Normal Delay]
Fri Nov 12 16:42:45 2004: new=0.009s, = bind=0.002s, search=0.015s, unbind=0.001s [local port=50380] [Normal Delay]

The ldap-ping.pl script accepts three arguments; the "-s" option indicates the server to connect to, the "-p" option specifies the TCP port the directory server is listening on, and the "-d" option allows the admin to specify the delay between probes. If the pfiles binary is present, the script will also print the local port number.

Monitoring Performance


The OpenLDAP server can be configured to provide real time performance statistics through the monitor branch. The currently available statistics include: bytes sent, entries returned to clients, total connections to the server, current active connections, read and write waiters, and individual operation (e.g., read, search, modify) breakdowns. The following example shows the slapd.conf directives required to setup the monitor branch, and limit read access to IP address 192.168.1.8:

database                   monitor

access to dn="cn=monitor"
        by peername=192.168.1.8   read
        by * none

Once the monitor branch is configured, we can view all of the available statistics with the "ldapsearch" utility:

$ ldapsearch -x -b "cn=monitor" -H ldaps://ldap.prefetch.net objectclass=*

We can also retrieve individual statistics by adjusting the search base:

$ ldapsearch -LLL -x -b "cn=Current,cn=Connections,cn=Monitor" -H ldaps://ldap.prefetch.net objectclass=*
dn: cn=Current,cn=Connections,cn=Monitor
objectClass: top
objectClass: monitor
objectClass: extensibleObject
cn: Current
description: 46

Collecting Performance Data


The ldapsearch utility allows us to grab a point in time snapshot of the performance metrics, but how do we get historical utilization data? To address this problem, I developed ldap-gather.pl. The ldap-gather.pl script collects statistics from the monitor branch, and writes them to the directory passed as an argument:

$ ldap-gather.pl -s ldap.prefetch.net -p 389 -d /usr/local/orca/var/orca/ldapallator/ldap.prefetch.net:389

The initial invocation of ldap-gather.pl will produce a text file with a header describing the data, and one line of actual data:

TIMESTAMP TOTAL_CONNECTIONS BYTES_SENT COMPLETED_OPERATIONS REFERRALS_SENT ENTRIES_SENT BIND_OPERATIONS UNBIND_OPERATIONS ADD_OPERATIONS DELETE_OPERATIONS MODIFY_OPERATIONS COMPARE_OPERATIONS SEARCH_OPERATIONS
1100656501 118 649271 165 0 4620 24 24 0 0 0 0 117

The ldap-gather.pl script will create a new data file if one doesn't exist, and append new data if the file exists. The file name includes the word "ldapallator," and a date timestamp (e.g., filename-YYYY-MO-DD-INDEX ):

$ ls -la | tail -1
-rw-r--r--   1 orca     other       6424 Nov 29 17:25 ldapallator-2004-10-29-000

To automate capturing data at various intervals, we can setup a cron job to run ldap-gather.pl:

5,15,25,35,45,55 * * * * sh -c "/usr/local/etc/ldap-gather.pl 
                            -s ldap.prefetch.net 
                            -p 389 -d /usr/local/orca/var/orca/ldapallator/ldap.prefetch.net:389"

This will cause ldap-gather.pl to grab performance data every ten minutes. Once the data has been captured, we can use ORCA to generate graphical performance reports.

Graphing LDAP Performance Data


The ORCA package contains a set of perl scripts and configuration files to graph arbitrary data. ORCA uses RRD for consolidating and storing data, and is configured with a single configuration file. ORCA utilizes the typical "configure," "make" and "make install" process to build the packages. The ORCA Perl scripts use the Data::Dumper, Digest::MD5, Math::IntervalSearch, RRDs, and Storable modules, and includes the "make modules_install" option to integrate these with an existing Perl installation. Once the installation completes, "orca" can be invoked to ensure the build process completed successfully:

$ /usr/local/orca/bin/orca
/usr/local/orca/bin/orca: no configuration file specified
usage: /usr/local/orca/bin/orca [options] configuration_file
Options:
  -daemon           Run Orca in daemon mode
  -gifs             Output GIFs instead of PNGs
  -logfile filename Output all messages
  -no-html          Update RRD files and images but not HTML files
  -no-images        Update RRD files but not image and HTML files
  -once             Run only once and do not continue to monitor input files
  -verbose          Verbose; list multiple times for increased verbosity
Orca understands the first unique command line option, i.e. -d for -daemon.

If the Perl interpreter cannot find one of the required modules, you will see a variety of errors on the console, and the process will exit. Once the Perl interpreter is happy, we can create an ORCA configuration file, and start putting the ldap-gather.pl data to good use.

The ORCA configuration file contains the directives required to locate and graph data. The configuration file contains three main sections. The first section defines several variables (e.g., base_dir, rrd_dir, html_dir ) that are used to control where the rrd and image files are stored. This section also contains several variables to define the formatting of the web pages ORCA generates.

The second section contains a series of "group" entries, which define the data to graph. A sample group entry to match the files collected with ldap-gather.pl is show below:

group ldapallator {
find_files              /usr/local/orca/var/orca/ldapallator/(.*)/(?:ldapallator)-\d{4}-\d{2}-\d{2}(?:-\d{3,})?(?:\.(?:Z|gz
|bz2))?
column_description      first_line
date_source             column_name TIMESTAMP
interval                600
filename_compare        sub {
                          my ($ay, $am, $ad) = $a =~ /-(\d{4})-(\d\d)-(\d\d)/;
                          my ($by, $bm, $bd) = $b =~ /-(\d{4})-(\d\d)-(\d\d)/;
                          if (my $c = (( $ay       <=>  $by) ||
                                       ( $am       <=>  $bm) ||
                                       (($ad >> 3) <=> ($bd >> 3)))) {
                            return 2*$c;
                          }
                          $ad <=> $bd;
                        }
}

In this example, the "find_files" keyword tells ORCA what files to use as input (the files are matched with a regular expression). The "interval" keyword defines the number of seconds between updates for data files in this group, and "column_description" describes where the column descriptions are located. The "column_description" value "first_line" indicates that the column descriptions will be located on the first line of each file.

The third section contains a set of plots to indicate which items to graph. The following example shows what directives are required to graph the number of connections to a directory server:

plot {
title                   %g Total Connections
source                  ldapallator
data                    TOTAL_CONNECTIONS
data_type               derive
line_type               line2
legend                  Connections
y_legend                Connections
data_min                0
}

The "plot" keyword defines a new graph, which contains the title specified in "title." The "source" keyword indicates the "group" entry to source for this graph. Each graph uses the "data" keyword to reference specific columns in the data by name (e.g., TOTAL_CONNECTIONS is a column in each datafile ldap-gather.pl collects.) The "data_type" keyword specifies whether data will be graphed as an absolute value, a derivative of the previous value, or a counter. The X axis values of a graph can be controlled with the "data_min" and "data_max." And finally, the legends allow you to place human readable descriptions in the PNG files ORCA creates.

ORCA can be invoked to process data once and exit, or setup to run as a daemon and continuously check for new data. The following example runs ORCA once to process all of the data since the last invocation:

$ /bin/sh -c "cd /usr/local/orca && /usr/local/orca/bin/orca -once /usr/local/orca/lib/ldapallator.cfg"

If ORCA runs successfully, the HTML and PNG files should be available in "html_dir." If you choose not to run ORCA as a daemon, you will need to add a cron job to process the files at various intervals:

0 0 * * * /bin/sh -c "cd /usr/local/orca && /usr/local/orca/bin/orca -once
          /usr/local/orca/lib/ldapallator.cfg" > /dev/null 2>&1

This will run ORCA everyday at midnight, and process the data from the previous day. Figure 3 contains a complete ORCA configuration file, and figures 4, 5 and 6 contain the ORCA graphs for total connections, total operations and the number of entries returned.

Conclusion


This article provided an overview of several techniques to monitor an LDAP server. We used the OpenLDAP server in our examples, but the techniques are also applicable to the commercial directory servers. For additional information on ORCA, and the numerous ways it can be used to graph arbitrary data, please see the references. If you have questions or comments on the article, please feel free to E-mail the author.

References


The following references were used while writing this article:

Acknowledgements


Ryan would like to thank Clay McClure for the original work on ldap-ping.pl. Ryan would also like to thank the ORCA and OpenLDAP team members for their awesome contributions!

* Originally published in the June '06 issue of SysAdmin Magazine