LDAP has become the Internet standard directory access protocol, and is used to access everything from DNS zone files to user account information. As companies and software vendors rely more heavily on LDAP directory servers, the need to measure server throughput and performance becomes imperative. This article will cover several tools that can be used to monitor the health and performance of an LDAP directory server, and will explain how ORCA can be used to trend directory server performance over time.
When investigating the performance characteristics of an LDAP server, the first place to start is usually the logfiles. The OpenLDAP server provides a flexible logging subsystem, and defines several log levels to control the verbosity of the logfiles:
Level Description
-1 enable all debugging
0 no debugging
1 trace function calls
2 debug packet handling
4 heavy trace debugging
8 connection management
16 print out packets sent and received
32 search filter processing
64 configuration file processing
128 access control list processing
256 stats log connections/operations/results
512 stats log entries sent
1024 print communication with shell back-ends
2048 print entry parsing debugging
each loglevel is additive, and can be configured through the slapd.conf “loglevel” directive, or passed as an argument to slapd’s “-d” option. The following example shows how to log detailed information on ACL and Search filter processing:
$ slapd -4 -f /etc/slapd.conf -u openldap -g openldap \
-h "ldap://ldap.prefetch.net ldaps://ldap.prefetch.net" -d 160
OpenLDAP will log all information by default to syslog’s LOCAL4 facility. If you want to use a different facility, you can pass the facility name to slapd’s “-l” option.
When LDAP clients and servers are separated by routers and firewalls, sporadic network behavior (e.g., lost TCP segments or bad CRCs) can cause unexpected application behavior. To help measure the latency between an LDAP client and server, I developed LDAP ping. ldap-ping.pl is written in Perl, and relies on the Time::HiRes, Getopt::Std, Net::LDAP and Net::LDAPS modules.
ldap-ping.pl works by opening a TCP connection to a directory server, issuing an anonymous bind, searching the RootDSE, and unbinding from the server. Each of these operations is measured with the Perl hi resolution timers, and displayed in “ping” style format:
$ ldap-ping.pl -s ldap.prefetch.net -p 389 -d 10
Querying LDAP server ldap.prefetch.net:389 every 10 seconds (Ctrl-C to stop):
Fri Nov 12 16:42:14 2004: new=0.025s, = bind=0.008s, search=0.067s, unbind=0.003s [local port=50377] [Normal Delay]
Fri Nov 12 16:42:25 2004: new=0.011s, = bind=0.001s, search=0.015s, unbind=0.001s [local port=50378] [Normal Delay]
Fri Nov 12 16:42:35 2004: new=0.010s, = bind=0.002s, search=0.015s, unbind=0.001s [local port=50379] [Normal Delay]
Fri Nov 12 16:42:45 2004: new=0.009s, = bind=0.002s, search=0.015s, unbind=0.001s [local port=50380] [Normal Delay]
The ldap-ping.pl script accepts three arguments; the “-s” option indicates the server to connect to, the “-p” option specifies the TCP port the directory server is listening on, and the “-d” option allows the admin to specify the delay between probes. If the pfiles binary is present, the script will also print the local port number.
The OpenLDAP server can be configured to provide real time performance statistics through the monitor branch. The currently available statistics include: bytes sent, entries returned to clients, total connections to the server, current active connections, read and write waiters, and individual operation (e.g., read, search, modify) breakdowns. The following example shows the slapd.conf directives required to setup the monitor branch, and limit read access to IP address 192.168.1.8:
database monitor
access to dn="cn=monitor"
by peername=192.168.1.8 read
by * none
Once the monitor branch is configured, we can view all of the available statistics with the “ldapsearch” utility:
$ `ldapsearch -x -b "cn=monitor" -H ldaps://ldap.prefetch.net objectclass=*`
We can also retrieve individual statistics by adjusting the search base:
$ ldapsearch -LLL -x -b "cn=Current,cn=Connections,cn=Monitor" -H ldaps://ldap.prefetch.net objectclass=*
dn: cn=Current,cn=Connections,cn=Monitor
objectClass: top
objectClass: monitor
objectClass: extensibleObject
cn: Current
description: 46
The ldapsearch utility allows us to grab a point in time snapshot of the performance metrics, but how do we get historical utilization data? To address this problem, I developed ldap-gather.pl. The ldap-gather.pl script collects statistics from the monitor branch, and writes them to the directory passed as an argument:
$ ldap-gather.pl -s ldap.prefetch.net -p 389 -d /usr/local/orca/var/orca/ldapallator/ldap.prefetch.net:389
The initial invocation of ldap-gather.pl will produce a text file with a header describing the data, and one line of actual data:
TIMESTAMP TOTAL_CONNECTIONS BYTES_SENT COMPLETED_OPERATIONS REFERRALS_SENT ENTRIES_SENT BIND_OPERATIONS UNBIND_OPERATIONS ADD_OPERATIONS DELETE_OPERATIONS MODIFY_OPERATIONS COMPARE_OPERATIONS SEARCH_OPERATIONS
1100656501 118 649271 165 0 4620 24 24 0 0 0 0 117
The ldap-gather.pl script will create a new data file if one doesn’t exist, and append new data if the file exists. The file name includes the word “ldapallator,” and a date timestamp (e.g., filename-YYYY-MO-DD-INDEX ):
$ ls -la | tail -1
-rw-r--r-- 1 orca other 6424 Nov 29 17:25 ldapallator-2004-10-29-000
To automate capturing data at various intervals, we can setup a cron job to run ldap-gather.pl:
5,15,25,35,45,55 * * * * sh -c "/usr/local/etc/ldap-gather.pl
-s ldap.prefetch.net
-p 389 -d /usr/local/orca/var/orca/ldapallator/ldap.prefetch.net:389"
This will cause ldap-gather.pl to grab performance data every ten minutes. Once the data has been captured, we can use ORCA to generate graphical performance reports.
The ORCA package contains a set of perl scripts and configuration files to graph arbitrary data. ORCA uses RRD for consolidating and storing data, and is configured with a single configuration file. ORCA utilizes the typical “configure,” “make” and “make install” process to build the packages. The ORCA Perl scripts use the Data::Dumper, Digest::MD5, Math::IntervalSearch, RRDs, and Storable modules, and includes the “make modules_install” option to integrate these with an existing Perl installation. Once the installation completes, “orca” can be invoked to ensure the build process completed successfully:
$ /usr/local/orca/bin/orca
/usr/local/orca/bin/orca: no configuration file specified
usage: /usr/local/orca/bin/orca [options] configuration_file
Options:
-daemon Run Orca in daemon mode
-gifs Output GIFs instead of PNGs
-logfile filename Output all messages
-no-html Update RRD files and images but not HTML files
-no-images Update RRD files but not image and HTML files
-once Run only once and do not continue to monitor input files
-verbose Verbose; list multiple times for increased verbosity
Orca understands the first unique command line option, i.e. -d for -daemon.
If the Perl interpreter cannot find one of the required modules, you will see a variety of errors on the console, and the process will exit. Once the Perl interpreter is happy, we can create an ORCA configuration file, and start putting the ldap-gather.pl data to good use.
The ORCA configuration file contains the directives required to locate and graph data. The configuration file contains three main sections. The first section defines several variables (e.g., base_dir, rrd_dir, html_dir ) that are used to control where the rrd and image files are stored. This section also contains several variables to define the formatting of the web pages ORCA generates.
The second section contains a series of “group” entries, which define the data to graph. A sample group entry to match the files collected with ldap-gather.pl is show below:
group ldapallator {
find_files /usr/local/orca/var/orca/ldapallator/(.*)/(?:ldapallator)-\d{4}-\d{2}-\d{2}(?:-\d{3,})?(?:\.(?:Z|gz
|bz2))?
column_description first_line
date_source column_name TIMESTAMP
interval 600
filename_compare sub {
my ($ay, $am, $ad) = $a =~ /-(\d{4})-(\d\d)-(\d\d)/;
my ($by, $bm, $bd) = $b =~ /-(\d{4})-(\d\d)-(\d\d)/;
if (my $c = (( $ay <=> $by) ||
( $am <=> $bm) ||
(($ad >> 3) <=> ($bd >> 3)))) {
return 2*$c;
}
$ad <=> $bd;
}
}
In this example, the “find_files” keyword tells ORCA what files to use as input (the files are matched with a regular expression). The “interval” keyword defines the number of seconds between updates for data files in this group, and “column_description” describes where the column descriptions are located. The “column_description” value “first_line” indicates that the column descriptions will be located on the first line of each file.
The third section contains a set of plots to indicate which items to graph. The following example shows what directives are required to graph the number of connections to a directory server:
plot {
title %g Total Connections
source ldapallator
data TOTAL_CONNECTIONS
data_type derive
line_type line2
legend Connections
y_legend Connections
data_min 0
}
The “plot” keyword defines a new graph, which contains the title specified in “title.” The “source” keyword indicates the “group” entry to source for this graph. Each graph uses the “data” keyword to reference specific columns in the data by name (e.g., TOTAL_CONNECTIONS is a column in each datafile ldap-gather.pl collects.) The “data_type” keyword specifies whether data will be graphed as an absolute value, a derivative of the previous value, or a counter. The X axis values of a graph can be controlled with the “data_min” and “data_max.” And finally, the legends allow you to place human readable descriptions in the PNG files ORCA creates.
ORCA can be invoked to process data once and exit, or setup to run as a daemon and continuously check for new data. The following example runs ORCA once to process all of the data since the last invocation:
$ /bin/sh -c "cd /usr/local/orca && /usr/local/orca/bin/orca -once /usr/local/orca/lib/ldapallator.cfg"
If ORCA runs successfully, the HTML and PNG files should be available in “html_dir.” If you choose not to run ORCA as a daemon, you will need to add a cron job to process the files at various intervals:
0 0 * * * /bin/sh -c "cd /usr/local/orca && /usr/local/orca/bin/orca -once
/usr/local/orca/lib/ldapallator.cfg" > /dev/null 2>&1
This will run ORCA everyday at midnight, and process the data from the previous day. Here is a complete ORCA configuration file for reference:
# Orca configuration file for orcallator files.
# Require at least this version of Orca.
require Orca 0.265
# base_dir is prepended to the paths find_files, html_dir, rrd_dir,
# and state_file only if the path does not match the regular
# expression ^\\?\.{0,2}/, which matches /, ./, ../, and \./.
base_dir /usr/local/orca/var/orca/rrd/ldapallator
# rrd_dir specifies the location of the generated RRD data files. If
# rrd_dir is a relative path, then it is made relative to base_dir if
# base_dir is set.
rrd_dir .
# state_file specifies the location of the state file that remembers
# the modification time of each source data file. If state_file is a
# relative path, then it is made relative to base_dir is base_dir is
# set.
state_file orca.state
# html_dir specifies the top of the HTML tree created by Orca.
html_dir /usr/local/apache2/htdocs
# By default create .meta tag files for all PNGs or GIFs so that the
# web browser will automatically reload them.
expire_images 1
# Find files at the following times:
# 0:10 to pick up new orcallator files for the new day.
# 1:00 to pick up late comer orcallator files for the new day.
# 6:00 to pick up new files before the working day.
# 12:00 to pick up new files during the working day.
# 19:00 to pick up new files after the working day.
find_times 0:10 1:00 6:00 12:00 19:00
# This defines the email address of people to warn when a file that is
# being updated constantly stops being updated. For mathematical
# expressions use the word `interval' to get the interval number for
# the data source.
warn_email admin@company.com
late_interval interval + 30
# These parameters specify which plots to generate.
generate_hourly_plot 1
generate_daily_plot 1
generate_weekly_plot 1
generate_monthly_plot 1
generate_quarterly_plot 1
generate_yearly_plot 1
# This sets the HTML markup that is placed at the very top of every
# web page and is primarly used to display the site's logo.
html_page_header <h3>Welcome To The LDAP Monitoring Page</h3>
# This sets the text that is placed in the pages' <title></title>
# element and just after the html_page_header HTML markup text is
# placed on the page.
html_top_title LDAP Status
# This sets the HTML markup that is placed at the bottom of every web
# page.
html_page_footer
<font face="verdana,geneva,arial,helvetica">
These plots brought to you by your local system administrator.
</font>
# This defines where the find the source data files and the format of
# those files. Notes about the fields:
# find_files
# You'll notice that all but the first () has the form (?:...).
# This tells Perl to match the expression but not save the matched
# text in the $1, $2, variables. Orca uses the matched text to
# generate a subgroup name, which is used to place files into
# different subgroups. Here, only the hostname should be used to
# generate a subgroup name, hence all the (?:...) for matching
# anything else.
# interval
# The interval here must match the interval used by orcallator to
# record data. Do not change this, as it has an effect on the
# generated RRD data files.
group ldapallator {
find_files /usr/local/orca/var/orca/ldapallator/(.*)/(?:ldapallator)-\d{4}-\d{1,2}-\d{1,2}(?:-\d{3,})?(?:\.(?:Z|gz|bz2))?
column_description first_line
date_source column_name TIMESTAMP
interval 600
filename_compare sub {
my ($ay, $am, $ad) = $a =~ /-(\d{4})-(\d\d)-(\d\d)/;
my ($by, $bm, $bd) = $b =~ /-(\d{4})-(\d\d)-(\d\d)/;
if (my $c = (( $ay <=> $by) ||
( $am <=> $bm) ||
(($ad >> 3) <=> ($bd >> 3)))) {
return 2*$c;
}
$ad <=> $bd;
}
}
plot {
title %g Total Connections
source ldapallator
data TOTAL_CONNECTIONS
data_type derive
line_type line2
legend Connections
y_legend Connections
data_min 0
}
plot {
title %g LDAP Operations
source ldapallator
data INITIATED_OPERATIONS
data COMPLETED_OPERATIONS
data_type derive
line_type line2
legend Initiated Operations
legend Completed Operations
y_legend Operations
data_min 0
}
plot {
title %g LDAP Operation Breakdown
source ldapallator
data BIND_OPERATIONS
data UNBIND_OPERATIONS
data ADD_OPERATIONS
data DELETE_OPERATIONS
data MODIFY_OPERATIONS
data COMPARE_OPERATIONS
data SEARCH_OPERATIONS
data MODRDN_OPERATIONS
data_type derive
line_type area
line_type stack
line_type stack
line_type stack
line_type stack
line_type stack
line_type stack
line_type stack
line_type stack
legend Bind Operations
legend Unbind Operations
legend Add Operations
legend Delete operations
legend Modify Operations
legend Compare Operations
legend Search Operations
legend Modrdn Operations
y_legend Operations
data_min 0
}
plot {
title %g Bytes Sent
source ldapallator
data BYTES_SENT
data_type derive
line_type line2
legend Bytes Transmitted
y_legend Bytes
data_min 0
}
plot {
title %g Entries Sent
source ldapallator
data ENTRIES_SENT
data_type derive
line_type line2
legend Entries Sent
y_legend Entries
data_min 0
}
plot {
title %g Referrrals
source ldapallator
data REFERRALS_SENT
data_type derive
line_type line2
legend Referrals Sent
y_legend Referrals
data_min 0
}
plot {
title %g Read/Write Waiters
source ldapallator
data WRITE_WAITERS
data READ_WAITERS
data_type derive
line_type line2
legend Write Waiters
legend Read Waiters
y_legend Waiters
data_min 0
}
This article provided an overview of several techniques to monitor an LDAP server. We used the OpenLDAP server in our examples, but the techniques are also applicable to the commercial directory servers. For additional information on ORCA, and the numerous ways it can be used to graph arbitrary data, please see the references.
The following references were used while writing this article:
Ryan would like to thank Clay McClure for the original work on ldap-ping.pl. Ryan would also like to thank the ORCA and OpenLDAP team members for their awesome contributions!
Originally published in the June ‘06 issue of SysAdmin Magazine