Measuring website latency with http_ping

A year or so ago, I modified my ldap-ping.pl script to create a script (http-ping.pl) that would measure the time it took to retrieve a specific URI from a web server. While scouring the OpenBSD ports collection for website monitoring tools, I came across http_ping. This is a great tool for measuring the time it takes to retrieve a URI, and is a far superior tool to the one I wrote. Here is an example of http_ping in action:

$ http_ping http://prefetch.net/

6220 bytes from http://prefetch.net/: 290.232 ms (79.652c/89.816r/120.764d)
6220 bytes from http://prefetch.net/: 281.06 ms (70.564c/90.036r/120.46d)
6220 bytes from http://prefetch.net/: 281.274 ms (70.968c/89.61r/120.696d)
6220 bytes from http://prefetch.net/: 290.858 ms (80.459c/89.81r/120.589d)
^C
--- http://prefetch.net/ http_ping statistics ---
4 fetches started, 4 completed (100%), 0 failures (0%), 0 timeouts (0%)
total    min/avg/max = 281.06/285.856/290.858 ms
connect  min/avg/max = 70.564/75.4108/80.459 ms
response min/avg/max = 89.61/89.818/90.036 ms
data     min/avg/max = 120.46/120.627/120.764 ms

There are all kinds of nifty pieces of software stashed away in the OpenBSD ports collection, and I am on a mission to locate and blog about each and every one of them! :)

Measuring website latency

While reading a slew of information on web application monitoring, I came across echoping. This nifty little utility allows you to measure the time it takes to ping a server, perform an SMTP exchange with a mail server, or to request a URI from a web server. The following example uses the echoping “-h” (URI to fetch) option to measure the time it takes to request a URI from the prefetch.net web server:

$ echoping -v -4 -h /code/ssl-cert-check prefetch.net:80

This is echoping, version 5.2.0.

Trying to connect to internet address 66.148.84.65 80 to transmit 111 bytes...
Trying to send 256 bytes to internet address 66.148.84.65...
Connected...
TCP Latency: 0.025887 seconds
Sent (111 bytes)...
Application Latency: 0.032952 seconds
7075 bytes read from server.
Elapsed time: 0.092921 seconds

To measure the time it takes to complete an SMTP transaction with mail.prefetch.net, smokeping’s “-S” (perform SMTP exchange) option can be used:

$ echoping -v -4 -S mail.prefetch.net

This is echoping, version 5.2.0.

Trying to connect to internet address 206.222.17.179 25 to transmit 6 bytes...
Trying to send 256 bytes to internet address 206.222.17.179...
Connected...
TCP Latency: 0.051737 seconds
Sent (6 bytes)...
Application Latency: 0.052778 seconds
30 bytes read from server.
Elapsed time: 0.107167 seconds

echoping supports TLS and SSL, and the data generated by echoping can be graphed with cacti or smokeping.

Measuring packet loss

In continuing on with my commitment to describe my favorite network utilities, I bring to you mtr:

$ mtr -r -c 10 mail.prefetch.net

HOST: me                           Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. 10.238.4.1                    0.0%    10    9.2   8.2   5.4  12.6   2.1
  2. 68.86.108.13                  0.0%    10    9.2   9.4   6.3  14.9   2.8
  3. 68.86.106.45                  0.0%    10   15.1  10.1   6.9  17.5   3.5
  4. 68.86.106.13                  0.0%    10   14.0  11.0   5.7  22.1   4.8
  5. 68.86.106.9                   0.0%    10   14.6  11.4   7.2  18.5   3.6
  6. 12.118.120.89                 0.0%    10   10.2  10.3   8.2  14.6   2.2
  7. tbr1-p012201.attga.ip.att.ne  0.0%    10   23.8  25.8  21.8  33.9   3.6
  8. tbr2-cl1.wswdc.ip.att.net     0.0%    10   27.3  26.9  23.7  33.4   3.0
  9. ggr2-p390.wswdc.ip.att.net    0.0%    10   23.0  27.6  22.6  41.0   5.6
 10. so6-3-0-2488M.ar1.DCA3.gblx.  0.0%    10   26.0  30.4  23.9  43.3   6.3
 11. so0-0-0-622M.ar2.CLE1.gblx.n  0.0%    10  130.3  49.8  37.7 130.3  28.4
 12. Enet-Inc-Internet.so-0-3-3.a  0.0%    10   51.2  45.9  42.5  54.9   4.0
 13. extreme.xlhost.com            0.0%    10   44.2  67.1  41.4 261.4  68.3
 14. mail.prefetch.net            0.0%    10   48.3  56.6  41.5 172.9  40.9

mtr allows you to view hop-by-hop latency metrics, and is invaluable for finding busy devices between two endpoints. Some routers will place low QOS priorities on ICMP traffic, so it is important to capture traffic at various intervals to see if a device is truly overloaded.

Siege web benchmarking utility

While I am waiting for Apache to build with the libtool –debug option, I thought I would share my thoughts on the siege utility. Siege is an open source HTTP regression and debugging utility. I have used curl and ab in the past to debug and benchmark websites, but prefer siege, since it allows me to debug and benchmark websites with a single well written utility. To debug the HTTP requests to a server, you can run run siege with the “-g” (Grab headers) and “-u” (URL to grab) option:

$ siege -g -v -u https://prefetch.net

** siege 2.62
** Preparing 1 concurrent users for battle.
The server is now under siege...
GET / HTTP/1.0
Host: prefetch.net
Accept: */*
Accept-Encoding: *
User-Agent: JoeDog/1.00 [en] (X11; I; Siege 2.62)
Connection: close


HTTP/1.1 200 OK
Date: Thu, 14 Apr 2005 23:35:19 GMT
Server: Apache/2.0.52
Set-Cookie: Horde=b18606e131eb443ce6557c790ed17bdd; path=/; domain=prefetch.net
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Connection: close
Content-Type: text/html; charset=ISO-8859-1

This displays a summary of the request/entity headers for the entire HTTP transaction, and is super useful for debugging. To simulate 5-users hitting a website with a delay between successive page loads, siege can be run with the “-c” (user count), “-r” (number of times to run the test), and “-d” (add delay between 1 and the option passed to “-d”) options:

$ siege -b -v -c 5 -r 5 -d 10 -u https://prefetch.net

** siege 2.62
** Preparing 5 concurrent users for battle.
The server is now under siege...
HTTP/1.1 200   0.57 secs:     578 bytes ==> /
HTTP/1.1 200   0.66 secs:     578 bytes ==> /
HTTP/1.1 200   0.76 secs:     578 bytes ==> /
HTTP/1.1 200   0.76 secs:     578 bytes ==> /
HTTP/1.1 200   0.95 secs:     578 bytes ==> /
HTTP/1.1 200   0.74 secs:     500 bytes ==> /
HTTP/1.1 200   0.95 secs:     500 bytes ==> /
HTTP/1.1 200   0.90 secs:     500 bytes ==> /
HTTP/1.1 200   0.65 secs:     500 bytes ==> /
HTTP/1.1 200   0.95 secs:     500 bytes ==> /
HTTP/1.1 200   0.66 secs:     500 bytes ==> /
HTTP/1.1 200   0.67 secs:     500 bytes ==> /
HTTP/1.1 200   0.98 secs:     500 bytes ==> /
HTTP/1.1 200   0.98 secs:     500 bytes ==> /
HTTP/1.1 200   1.18 secs:     500 bytes ==> /
HTTP/1.1 200   0.68 secs:     500 bytes ==> /
HTTP/1.1 200   0.83 secs:     500 bytes ==> /
HTTP/1.1 200   0.80 secs:     500 bytes ==> /
HTTP/1.1 200   0.85 secs:     500 bytes ==> /
HTTP/1.1 200   0.73 secs:     500 bytes ==> /
HTTP/1.1 200   0.45 secs:     500 bytes ==> /
HTTP/1.1 200   0.78 secs:     500 bytes ==> /
HTTP/1.1 200   0.56 secs:     500 bytes ==> /
HTTP/1.1 200   0.62 secs:     500 bytes ==> /
HTTP/1.1 200   0.73 secs:     500 bytes ==> /
done.
Transactions:                     25 hits
Availability:                 100.00 %
Elapsed time:                   4.28 secs
Data transferred:               0.01 MB
Response time:                  0.78 secs
Transaction rate:               5.84 trans/sec
Throughput:                     0.00 MB/sec
Concurrency:                    4.53
Successful transactions:          25
Failed transactions:               0
Longest transaction:            1.18
Shortest transaction:           0.45

The Failed transactions, Response time, and Longest and Shortest transactions statistics can be used to profile a website, and to get an accurate picture of the time it takes to process a transaction. Siege also allows you to define a slew of URLs in a file called urls.txt, which will be processed during a connection from each user. I would like to thank Jeffrey Fulmer for writing the software! Make sure to check out his article in this month’s SysAdmin magazine.

Measuring Website Page Load Times

I was curious to see how long it would take to load “index.html” from my web server, so I created http-ping.pl to measure the time it took to retrieve a webpage from my web server:
$ http-ping.pl -s prefetch.net -p 80 -d 2

Querying HTTP server prefetch.net:80 every 2 seconds (Ctrl-C to stop):
Mon Nov 29 18:00:21 2004: TCP Connection Time=0.034s HTTP GET Time=0.052s [Normal Delay]
Mon Nov 29 18:00:23 2004: TCP Connection Time=0.035s HTTP GET Time=0.052s [Normal Delay]
Mon Nov 29 18:00:25 2004: TCP Connection Time=0.035s HTTP GET Time=0.055s [Normal Delay]
Mon Nov 29 18:00:27 2004: TCP Connection Time=0.035s HTTP GET Time=0.053s [Normal Delay]
Mon Nov 29 18:00:30 2004: TCP Connection Time=0.035s HTTP GET Time=0.051s [Normal Delay]

This will display the time it takes to perform the TCP three way handshake ( Syn, Syn/Ack, Ack), and GET the index page from a web server. This was a quick and dirty hack, and a full rewrite in C (w/ SSL support and timings!) is forthcoming. If you are interested in using http-ping.pl, you can grab the code from prefetch.net.