Measuring website latency with http_ping

A year or so ago, I modified my ldap-ping.pl script to create a script (http-ping.pl) that would measure the time it took to retrieve a specific URI from a web server. While scouring the OpenBSD ports collection for website monitoring tools, I came across http_ping. This is a great tool for measuring the time it takes to retrieve a URI, and is a far superior tool to the one I wrote. Here is an example of http_ping in action:

$ http_ping http://prefetch.net/

6220 bytes from http://prefetch.net/: 290.232 ms (79.652c/89.816r/120.764d)
6220 bytes from http://prefetch.net/: 281.06 ms (70.564c/90.036r/120.46d)
6220 bytes from http://prefetch.net/: 281.274 ms (70.968c/89.61r/120.696d)
6220 bytes from http://prefetch.net/: 290.858 ms (80.459c/89.81r/120.589d)
^C
--- http://prefetch.net/ http_ping statistics ---
4 fetches started, 4 completed (100%), 0 failures (0%), 0 timeouts (0%)
total    min/avg/max = 281.06/285.856/290.858 ms
connect  min/avg/max = 70.564/75.4108/80.459 ms
response min/avg/max = 89.61/89.818/90.036 ms
data     min/avg/max = 120.46/120.627/120.764 ms

There are all kinds of nifty pieces of software stashed away in the OpenBSD ports collection, and I am on a mission to locate and blog about each and every one of them! :)

HTTP Cookies

The HTTP protocol was originally designed to be stateless protocol, which provides some serious hurdles for applications that need to be “session” aware. To address this issue, the HTTP protocol added a lovely thing called cookies. Cookies are sent to a client with the “Set-Cookie:” attribute in the HTTP header, and contain an expiration date and a path to indicate which parts of the URL namespace the cookie applies to. To see which cookies a server attempts to set, the curl utilities “-s” and “-D -” options can be used:

$ curl -s -D – www.google.com | grep “Set-Cookie:”
Set-Cookie: PREF=ID=07ea94644d5a8aa2:TM=1136527125:LM=1136527125:S=Cs8EZN914EXiHOts; \
expires=Sun, 17-Jan-2038 19:14:07 GMT; \
path=/; domain=.google.com

Since your browser is a nice HTTP compliant entity, it will stores these cookies locally, and send them along with each HTTP request in a “Cookie:” header. If you would like to see how the adservers use cookies to provide clever marketing, you can watch cookies wiz by with the Firefox view cookie plugin. If you are super concerned about privacy, you can limit cookies to the domain you visited (this is an option with Firefox and Safari), or you can be super hard core and link your cookie repository to /dev/null (this of course causes issues with some sites).

Using StartTLS with HTTP connections

While catching up with some news groups today, I came across RFC 2817. This RFC describes HTTP protocol extensions to allow a client and server to initiate a TLS session over an existing connection. This has numerous benefits, and could definitely speed up web-based commerce (e.g., a dedicated secure connection is not required, slow start is avoided, etc.) . Now if only the browser developers would implement this! :)

Printing HTTP headers with curl

When debugging web applications, most adminstrators will review the HTTP request and response headers for errors. This information can be retrieved with Firefox’s HTTP Live headers plugin, ethereal, or with curl’s “-v” (make the operation more talkative) option:

$ curl -v http://www.google.com
* About to connect() to www.google.com port 80
* Trying 64.233.187.99… * connected
* Connected to www.google.com (64.233.187.99) port 80
> GET / HTTP/1.1
User-Agent: curl/7.13.1 (powerpc-apple-darwin8.0) libcurl/7.13.1 OpenSSL/0.9.7g zlib/1.2.3
Host: www.google.com
Pragma: no-cache
Accept: */*

< HTTP/1.1 200 OK < Cache-Control: private < Content-Type: text/html < Set-Cookie: PREF=ID=12; expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.com < Server: GWS/2.1 < Transfer-Encoding: chunked < Date: Tue, 18 Oct 2005 03:52:08 GMT The “>” and “<" characters are used to indicate the direction the requests are sent and received. The curl(1) manual page indicates that the "-i" (Include protocol headers in the output) option should print protocol headers, but for some reason it only prints the HTTP response headers: $ curl -i http://www.google.com

HTTP/1.1 200 OK
Cache-Control: private
Content-Type: text/html
Set-Cookie: PREF=ID=3eb84ab15b6724e3:TM=12; expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.com
Server: GWS/2.1
Transfer-Encoding: chunked
Date: Tue, 18 Oct 2005 03:54:38 GMT

When I get more time, I will have to go wandering through the curl source code to see why.