Dumping HTTP requests and reponses to the Apache error log

When debugging problems with web applications, it is often useful to display the HTTP request and response headers along with the HTTP entity bodies. There are numerous ways to do this, and I covered several client centric solutions in my SysAdmin article Debugging Web Applications. Client side tools are extremely useful for pinpointing problems, but periodically it is helpful to see the raw requests from the client, and the responses from the server. If you happen to be running Apache, you can use the dumpio module to dump the request and response data to the Apache error log.

Setting up dumpio is super easy. To enable the dumpio module during the Apache build process, you can pass the “–enable-dumpio=shared” option to the configure script:

$ ./configure –enable-mods-shared=most –enable-dumpio-shared –with-mpm=worker \
–enable-deflate=shared –prefix=/opt/apache-2.2.3

To enable the module, you will first need to load the module with “LoadModule” directive. Once the module is configured to load, you can add the “DumpIOInput On” directive to to dump incoming data, and “DumpIOOutput On” to dump data sent by Apache. This will result in messages similar to the following appearing in the Apache error logfile:

[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio:  dumpio_in (data-HEAP): 16 bytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio:  dumpio_in (data-HEAP): GET / HTTP/1.1\r\n
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio:  dumpio_in (data-HEAP): 13 bytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio:  dumpio_in (data-HEAP): Accept: */*\r\n
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio:  dumpio_in (data-HEAP): 21 bytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio:  dumpio_in (data-HEAP): Accept-Language: en\r\n
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio:  dumpio_in (data-HEAP): 32 bytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio:  dumpio_in (data-HEAP): Accept-Encoding: gzip, deflate\r\n
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio:  dumpio_in (data-HEAP): 24 bytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio:  dumpio_in (data-HEAP): Connection: keep-alive\r\n
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio:  dumpio_in (data-HEAP): 20 bytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio:  dumpio_in (data-HEAP): Host: 192.168.9.24\r\n
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio:  dumpio_in (data-HEAP): 2 bytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio:  dumpio_in (data-HEAP): \r\n
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(129): mod_dumpio: dumpio_out
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio:  dumpio_out (data-HEAP): 281 bytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio:  dumpio_out (data-HEAP): HTTP/1.1 200 OK\r\nDate: Sat, 28 Oct 2006 14:52:31 GMT\r\nServer: Apache/2.2.3 (Unix)\r\nLast-Modified: Sat, 20 Nov 2004 20:16:24 GMT\r\nETag: "45b81-2c-4c23b600"\r\nAccept-Ranges: bytes\r\nContent-Length: 44\r\nKeep-Alive: timeout=5, max=100\r\nConnection: Keep-Alive\r\nContent-Type: text/html\r\n\r\n

<.....>

Dumpio is an extremely useful module, and is an invaluable tool for debugging web application.

Securing PHP installations

I have been using PHP for quite some time, and have developed a love-hate relationship with it. The ability to rapidly create dynamic web applications is extremely powerful, but PHP’s absymal security track record often leads me to wonder if I should be using it (especially since a fair number of opensource PHP applications are poorly coded, and using them can put your hosting platform at risk).

Over the past few months, I have spent a fair amount of time researching ways to protect my PHP enabled web servers from insecure code, and unknown PHP vulnerabilities. There are a number of ways you can tackle PHP security, the most important being designing and writing secure PHP code. The second way to protect your PHP applications is by applying the suhosin (formerly known as the hardended PHP patch) patch, which integrates a number of advanced security protections directly into the PHP runtime environment. The third method is to use an operating system that protects against stack and heap overflows, and the the last method is to tighten up the runtime environment by enabling one or more variables in the php.ini file.

Installing suhosin is extremely easy, and the benefits it provides are described on the suhosin website. To integrate suhosin into the PHP source code, you will first need to download and extract the PHP source code (you should also verify the public key signatures / MD5 checksums distributed with the source code):

$ wget http://us3.php.net/get/php-5.1.6.tar.gz/from/www.php.net/mirror

$ gtar xfvz php-5.1.6.tar

Once the PHP source code has been extracted, you will need to retrieve and apply the suhosin patch to the PHP source code (you can also compile suhosin as an extension, but I prefer to use the patch):

$ wget http://www.hardened-php.net/suhosin/_media/suhosin-patch-5.1.6-0.9.5.patch.gz

$ gunzip suhosin-patch-5.1.6-0.9.5.patch.gz

$ gpatch -p0 < suhosin-patch-5.1.6-0.9.5.patch
patching file php-5.1.6/configure
patching file php-5.1.6/configure.in
patching file php-5.1.6/ext/standard/basic_functions.c
patching file php-5.1.6/ext/standard/dl.c
patching file php-5.1.6/ext/standard/file.c
patching file php-5.1.6/ext/standard/file.h
patching file php-5.1.6/ext/standard/info.c
patching file php-5.1.6/ext/standard/syslog.c
patching file php-5.1.6/main/fopen_wrappers.c
patching file php-5.1.6/main/main.c
patching file php-5.1.6/main/php_config.h.in
patching file php-5.1.6/main/php.h
patching file php-5.1.6/main/php_logos.c
patching file php-5.1.6/main/snprintf.c
patching file php-5.1.6/main/spprintf.c
patching file php-5.1.6/main/suhosin_globals.h
patching file php-5.1.6/main/suhosin_logo.h
patching file php-5.1.6/main/suhosin_patch.c
patching file php-5.1.6/main/suhosin_patch.h
patching file php-5.1.6/main/suhosin_patch.m4
patching file php-5.1.6/sapi/apache/mod_php5.c
patching file php-5.1.6/sapi/apache2filter/sapi_apache2.c
patching file php-5.1.6/sapi/apache2handler/sapi_apache2.c
patching file php-5.1.6/sapi/cgi/cgi_main.c
patching file php-5.1.6/sapi/cli/php_cli.c
patching file php-5.1.6/TSRM/TSRM.h
patching file php-5.1.6/TSRM/tsrm_virtual_cwd.c
patching file php-5.1.6/TSRM/tsrm_virtual_cwd.h
patching file php-5.1.6/win32/build/config.w32
patching file php-5.1.6/Zend/Makefile.am
patching file php-5.1.6/Zend/zend_alloc.c
patching file php-5.1.6/Zend/zend_alloc.h
patching file php-5.1.6/Zend/zend.c
patching file php-5.1.6/Zend/zend_canary.c
patching file php-5.1.6/Zend/zend_compile.c
patching file php-5.1.6/Zend/zend_compile.h
patching file php-5.1.6/Zend/zend_constants.c
patching file php-5.1.6/Zend/Zend.dsp
patching file php-5.1.6/Zend/zend_errors.h
patching file php-5.1.6/Zend/zend.h
patching file php-5.1.6/Zend/zend_hash.c
patching file php-5.1.6/Zend/zend_llist.c
patching file php-5.1.6/Zend/ZendTS.dsp

After the suhosin patch is applied, you can configure and build PHP according to the PHP installation guide. The following example shows how to compile PHP with the most basic options:

$ ./configure –prefix=/opt/apache/php-5.1.6 –with-apxs2=/opt/apache/bin/apxs

$ make

$ make install

$ ln -s /opt/apache/php-5.1.6 /opt/apache/php

To verify that your module is built with the suhosin patch, you can run the php utility with the the “-i” (print PHP information) option:

$ /opt/apache/php/bin/php -i |more
phpinfo()
PHP Version => 5.1.6

System => SunOS web 5.10 Generic_118855-19 i86pc
Build Date => Oct 10 2006 17:10:49

[ ….. ]

This server is protected with the Suhosin Patch 0.9.5
Copyright (c) 2006 Hardened-PHP Project

If the string “This server is protected with the Suhosin Patch” is not listed in the php output, something most likely went wrong, and you will need to check the steps you used to integrate suhosin into the PHP source code. If the string is present, you can turn your attention to enabling stack and heap protections in your operating system (each OS handles this differently, so check google if you want more information), and modifying the php.ini file with more secure defaults.

The php.ini file is used to configure the PHP runtime environment, and there are several variables that can be set in the initiatialization file to provide a more secure execution environment. On the servers I support, I typically set the following variables (the items prefixed with a “;” are comments):

; Disable global variable registration
register_globals = Off

; Don’t display PHP errors
display_errors = Off

; Log all PHP errors to the Apache error log
log_errors = On

; Tell suhosin to log all security errors to syslog
suhosin.log.syslog = 511
suhosin.log.syslog.facility = 11
suhosin.log.syslog.priority = 4

; Verify that the uid the PHP script runs as is the
; same as the file they are trying to access, and
; escape backticks and disable shell exec()s
safe_mode = On

; Disable any and all execution of programs by
; setting safe_mode_exec_dir to an empty directory
; (and one that cannot have programs uploaded to it).
safe_mode_exec_dir = /opt/web/empty

; List the set of variables a user cannot change with putenv()
safe_mode_protected_env_vars = LD_PRELOAD, LD_LIBRARY_PATH, LD_RUN_PATH, PATH

; Set the directory where PHP programs can be run from
open_basedir = /opt/web/php_content

; Tell the PHP engine not to append anything to the
; web server “Server” header (NOTE: set ServerTokens
; to Prod if you are using Apache)
expose_php = Off

; Escape single quotes, double quotes, backslashes and
; NUL characters with a backslash
magic_quotes_gpc = On

; Disable dynamic loading of PHP extensions (this should be
; covered by safe_mode, but it’s ok to be extra safe)
enable_dl = Off

; Disable HTTP uploads to the server
file_uploads = Off

; Disable remote file opening
allow_url_fopen = off

Several of these variables WILL break existing applications, so you should evaluate their use prior to enabling them in a production environment. The combination of secure coding practices, operating system stack and heap overflow protections, suhosin, and a secure php.ini file should be used on every server that hosts a PHP runtime environment. If you take additional steps to secure PHP on your servers, please leave me a comment. I am always looking for new and interesting ways to protect the services I support!

Tuning Apache for performance

I recently came across Colm MacCarthaigh’s Apache tuning presentation and technical white paper:

Tuning Apache and Linux for performance presentation:
http://www.stdlib.net/~colmmacc/Apachecon-EU2005/scaling-apache-presentation.pdf

Tuning Apache and Linux for performance paper:
http://www.stdlib.net/~colmmacc/Apachecon-EU2005/scaling-apache-handout.pdf

Colm is an admin at heanet, which runs some of the busiest web servers in the world. The presentation and white paper cover the entire software stack, which includes kernel, file system and of course Apache web server tuning. These are must reads for website administrators.

PHP security links

While reading up on PHP security, I came across several links that I thought I would pass on:

Security Focus article on securing PHP installations:
http://www.securityfocus.com/infocus/1706

Article on PHP attack vectors and defenses:
http://www.securereality.com.au/studyinscarlet.txt

Bugtraq discussion on PHP security:
http://www.securityfocus.com/archive/1/438417/30/0/threaded

Viewing Apache apxs compiler flags

If you have used apxs to build an Apache module, you may have wondered where apxs got the linker and compiler options since they are not embedded in the apxs Perl script? The linker and compiler flags are generated by the apr-config and apu-config scripts, which contain platform specific options that are assigned at Apache build time. These options can be viewed by running apr-config or apu-config with one or more options:

$ apr-config –cppflags –cflags
-DSOLARIS2=10 -D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT -g -O2 -Wall -Wmissing-prototypes -Wstrict-prototypes -Wmissing-declarations

The Apache build system is super interesting, and I love the fact that you can easily add modules to an existing Apache installation.