While doing a bit of research tonight I came across a reference to mod_substitute. This nifty module allows you to substitute text in the HTTP request body, which provides an easy way to do things similar to the following:
<Location /private>
AddOutputFilterByType SUBSTITUTE text/html
Substitute s/SECRET/XXXXX/ni
</Location>
I digs me some Apache!
Building and maintaining Apache chroot environments can be a royal pain. Creating a chroot environment for Apache requires you to first identify all the libraries and applications that are required to run the httpd processes. Once you identify the dependencies, you need to create a chroot environment that contains these files. After you successfully create the chroot environment, you need to update it when security and reliability updates are released. This can be a time consuming process, and even though several tools (e.g., mock, makejail, etc.) exist to ease this process, there is still a fair amount of work that needs to occur to get things running properly.
One way to get around the hassles of creating chroot environments is to use mod_chroot. Mod_chroot will issue the chroot() system call after the runtime linker loads dependent libraries, and Apache processes its configuration file and opens the access and error logs. Delaying the chroot() system call until after Apache is initialized can greatly reduce the amount of work required to configure the chroot environment, since libraries don’t need to be copied* into the jail, and logs and configuration files can live outside of the chroot environment.
Installing and configuring mod_chroot is a snap. To compile and install mod_chroot from source, you can use the apxs utility from the Apache installation you want to run in the chroot environment:
$ tar xfvz mod_chroot-0.5.tar.gz
$ apxs -cai mod_chroot-0.5/src/apache20/mod_chroot.c
This will compile mod_chroot and install it the Apache loadable modules directory. To configure mod_chroot, you will first need to add a “LoadModule” directive to your httpd.conf to load mod_chroot:
LoadModule chroot_module modules/mod_chroot.so
Next you will need to add a “ChrootDir” directive with the directory you want to chroot Apache to:
ChrootDir /var/chroot/apache
The chroot directory should contain the content your web server serves, and any dependencies that can’t be resolved prior to the web server starting. Configuration is extremely simple, though there are a few caveats to watch out for. The web server cannot be gracefully restarted unless the web server configuration file is moved into the chroot, and programs that lazily load shared libraries will fail. Mod_chroot is an incredible module, and can definitely make managing chroot environments a whole lot easier! Nice!
* If a program uses dlopen() to load a library, you will need to copy the library into the chroot environment, or use the Apache “LoadFile” directive to load it at initialization time.
A few weeks back I helped a friend build PHP on a server with a non-standard directory structure. Changing the structure to use common defaults wasn’t an option, so we needed to adjust the PHP configure script to point to the pertinent places. Here is what we came up with:
$ export CPPFLAGS=”-I/home/apps/include -I/home/apps/include/mysql”
$ export LDFLAGS=”-L/home/apps/lib -L/home/apps/lib/mysql”
$ ./configure \\
--prefix=/home/apps/sfw/php-5.1.4 \\
--with-apxs2=/home/apps/httpd/bin/apxs \\
--with-libxml=/home/apps \\
--with-libxml-dir=/home/apps \\
--with-mysql=/home/apps \\
--with-zlib=/home/apps
This will build PHP using an apxs utility that resides in /home/apps/bin/apxs, and will look for the MySQL, libxml and zlib libraries and headers in /home/apps/lib and /home/apps/include.
Apache allows you to create hundreds of virtual host containers. Each container is required to have a ServerName directive, which contains the domain name associated with the virtual host. In addition to a server name, one ore more aliases can be associated with the virtual host with the ServerAlias directive. Aliases can contain a domain, or a regex that allows you to match based on some specific criteria. This is super useful, and allows you to do things like this:
NameVirtualHost 192.168.1.18:8080
<VirtualHost 192.168.1.18:8080>
ServerName foo.com
ServerAlias *.foo.com
</VirtualHost>
<VirtualHost 192.168.1.18:8080>
ServerName bar.com
ServerAlias *.bar.com
</VirtualHost>
This opens a whole slew of cool and interesting possibilities for virtual hosting. Niiiiice!
I support a fair number of Apache web server instances, and periodically need to measure the time it takes Apache (and it’s various modules) to process a request. On Solaris 10 hosts, I can use DTrace to retrieve this information on the fly. Since Solaris 9 and CentOS and Redhat Linux don’t come with DTrace, I use a different approach on these platforms.
To get the time when each request was received by Apache, I used mod_header’s “Header” directive, and “%t” option (time when a request was received, measured in milliseconds from the epoch), to add a response header with the time each request was received:
Header set X-Request-Received: %t
To get the total time Apache spent processing a request, I use mod_header’s “Header” directive, and “%D” option (milliseconds spent processing the request), to add a response header with the total time Apache spent processing each request:
Header set X-Request-Processing-Time: %D
Since I don’t always need the headers to be present, I like to be able to enable and disable them from the command line. The easiest way to do this is by enclosing the directives in a conditional block similar to the following:
<IfDefine RequestTime>
Header set X-Request-Received: %t
Header set X-Request-Processing-Time: %D
</IfDefine>
And using the httpd “-D” option to enable them:
$ httpd -k start -DRequestTime
After the headers are enabled, you will see entries similar to the following in each HTTP response:
$ curl -v http://192.168.1.13:8080/
* About to connect() to 192.168.1.13 port 8080 < ..... > < HTTP/1.1 200 OK < Date: Tue, 02 Jan 2007 17:59:42 GMT < Server: Apache/2.2.3 (Unix) DAV/2 < Last-Modified: Sat, 20 Nov 2004 20:16:24 GMT < ETag: "34d37-2c-4c23b600" < Accept-Ranges: bytes < Content-Length: 44 < X-Request-Received: t=1167760782452525 < X-Request-Processing-Time: D=3513 < Content-Type: text/html * Connection #0 to host 192.168.1.13 left intact * Closing connection #0
Similar capabilities are available for measuring request processing time on the client. Total time is helpful, but knowing how much of that time was consumed by Apache is invaluable!
On busy web servers, the process of writing to the access_log can sometimes overwhelm the spindles in a server. In Apache 2.0.41, the developers added the experimental “BufferedLogs” directive to buffer access_log entries in memory, and write them out as a single group. The documentation indicates that setting “BufferedLogs” to “On” enables buffered logging, but I couldn’t find anything that described how to configure the size of the buffer. After a bit of poking around in mod_log_config.c, I noticed that the size of the buffer was controlled by the LOG_BUFSIZE macro:
if (len + buf->outcnt > LOG_BUFSIZE) {
flush_log(buf);
}
if (len >= LOG_BUFSIZE) {
apr_size_t w;
str = apr_palloc(r->pool, len + 1);
for (i = 0, s = str; i < nelts; ++i) {
memcpy(s, strs[i], strl[i]);
s += strl[i];
}
w = len;
rv = apr_file_write(buf->handle, str, &w);
}
To see what LOG_BUFSIZE was set to, I searched for the value in mod_log_config.c. This is what my search turned up:
/* POSIX.1 defines PIPE_BUF as the maximum number of bytes that is * guaranteed to be atomic when writing a pipe. And PIPE_BUF >= 512 * is guaranteed. So we'll just guess 512 in the event the system * doesn't have this. Now, for file writes there is actually no limit, * the entire write is atomic. Whether all systems implement this * correctly is another question entirely ... so we'll just use PIPE_BUF * because it's probably a good guess as to what is implemented correctly * everywhere. */ #ifdef PIPE_BUF #define LOG_BUFSIZE PIPE_BUF #else #define LOG_BUFSIZE (512) #endif
Nifty! So if PIPE_BUF is defined, that will be used as the size. Now to see what the value of PIPE_BUF is set to on my CentOS 4.4 server:
$ cd /usr/include && find . | xargs grep PIPE_BUF
./linux/limits.h:#define PIPE_BUF 4096 /* # bytes in atomic write to a pipe */
So PIPE_BUF is set to a 4k buffer on my CentOS 4.4 server, and this value will be used if it exists. I am curious to see if there are any downsides to using extremely large buffers (hosting providers might be interested in using something larger than 4k). More testing is needed …
When debugging problems with web applications, it is often useful to display the HTTP request and response headers along with the HTTP entity bodies. There are numerous ways to do this, and I covered several client centric solutions in my SysAdmin article Debugging Web Applications. Client side tools are extremely useful for pinpointing problems, but periodically it is helpful to see the raw requests from the client, and the responses from the server. If you happen to be running Apache, you can use the dumpio module to dump the request and response data to the Apache error log.
Setting up dumpio is super easy. To enable the dumpio module during the Apache build process, you can pass the “–enable-dumpio=shared” option to the configure script:
$ ./configure –enable-mods-shared=most –enable-dumpio-shared –with-mpm=worker \
–enable-deflate=shared –prefix=/opt/apache-2.2.3
To enable the module, you will first need to load the module with “LoadModule” directive. Once the module is configured to load, you can add the “DumpIOInput On” directive to to dump incoming data, and “DumpIOOutput On” to dump data sent by Apache. This will result in messages similar to the following appearing in the Apache error logfile:
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio: dumpio_in (data-HEAP): 16 bytes [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio: dumpio_in (data-HEAP): GET / HTTP/1.1\r\n [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio: dumpio_in (data-HEAP): 13 bytes [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio: dumpio_in (data-HEAP): Accept: */*\r\n [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio: dumpio_in (data-HEAP): 21 bytes [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio: dumpio_in (data-HEAP): Accept-Language: en\r\n [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio: dumpio_in (data-HEAP): 32 bytes [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio: dumpio_in (data-HEAP): Accept-Encoding: gzip, deflate\r\n [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio: dumpio_in (data-HEAP): 24 bytes [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio: dumpio_in (data-HEAP): Connection: keep-alive\r\n [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio: dumpio_in (data-HEAP): 20 bytes [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio: dumpio_in (data-HEAP): Host: 192.168.9.24\r\n [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio: dumpio_in (data-HEAP): 2 bytes [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio: dumpio_in (data-HEAP): \r\n [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(129): mod_dumpio: dumpio_out [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio: dumpio_out (data-HEAP): 281 bytes [Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio: dumpio_out (data-HEAP): HTTP/1.1 200 OK\r\nDate: Sat, 28 Oct 2006 14:52:31 GMT\r\nServer: Apache/2.2.3 (Unix)\r\nLast-Modified: Sat, 20 Nov 2004 20:16:24 GMT\r\nETag: "45b81-2c-4c23b600"\r\nAccept-Ranges: bytes\r\nContent-Length: 44\r\nKeep-Alive: timeout=5, max=100\r\nConnection: Keep-Alive\r\nContent-Type: text/html\r\n\r\n <.....>
Dumpio is an extremely useful module, and is an invaluable tool for debugging web application.
I have been using PHP for quite some time, and have developed a love-hate relationship with it. The ability to rapidly create dynamic web applications is extremely powerful, but PHP’s absymal security track record often leads me to wonder if I should be using it (especially since a fair number of opensource PHP applications are poorly coded, and using them can put your hosting platform at risk).
Over the past few months, I have spent a fair amount of time researching ways to protect my PHP enabled web servers from insecure code, and unknown PHP vulnerabilities. There are a number of ways you can tackle PHP security, the most important being designing and writing secure PHP code. The second way to protect your PHP applications is by applying the suhosin (formerly known as the hardended PHP patch) patch, which integrates a number of advanced security protections directly into the PHP runtime environment. The third method is to use an operating system that protects against stack and heap overflows, and the the last method is to tighten up the runtime environment by enabling one or more variables in the php.ini file.
Installing suhosin is extremely easy, and the benefits it provides are described on the suhosin website. To integrate suhosin into the PHP source code, you will first need to download and extract the PHP source code (you should also verify the public key signatures / MD5 checksums distributed with the source code):
$ wget http://us3.php.net/get/php-5.1.6.tar.gz/from/www.php.net/mirror
$ gtar xfvz php-5.1.6.tar
Once the PHP source code has been extracted, you will need to retrieve and apply the suhosin patch to the PHP source code (you can also compile suhosin as an extension, but I prefer to use the patch):
$ wget http://www.hardened-php.net/suhosin/_media/suhosin-patch-5.1.6-0.9.5.patch.gz
$ gunzip suhosin-patch-5.1.6-0.9.5.patch.gz
$ gpatch -p0 < suhosin-patch-5.1.6-0.9.5.patch
patching file php-5.1.6/configure
patching file php-5.1.6/configure.in
patching file php-5.1.6/ext/standard/basic_functions.c
patching file php-5.1.6/ext/standard/dl.c
patching file php-5.1.6/ext/standard/file.c
patching file php-5.1.6/ext/standard/file.h
patching file php-5.1.6/ext/standard/info.c
patching file php-5.1.6/ext/standard/syslog.c
patching file php-5.1.6/main/fopen_wrappers.c
patching file php-5.1.6/main/main.c
patching file php-5.1.6/main/php_config.h.in
patching file php-5.1.6/main/php.h
patching file php-5.1.6/main/php_logos.c
patching file php-5.1.6/main/snprintf.c
patching file php-5.1.6/main/spprintf.c
patching file php-5.1.6/main/suhosin_globals.h
patching file php-5.1.6/main/suhosin_logo.h
patching file php-5.1.6/main/suhosin_patch.c
patching file php-5.1.6/main/suhosin_patch.h
patching file php-5.1.6/main/suhosin_patch.m4
patching file php-5.1.6/sapi/apache/mod_php5.c
patching file php-5.1.6/sapi/apache2filter/sapi_apache2.c
patching file php-5.1.6/sapi/apache2handler/sapi_apache2.c
patching file php-5.1.6/sapi/cgi/cgi_main.c
patching file php-5.1.6/sapi/cli/php_cli.c
patching file php-5.1.6/TSRM/TSRM.h
patching file php-5.1.6/TSRM/tsrm_virtual_cwd.c
patching file php-5.1.6/TSRM/tsrm_virtual_cwd.h
patching file php-5.1.6/win32/build/config.w32
patching file php-5.1.6/Zend/Makefile.am
patching file php-5.1.6/Zend/zend_alloc.c
patching file php-5.1.6/Zend/zend_alloc.h
patching file php-5.1.6/Zend/zend.c
patching file php-5.1.6/Zend/zend_canary.c
patching file php-5.1.6/Zend/zend_compile.c
patching file php-5.1.6/Zend/zend_compile.h
patching file php-5.1.6/Zend/zend_constants.c
patching file php-5.1.6/Zend/Zend.dsp
patching file php-5.1.6/Zend/zend_errors.h
patching file php-5.1.6/Zend/zend.h
patching file php-5.1.6/Zend/zend_hash.c
patching file php-5.1.6/Zend/zend_llist.c
patching file php-5.1.6/Zend/ZendTS.dsp
After the suhosin patch is applied, you can configure and build PHP according to the PHP installation guide. The following example shows how to compile PHP with the most basic options:
$ ./configure –prefix=/opt/apache/php-5.1.6 –with-apxs2=/opt/apache/bin/apxs
$ make
$ make install
$ ln -s /opt/apache/php-5.1.6 /opt/apache/php
To verify that your module is built with the suhosin patch, you can run the php utility with the the “-i” (print PHP information) option:
$ /opt/apache/php/bin/php -i |more
phpinfo()
PHP Version => 5.1.6
System => SunOS web 5.10 Generic_118855-19 i86pc
Build Date => Oct 10 2006 17:10:49
[ ….. ]
This server is protected with the Suhosin Patch 0.9.5
Copyright (c) 2006 Hardened-PHP Project
If the string “This server is protected with the Suhosin Patch” is not listed in the php output, something most likely went wrong, and you will need to check the steps you used to integrate suhosin into the PHP source code. If the string is present, you can turn your attention to enabling stack and heap protections in your operating system (each OS handles this differently, so check google if you want more information), and modifying the php.ini file with more secure defaults.
The php.ini file is used to configure the PHP runtime environment, and there are several variables that can be set in the initiatialization file to provide a more secure execution environment. On the servers I support, I typically set the following variables (the items prefixed with a “;” are comments):
; Disable global variable registration
register_globals = Off
; Don’t display PHP errors
display_errors = Off
; Log all PHP errors to the Apache error log
log_errors = On
; Tell suhosin to log all security errors to syslog
suhosin.log.syslog = 511
suhosin.log.syslog.facility = 11
suhosin.log.syslog.priority = 4
; Verify that the uid the PHP script runs as is the
; same as the file they are trying to access, and
; escape backticks and disable shell exec()s
safe_mode = On
; Disable any and all execution of programs by
; setting safe_mode_exec_dir to an empty directory
; (and one that cannot have programs uploaded to it).
safe_mode_exec_dir = /opt/web/empty
; List the set of variables a user cannot change with putenv()
safe_mode_protected_env_vars = LD_PRELOAD, LD_LIBRARY_PATH, LD_RUN_PATH, PATH
; Set the directory where PHP programs can be run from
open_basedir = /opt/web/php_content
; Tell the PHP engine not to append anything to the
; web server “Server” header (NOTE: set ServerTokens
; to Prod if you are using Apache)
expose_php = Off
; Escape single quotes, double quotes, backslashes and
; NUL characters with a backslash
magic_quotes_gpc = On
; Disable dynamic loading of PHP extensions (this should be
; covered by safe_mode, but it’s ok to be extra safe)
enable_dl = Off
; Disable HTTP uploads to the server
file_uploads = Off
; Disable remote file opening
allow_url_fopen = off
Several of these variables WILL break existing applications, so you should evaluate their use prior to enabling them in a production environment. The combination of secure coding practices, operating system stack and heap overflow protections, suhosin, and a secure php.ini file should be used on every server that hosts a PHP runtime environment. If you take additional steps to secure PHP on your servers, please leave me a comment. I am always looking for new and interesting ways to protect the services I support!