Buffering Apache logfiles with the BufferedLogs directive

On busy web servers, the process of writing to the access_log can sometimes overwhelm the spindles in a server. In Apache 2.0.41, the developers added the experimental “BufferedLogs” directive to buffer access_log entries in memory, and write them out as a single group. The documentation indicates that setting “BufferedLogs” to “On” enables buffered logging, but I couldn’t find anything that described how to configure the size of the buffer. After a bit of poking around in mod_log_config.c, I noticed that the size of the buffer was controlled by the LOG_BUFSIZE macro:

if (len + buf->outcnt > LOG_BUFSIZE) {      
      flush_log(buf);
}
if (len >= LOG_BUFSIZE) {
      apr_size_t w;

      str = apr_palloc(r->pool, len + 1);
      for (i = 0, s = str; i < nelts; ++i) {
          memcpy(s, strs[i], strl[i]);
          s += strl[i];
      }
      w = len;
      rv = apr_file_write(buf->handle, str, &w);
}

To see what LOG_BUFSIZE was set to, I searched for the value in mod_log_config.c. This is what my search turned up:

/* POSIX.1 defines PIPE_BUF as the maximum number of bytes that is
 * guaranteed to be atomic when writing a pipe.  And PIPE_BUF >= 512
 * is guaranteed.  So we'll just guess 512 in the event the system
 * doesn't have this.  Now, for file writes there is actually no limit,
 * the entire write is atomic.  Whether all systems implement this
 * correctly is another question entirely ... so we'll just use PIPE_BUF
 * because it's probably a good guess as to what is implemented correctly
 * everywhere.
 */
#ifdef PIPE_BUF
#define LOG_BUFSIZE     PIPE_BUF
#else
#define LOG_BUFSIZE     (512)
#endif

Nifty! So if PIPE_BUF is defined, that will be used as the size. Now to see what the value of PIPE_BUF is set to on my CentOS 4.4 server:

$ cd /usr/include && find . | xargs grep PIPE_BUF
./linux/limits.h:#define PIPE_BUF 4096 /* # bytes in atomic write to a pipe */

So PIPE_BUF is set to a 4k buffer on my CentOS 4.4 server, and this value will be used if it exists. I am curious to see if there are any downsides to using extremely large buffers (hosting providers might be interested in using something larger than 4k). More testing is needed …

2 thoughts on “Buffering Apache logfiles with the BufferedLogs directive”

Leave a Reply

Your email address will not be published. Required fields are marked *