Apache DTrace module
The Apache DTrace module (mod_dtrace) utilizes the hook framework to add DTrace probes to the Apache web server. These probes can be used to observe and correlate web server and system behavior, and allow easy access to numerous pieces of realtime Apache data. The module comes as as two files. The first file contains the C code required to hook the module into Apache. The second file is the probe definition file. mod_dtrace has been tested with the Apache source distribution (2.0.55) and the version of Apache that resides in /usr/apache2 on Solaris 10 servers. If you have any comments, suggestions, scripts or feedback, shoot me an E-mail.
Code and build instructions
The Apache DTrace module and a set of build instructions can be downloaded from the following site:
Scripts
The scripts on this page were created to monitor a running Apache process, and to provide some insight into the Apache request processing flow. The results in some cases may be slightly skewed, but the overall results should help web administrators understand what is happening with an Apache web server.
*** NOTE***
Due to some issues with the DTrace "#include" support, these scripts currently only work under SPARC Solaris (you can adjust the offsets in the scripts if you are interested in using them under x86 or x64 Solaris)
View HTTP Requests
$ viewreqs.pl -d 5 Timestamp GETs POSTs HEADs TRACEs 02:05:59 0 0 0 0 02:06:04 12 0 0 0 02:06:09 15 0 0 0 02:06:14 59 0 0 0 02:06:19 68 0 0 0 02:06:24 70 0 0 0
View HTTP Requests By HTTPD Process
$ viewreqsbyproc.d Execname Process Requests Processed httpd 19103 16 httpd 19152 17 httpd 19101 17 httpd 19102 19 httpd 19104 20 httpd 19105 20
View HTTP Requests By Browser Type
$ viewbrowsers.pl Sampling browser data for 60 seconds... Browser GETs POSTs HEADs TRACEs Firefox 34 0 0 0 Safari 20 0 0 0 Wget 7 0 0 0
View HTTP Requests By Virtual Server
$ viewserver.pl -d 60 Sampling virtual server data for 60 seconds... Server GETs POSTs HEADs TRACEs www.foo.com 101 0 0 0 www.bar.com 23 0 0 0
View HTTP Response Codes
$ responsecodes.d Sampling response code data (Ctrl-C to stop): Response Code Count 404 4 200 22
View HTTP Response Codes by file and URI
$ responsecodes.pl Response Code URI File Name 200 / /tmp/apache/htdocs/index.html.var 304 /apache_pb.gif /tmp/apache/htdocs/apache_pb.gif 200 / /tmp/apache/htdocs/index.html.var 200 /apache_pb.gif /tmp/apache/htdocs/apache_pb.gif 404 /favicon.ico /tmp/apache/htdocs/favicon.ico $ responsecodes.pl -c 404 Response Code URI File Name 404 /favicon.ico /tmp/apache/htdocs/favicon.ico 404 /favicon.ico /tmp/apache/htdocs/favicon.ico 404 /favicon.ico /tmp/apache/htdocs/favicon.ico 404 /favicon.ico /tmp/apache/htdocs/favicon.ico
View HTTP 301/302 redirects
$ viewredirects.pl Response Code URI Requested Redirect Sent To Client 301 /yikes http://192.168.1.3:8080/yikes2 301 /foo http://192.168.1.3:8080/bar 301 /bling http://192.168.1.3:8080/crunk
View request sizes in a quantized distribution
$ requestsize.d
dtrace: script 'requestsize.d' matched 1 probe
httpd 5549
value ------------- Distribution ------------- count
-1 | 0
0 |@@@@@@@@@@@@@@@@@@@@@@@@ 43
1 | 0
2 | 0
4 | 0
8 | 0
16 | 0
32 | 0
64 | 0
128 | 0
256 |@@@@ 8
512 | 0
1024 | 0
2048 |@@@@@@@@@@@@ 22
4096 | 0
View requests in a top-like display (similar to the apachetop C program)
$ apachetop.pl
21:12:39 Requests: 347 ( 69/sec) Bytes: 35349b (7069b/sec, 101b/request)
Requests: GETs: 347 POSTs: 0 HEADs: 0 TRACEs: 0
Responses: 1XX: 0 2XX: 301 3XX: 46 4XX: 0 5XX: 0
Requests Requests/Sec Bytes Sent URI
13 2.6 13312 /blah
6 1.2 0 /manual/ja/mod/directives.html
6 1.2 0 /manual/ru/mod/directives.html
6 1.2 0 /manual/de/mod/directives.html
6 1.2 0 /manual/en/mod/directives.html
6 1.2 0 /manual/es/mod/directives.html
6 1.2 0 /manual/ko/mod/directives.html
2 0.4 0 /manual/ja/mod/
2 0.4 0 /manual/es/mod/
2 0.4 0 /manual/de/mod/
2 0.4 0 /manual/ko/mod/
2 0.4 0 /manual/en/mod/
1 0.2 0 /manual/ja/mod/mod_negotiation.html
1 0.2 0 /manual/ja/mod/mod_isapi.html
1 0.2 0 /manual/ko/mod/mod_file_cache.html
1 0.2 344 /manual/ko/ru/mod/directives.html
1 0.2 0 /manual/ja/mod/mod_asis.html
1 0.2 0 /manual/es/mod/mod_autoindex.html
1 0.2 0 /manual/ru/mod/beos.html
1 0.2 0 /manual/ko/mod/mod_setenvif.html
View Apache module function processing time
$ apachemodulefunctime `pgrep httpd` mod_alias.so Tracing Apache Module mod_alias.so (Control-C to stop) Module Function Time mod_alias.so alias_matches 558679 mod_alias.so fixup_redir 623683 mod_alias.so translate_alias_redir 938919 mod_alias.so try_alias_list 1169778 Total Time: 3291059
View Apache module processing time
$ apachemoduleprof `pgrep httpd`
Started Apache Request Profiler (Control-C to stop...)
*** New connection ***
Connection start: 2005 Dec 10 19:56:39
Connection stop : 2005 Dec 10 19:56:54
Method URI Count
GET /apache_pb.gif 1
GET / 1
Module Processing Time
mod_actions.so 43093 ns
mod_expires.so 66941 ns
mod_dir.so 92026 ns
mod_speling.so 100357 ns
mod_log_forensic.so 102187 ns
mod_userdir.so 102598 ns
mod_vhost_alias.so 104377 ns
mod_info.so 108378 ns
mod_autoindex.so 110851 ns
mod_usertrack.so 117790 ns
mod_asis.so 134288 ns
mod_cern_meta.so 138016 ns
mod_cgi.so 138344 ns
mod_imap.so 142541 ns
mod_headers.so 189257 ns
mod_status.so 205965 ns
mod_env.so 206865 ns
mod_dav.so 240208 ns
mod_access.so 245391 ns
mod_unique_id.so 321004 ns
mod_negotiation.so 331457 ns
mod_auth_digest.so 333495 ns
mod_dtrace.so 467512 ns
mod_rewrite.so 527799 ns
mod_include.so 874804 ns
mod_alias.so 1886377 ns
mod_log_config.so 3044315 ns
mod_mime.so 3684061 ns
mod_setenvif.so 4318186 ns
httpd 22451784 ns
Total 40 ms
Hook Name Module Hook Function Time
ap_run_pre_connection mod_dtrace.so apache_pre_connection 24398 ns
ap_run_create_connection mod_dtrace.so apache_create_connection 25998 ns
ap_run_handler mod_actions.so action_handler 43093 ns
ap_run_log_transaction mod_log_forensic.so log_after 43118 ns
ap_run_access_checker httpd do_nothing 46784 ns
ap_run_header_parser mod_setenvif.so match_headers 57872 ns
ap_run_post_read_request mod_log_forensic.so log_before 59069 ns
ap_run_insert_filter httpd core_insert_filter 61771 ns
ap_run_post_read_request mod_dtrace.so apache_receive_request 66115 ns
ap_run_access_checker mod_dtrace.so apache_check_access 66201 ns
ap_run_insert_filter mod_expires.so expires_insert_filter 66941 ns
ap_run_insert_filter mod_headers.so ap_headers_insert_output_filter 78055 ns
ap_run_handler mod_negotiation.so handle_map_file 90869 ns
ap_run_fixups mod_dir.so fixup_dir 92026 ns
ap_run_handler mod_dav.so dav_handler 96851 ns
ap_run_handler mod_rewrite.so handler_redirect 98021 ns
ap_run_fixups mod_speling.so check_speling 100357 ns
ap_run_translate_name mod_userdir.so translate_userdir 102598 ns
ap_run_translate_name mod_vhost_alias.so mva_translate 104377 ns
ap_run_translate_name mod_rewrite.so hook_uri2file 106062 ns
ap_run_handler mod_info.so display_info 108378 ns
ap_run_fixups mod_negotiation.so fix_encoding 109785 ns
ap_run_handler mod_autoindex.so handle_autoindex 110851 ns
ap_run_fixups mod_headers.so ap_headers_fixup 111202 ns
ap_run_fixups mod_rewrite.so hook_fixup 111632 ns
ap_run_fixups mod_usertrack.so spot_cookie 117790 ns
ap_run_fixups mod_auth_digest.so add_auth_info 118533 ns
ap_run_fixups httpd core_override_type 124018 ns
ap_run_map_to_storage httpd ap_send_http_trace 129160 ns
ap_run_type_checker mod_negotiation.so handle_multi 130803 ns
ap_run_handler mod_asis.so asis_handler 134288 ns
ap_run_log_transaction mod_dtrace.so apache_log_request 134483 ns
ap_run_fixups mod_cern_meta.so add_cern_meta_data 138016 ns
ap_run_handler mod_cgi.so cgi_handler 138344 ns
ap_run_handler mod_imap.so imap_handler 142541 ns
ap_run_fixups mod_dav.so dav_fixups 143357 ns
ap_run_create_request mod_dtrace.so apache_create_request 150317 ns
ap_run_pre_connection httpd core_pre_connection 153533 ns
ap_run_fixups mod_alias.so fixup_redir 199353 ns
ap_run_handler mod_status.so status_handler 205965 ns
ap_run_fixups mod_env.so fixup_env_module 206865 ns
ap_run_type_checker mod_rewrite.so hook_mimetype 212084 ns
ap_run_post_read_request mod_auth_digest.so parse_hdr_and_update_nc 214962 ns
ap_run_access_checker mod_access.so check_dir_access 245391 ns
ap_run_post_read_request mod_unique_id.so gen_unique_id 321004 ns
ap_run_create_request httpd http_create_request 505504 ns
ap_run_translate_name httpd ap_core_translate 542836 ns
ap_run_create_request httpd core_create_req 728807 ns
ap_run_create_connection httpd core_create_conn 874776 ns
ap_run_fixups mod_include.so include_fixup 874804 ns
ap_run_translate_name mod_alias.so translate_alias_redir 1687024 ns
ap_run_log_transaction mod_log_config.so multi_log_transaction 3044315 ns
ap_run_type_checker mod_mime.so find_ct 3684061 ns
ap_run_post_read_request mod_setenvif.so match_headers 4260314 ns
ap_run_map_to_storage httpd core_map_to_storage 8709887 ns
ap_run_handler httpd default_handler 10574708 ns