While perusing some of the new features in Solaris 10, I saw that rpcbind is now tcp_wrappers aware. This allows you to limit which services can send RPC requests to a server, and can be used to deny all RPC requests except those originating from the host itself. To enable tcp_wrappers and limit inbound requests to the loopback interface, the svccfg(1m) utility can be run to enter the SMF configuration console:
$ svccfg
Once in the SMF configuration console, you need to select the rpc/bind service with the ‘select’ command:
svc:>select network/rpc/bind
Once the service is selected, you can view the service properties using the ‘listprop’ command:
svc:/network/rpc/bind> listprop
config application
config/allow_indirect boolean true
config/verbose_logging boolean true
config/enable_tcpwrappers boolean true
fs dependency
fs/entities fmri svc:/system/filesystem/minimal
fs/grouping astring require_all
fs/restart_on astring none
fs/type astring service
identity dependency
identity/entities fmri svc:/system/identity:node
identity/grouping astring require_all
identity/restart_on astring refresh
identity/type astring service
sysidtool dependency
sysidtool/entities fmri svc:/system/sysidtool:net
sysidtool/grouping astring require_all
sysidtool/restart_on astring none
sysidtool/type astring service
routing dependency
routing/entities fmri svc:/network/initial:default
routing/grouping astring optional_all
routing/restart_on astring none
routing/type astring service
general framework
general/entity_stability astring Unstable
general/single_instance boolean true
start method
start/exec astring "/lib/svc/method/rpc-bind %m"
start/group astring root
start/limit_privileges astring :default
start/privileges astring basic,file_chown,file_chown_self,file_owner,net_privaddr,proc_setid,sys_nfs
start/project astring :default
start/resource_pool astring :default
start/supp_groups astring :default
start/timeout_seconds count 60
start/type astring method
start/use_profile boolean false
start/user astring root
start/working_directory astring :default
stop method
stop/exec astring "/lib/svc/method/rpc-bind %m %{restarter/contract}"
stop/group astring root
stop/limit_privileges astring :default
stop/privileges astring basic,proc_owner
stop/project astring :default
stop/resource_pool astring :default
stop/supp_groups astring :default
stop/timeout_seconds count 60
stop/type astring method
stop/use_profile boolean false
stop/user astring root
stop/working_directory astring :default
tm_common_name template
tm_common_name/C ustring "RPC bindings"
tm_man_rpcbind template
tm_man_rpcbind/manpath astring /usr/share/man
tm_man_rpcbind/section astring 1M
tm_man_rpcbind/title astring rpcbind
Once the properties have been reviewed, you can enable tcp_wrappers by setting the ‘config/enable_tcpwrappers’ attribute to ‘true’:
svc:/network/rpc/bind> **setprop config/enable_tcpwrappers=true**
To exit the SMF configuration console, the ‘end’ keyword can be used:
svc:/network/rpc/bind> end
After the SMF service has been configured, the tcp_wrappers hosts.deny and hosts.allow files need to be updated to limit RPC requests to the loopback interface:
$ cat /etc/hosts.deny
ALL: ALL
$ cat /etc/hosts.allow
rpcbind: 127.0.0.1
Once these files are created, you can start using a tcp_wrapped rpcbind by restarting the rpc/bind process with the svcadm utility:
$ svcadm restart rpc/bind
After the rpcbind process has been restarted, each non-local connection to TCP port 111 will be RST:
$ telnet tigger 1111
Trying 192.168.1.3…
telnet: connect to address 192.168.1.3: Connection refused
I digs me some tcp_wrapper!
If you have the default Apache page installed on your web server, then mod_negotiate is going to be helpful and try to serve the content in the flavor and language your browser prefers. This of course comes at a hefty cost:
$ dtrace -ppgrep httpd -s shibby.d
dtrace: script 'shibby.d' matched 4 probes
^C
memcpy 2824
apr_file_read 2496
$ cat shibby.d
pid$target::memcpy:entry
{
@memcpy[probefunc] = count();
}
pid$target::apr_file_read:entry
{
@apr_file_read[probefunc] = count();
}
I came across this while looking for the busiest functions in Apache (this of course is not the case with normal pages).
While reading through some Fujitsu documentation I came across the following:
“Soft errors were formerly caused by alpha particles in the IC housing. This can now be precluded for the most part by an improved IC housing. Radioactive gamma radiation from solar eruptions is now the main cause of soft errors. In all probability, pure soft errors do not lead to uncorrectable multi-bit errors in a memory chip. Soft errors tend to occur by chance and can be eliminated for the most part by ECC. ECC memory protection is now a standard means of ensuring data security in servers. Single errors in more than 2 memory ICs can be neglected, since the likelihood of faults at identical addresses and at the same time is very low.”
This leads me to wonder if data centers can be partially shielded from gamma radiation?
I had a boat load of fun today debugging apxs and apr-config. The issue I was debugging came about when I decided to use mod_deflate to compress text, xml and style sheets on my web server. Since mod_deflate wasn’t built when I had originally configured Apache, I thought I would fire up apxs to build the Apache module:
$ LDFLAGS="-lz" apxs -c mod_deflate.c
/var/tmp/apache/build/libtool --silent --mode=compile gcc -prefer-pic
-DAP_HAVE_DESIGNATED_INITIALIZER -DSOLARIS2=10
-D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT -g -O2
-I/var/tmp/apache/include -I/var/tmp/apache/include
-I/var/tmp/apache/include -c -o mod_deflate.lo mod_deflate.c && touch
mod_deflate.slo
/var/tmp/apache/build/libtool --silent --mode=link gcc -o
mod_deflate.la -rpath /var/tmp/apache/modules -module -avoid-version
mod_deflate.lo
W00t! The module built, so I installed it and tried to fire up Apache:
$ cp .libs/mod_deflate.so /var/tmp/apache/modules/
$ apachectl start
Syntax error on line 268 of /var/tmp/apache/conf/httpd.conf:
Cannot load /var/tmp/apache/modules/mod_deflate.so into server:
ld.so.1: httpd: fatal: relocation error: file
/var/tmp/apache/modules/mod_deflate.so: symbol crc32: referenced symbol
not found
Ack! For some reason mod_deflate.so wasn’t able to find the symbol crc32, which I knew from past experience was part of libz.so. To see what was going on, I pulled out the trusy old ldd utility:
$ ldd /var/tmp/apache/modules/mod_deflate.so
libc.so.1 => /lib/libc.so.1
libgcc_s.so.1 => /usr/sfw/lib/libgcc_s.so.1
libm.so.2 => /lib/libm.so.2
/platform/SUNW,Ultra-5_10/lib/libc_psr.so.1
Hmmm – since libz.so was absent from the output, I quickly realized that apxs was ignoring the LDFLAGS variable. To get a better idea of how apxs figures out which libraries to add, I fired up vi and poked around the apxs Perl script. After two minutes of searching, I came across the following Perl code that controls the link process:
if ($opt_p == 1) {
my $apr_libs=`$apr_bindir/apr-config --ldflags --link-libtool --libs`;
chomp($apr_libs);
my $apu_libs=`$apu_bindir/apu-config --ldflags --link-libtool --libs`;
chomp($apu_libs);
$opt .= " ".apu_libs." ".apr_libs;
} else {
my $apr_ldflags=`$apr_bindir/apr-config --ldflags`;
chomp($apr_ldflags);
$opt .= "-rpath CFG_LIBEXECDIR -module -avoid-version apr_ldflags";`
}
push(@cmds, "$libtool $ltflags --mode=link $CFG_CC -o $dso_file $opt $lo");
Since the libraries are derived from the output of ‘apr-config –ldflags,” I thought I would run it manually to see what was going on:
$ apr-config --ldflags
$ LDFLAGS="-lz" apr-config --ldflags
So given the fact that apr-config doesn’t print “-lz” or pay attention to the LDFLAGS variable (it uses the value of $flags and $LDFLAGS to derive this information, and both are set in the apr-config script), I was forced to add “-lz” to the global LDFLAGS section in the apr-config script:
$ grep ^LDFLAGS apr-config
LDFLAGS=”-lz”
Once this hack was made to apr-config, libtool used the library:
$ make clean && /var/tmp/apache/bin/apxs -c mod_deflate.c
rm -f *.o *.lo *.slo *.obj *.a *.la mod_include.la
rm -rf .libs
/var/tmp/apache/build/libtool --silent --mode=compile gcc -prefer-pic
-DAP_HAVE_DESIGNATED_INITIALIZER -DSOLARIS2=10
-D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT -g -O2
-I/var/tmp/apache/include -I/var/tmp/apache/include
-I/var/tmp/apache/include -c -o mod_deflate.lo mod_deflate.c && touch
mod_deflate.slo
/var/tmp/apache/build/libtool --silent --mode=link gcc -o
mod_deflate.la -rpath /var/tmp/apache/modules -module -avoid-version
-lz mod_deflate.lo
And ldd showed the correct output:
$ ldd /var/tmp/apache/modules/mod_deflate.so
libz.so.1 => /usr/lib/libz.so.1
libc.so.1 => /lib/libc.so.1
libgcc_s.so.1 => /usr/sfw/lib/libgcc_s.so.1
libm.so.2 => /lib/libm.so.2
/platform/SUNW,Ultra-5_10/lib/libc_psr.so.1
While this fix is not ideal, it allowed me to build mod_deflate without incurring the overhead from a complete build. The better solution would have been to re-build Apache with “–enable-deflate=shared,” but that takes a while on my slow box. :)
While poking around the Apache source code this weekend, I started to wonder which Apache modules consumed the most CPU time servicing HTTP requests. Since I had Apache installed on a Solaris 10 box, I threw together a D script named apachemoduleprof to answer this question:
$ apachemoduleprof pgrep httpd
Started Apache Request Profiler (Control-C to stop...)
New connection
Connection start: 2005 Dec 12 18:38:03
Connection stop : 2005 Dec 12 18:38:04
Method URI Count
GET / 1
Module Processing Time
mod_actions.so 24005 ns
mod_expires.so 47337 ns
mod_log_forensic.so 61466 ns
mod_dir.so 74525 ns
mod_info.so 76032 ns
mod_autoindex.so 76927 ns
mod_userdir.so 80814 ns
mod_speling.so 83850 ns
mod_vhost_alias.so 84157 ns
mod_asis.so 94340 ns
mod_cgi.so 99906 ns
mod_usertrack.so 100358 ns
mod_imap.so 101102 ns
mod_cern_meta.so 112784 ns
mod_headers.so 143674 ns
mod_status.so 147744 ns
mod_access.so 169724 ns
mod_dav.so 175717 ns
mod_env.so 177622 ns
mod_unique_id.so 247845 ns
mod_negotiation.so 267640 ns
mod_auth_digest.so 272762 ns
mod_dtrace.so 372322 ns
mod_rewrite.so 437157 ns
mod_include.so 905436 ns
mod_alias.so 1717912 ns
mod_log_config.so 1982180 ns
mod_setenvif.so 2803176 ns
mod_mime.so 3748429 ns
httpd 17620455 ns
Total 32 ms
Hook Name Module Hook Function Time
ap_run_log_transaction mod_log_forensic.so log_after 20739 ns
ap_run_handler mod_actions.so action_handler 24005 ns
ap_run_pre_connection mod_dtrace.so apache_pre_connection 24587 ns
ap_run_access_checker httpd do_nothing 32272 ns
ap_run_header_parser mod_setenvif.so match_headers 40581 ns
ap_run_post_read_request mod_log_forensic.so log_before 40727 ns
ap_run_insert_filter httpd core_insert_filter 43937 ns
ap_run_post_read_request mod_dtrace.so apache_receive_request 45105 ns
ap_run_access_checker mod_dtrace.so apache_check_access 45678 ns
ap_run_insert_filter mod_expires.so expires_insert_filter 47337 ns
ap_run_handler mod_negotiation.so handle_map_file 48780 ns
ap_run_create_connection mod_dtrace.so apache_create_connection 53110 ns
ap_run_insert_filter mod_headers.so ap_headers_insert_output_filter 54983 ns
ap_run_handler mod_dav.so dav_handler 70031 ns
ap_run_handler mod_rewrite.so handler_redirect 71183 ns
ap_run_log_transaction mod_dtrace.so apache_log_request 71983 ns
ap_run_fixups mod_dir.so fixup_dir 74525 ns
ap_run_handler mod_info.so display_info 76032 ns
ap_run_handler mod_autoindex.so handle_autoindex 76927 ns
ap_run_translate_name mod_userdir.so translate_userdir 80814 ns
ap_run_fixups mod_speling.so check_speling 83850 ns
ap_run_translate_name mod_vhost_alias.so mva_translate 84157 ns
ap_run_translate_name mod_rewrite.so hook_uri2file 85999 ns
ap_run_fixups mod_headers.so ap_headers_fixup 88691 ns
ap_run_handler mod_asis.so asis_handler 94340 ns
ap_run_fixups mod_rewrite.so hook_fixup 96737 ns
ap_run_fixups mod_negotiation.so fix_encoding 98537 ns
ap_run_handler mod_cgi.so cgi_handler 99906 ns
ap_run_fixups mod_usertrack.so spot_cookie 100358 ns
ap_run_fixups mod_auth_digest.so add_auth_info 100773 ns
ap_run_handler mod_imap.so imap_handler 101102 ns
ap_run_fixups httpd core_override_type 105600 ns
ap_run_fixups mod_dav.so dav_fixups 105686 ns
ap_run_map_to_storage httpd ap_send_http_trace 108333 ns
ap_run_fixups mod_cern_meta.so add_cern_meta_data 112784 ns
ap_run_type_checker mod_negotiation.so handle_multi 120323 ns
ap_run_create_request mod_dtrace.so apache_create_request 131859 ns
ap_run_handler mod_status.so status_handler 147744 ns
ap_run_pre_connection httpd core_pre_connection 160052 ns
ap_run_access_checker mod_access.so check_dir_access 169724 ns
ap_run_fixups mod_alias.so fixup_redir 170886 ns
ap_run_post_read_request mod_auth_digest.so parse_hdr_and_update_nc 171989 ns
ap_run_fixups mod_env.so fixup_env_module 177622 ns
ap_run_type_checker mod_rewrite.so hook_mimetype 183238 ns
ap_run_post_read_request mod_unique_id.so gen_unique_id 247845 ns
ap_run_create_request httpd http_create_request 389379 ns
ap_run_translate_name httpd ap_core_translate 444739 ns
ap_run_create_request httpd core_create_req 624600 ns
ap_run_fixups mod_include.so include_fixup 905436 ns
ap_run_create_connection httpd core_create_conn 1042453 ns
ap_run_translate_name mod_alias.so translate_alias_redir 1547026 ns
ap_run_log_transaction mod_log_config.so multi_log_transaction 1982180 ns
ap_run_post_read_request mod_setenvif.so match_headers 2762595 ns
ap_run_type_checker mod_mime.so find_ct 3748429 ns
ap_run_handler httpd default_handler 6766633 ns
ap_run_map_to_storage httpd core_map_to_storage 7902457 ns
After running this a few times, I noticed that the httpd core along with a few other modules consumed most of the CPU time. To see which functions were the busiest in each module, I threw together the apachemodulefunctime D script:
$ apachemodulefunctime pgrep httpd mod_log_config.so
Tracing Apache Module mod_log_config.so (Control-C to stop)
Module Function Time
mod_log_config.so log_remote_user 21732
mod_log_config.so log_request_line 22737
mod_log_config.so clf_log_bytes_sent 24603
mod_log_config.so log_remote_logname 51210
mod_log_config.so log_remote_host 52674
mod_log_config.so constant_item 118539
mod_log_config.so log_status 140340
mod_log_config.so ap_default_log_writer 444106
mod_log_config.so config_log_transaction 523044
mod_log_config.so multi_log_transaction 597611
mod_log_config.so log_request_time 896752
Total Time: 2893348
This example shows each function that is called in the mod_log_config Apache module, and provides a timestamp from the function entry to the function exit. This was super useful information, but I also wanted to see which functions were called from mod_log_config. This was easy enough with the following DTrace script:
$ cat whocall.d
dtrace:::BEGIN
{
once = 1;
}
pid$target::log_request_time:entry
{
self->follow = 1;
}
pid$target::log_request_time:return
{
self->follow = 0;
once = 0;
}
pid$target:::entry
/self->follow && once/
{
self->ts = timestamp;
}
pid$target:::return
/self->follow && once/
{
@functions[probefunc] = count();
@total[probefunc] = sum(timestamp - self->ts);
}
dtrace:::END
{
printf("%-30s %-20sn","Function", "Times Called");
printa("%-30s %-@20dn",@functions);
printf("n%-30s %-20sn","Function", "Total Execution Time");
printa("%-30s %-@20dn",@total);
}
$ dtrace -p pgrep httpd -q -s whocall.d
Function Times Called
.rem 1
ap_explode_recent_localtime 1
apr_palloc 1
apr_snprintf 1
apr_time_exp_lt 1
apr_vformatter 1
cached_explode 1
explode_time 1
findenv 1
getenv 1
getsystemTZ 1
lmutex_lock 1
lmutex_unlock 1
localtime_r 1
ltzset_u 1
offtime_u 1
set_zone_context 1
strlen 1
.div 2
__moddi3 2
strcmp 2
__divdi3 3
conv_10 7
.udiv 14
Function Total Execution Time
.rem 28896
strlen 30713
lmutex_lock 32096
lmutex_unlock 32096
offtime_u 34203
findenv 35492
ap_explode_recent_localtime 36634
set_zone_context 40452
.div 52620
strcmp 59259
__moddi3 61988
ltzset_u 84287
getsystemTZ 86064
cached_explode 88502
getenv 90303
localtime_r 91428
__divdi3 98886
apr_vformatter 140929
explode_time 147786
apr_snprintf 196226
apr_time_exp_lt 204179
.udiv 399397
conv_10 582217
apr_palloc 2114263
AS we can see from this output, the apr_palloc routine takes the longest time to execute. This is as expected, since apr_palloc() needs to call the Apache allocator to find free memory and allocate memory if allocation slots aren’t availablet. Now to see if there are any efficiency gains to be made in the allocator! :)