Testing NTP services

Periodically I need to update my systems with new NTP server information. Since I have no idea if the remote servers are well kept, I tend to run ntpdate in test mode prior to starting ntpd or adding ntpdate to root’s crontab. To run ntpdate in test mode, you can use the “-d” option:

$ /usr/sbin/ntpdate -d 192.168.1.1

12 Feb 17:36:36 ntpdate[24068]: ntpdate 4.1.2@1.892 Tue Feb 24 06:32:26 EST 2004 (1)
transmit(192.168.1.1)
receive(192.168.1.1)
transmit(192.168.1.1)
receive(192.168.1.1)
transmit(192.168.1.1)
receive(192.168.1.1)
transmit(192.168.1.1)
receive(192.168.1.1)
transmit(192.168.1.1)
server 192.168.1.1, port 123
stratum 3, precision -18, leap 00, trust 000
refid [192.168.1.1], delay 0.02592, dispersion 0.00000
transmitted 4, in filter 4
reference time:    c97b6a71.b4251202  Mon, Feb 12 2007 17:36:33.703
originate timestamp: c97b6ab2.f2ec3995  Mon, Feb 12 2007 17:37:38.948
transmit timestamp:  c97b6a74.f210f0e9  Mon, Feb 12 2007 17:36:36.945
filter delay:  0.02609  0.02592  0.02594  0.02596 
         0.00000  0.00000  0.00000  0.00000 
filter offset: 62.00322 62.00318 62.00317 62.00316
         0.000000 0.000000 0.000000 0.000000
delay 0.02592, dispersion 0.00000
offset 62.003182

In addition to printing the timestamps, it also provides the offset the clock will be adjusted by. Certain applications dislike the time moving forward or back, which makes the ntpdate test option even more useful.

Changing the server name Sun One Web Server reports

I am a big fan of the Sun One Web Server, although I dislike the fact that it provides the server software and version by default in the HTTP header:

$ telnet localhost 80

Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
HEAD / HTTP/1.0

HTTP/1.1 200 OK
Server: Sun-ONE-Web-Server/6.1
Date: Fri, 23 Feb 2007 22:41:21 GMT
Content-length: 179
Content-type: text/html
Last-modified: Tue, 20 Feb 2007 14:30:21 GMT
Accept-ranges: bytes
Connection: close

Connection closed by foreign host.

This gives out more information that I care to share, and provides remote attackers with an extra piece of information to determine the software stack that is in use. Luckily the value reported in the “Server” attribute can be changed by adding the “ServerString” directive to the magnus.conf. Here is a sample magnus.conf entry that sets the “Server” attribute to the string “Apache”:

ServerString Apache

Once this directive is set, the web server will return the string “Apache” instead of the string “Sun-ONE-Web-Server/6.1”:

$ telnet localhost 80

Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
HEAD / HTTP/1.0

HTTP/1.1 200 OK
Server: Apache
Date: Fri, 23 Feb 2007 22:43:58 GMT
Content-length: 179
Content-type: text/html
Last-modified: Tue, 20 Feb 2007 14:30:21 GMT
Accept-ranges: bytes
Connection: close

Connection closed by foreign host.

Tis all about not disclosing information if you don’t have to!

Configuring the V40z SP to email home when problems occur

Having now worked with the Sun V40Z for more than a year, I can safely say that it is one of the best server platforms I have ever used. It has incredible lights out management, does a killer job of monitoring the platform environmentals, and can be configured to alert staff to problems it detects. All of these featured are made available through the service processor, which is an out-of-band device dedicated to monitoring and management. Since the service processor is constantly polling the platform environmentals, it knows immediately when a problem arises, and can be configured to send email or an SNMP trap with a detailed explanation of the issue that is detected.

To configure email notifications, you will first need to configure one or more DNS servers so the service processor can resolve the SMTP servers (you can also use IP addresses, but that is a maintenance headache). To configure two DNS servers, the service process “sp” command can be run with the “enable” option, the “dns” keyword and one or more DNS servers:

$ sp enable dns -n 192.168.1.1 -n 192.168.1.2

To view the configured DNS servers, the sp command can be run with the “get dns” option:

$ sp get dns
Name Server(s) Search Domain(s)
192.168.140.7,192.168.140.6

After DNS is configured and verified, the sp utility can be used to set an SMTP server. The following example sets the “From:” line that will be used in all outbound emails, and configures an SMTP server to route mail through:

$ sp set smtp server -f loopy@prefetch.net smtp.prefetch.net

To verify the SMTP settings, the sp utility can be run with the “get smtp server” option:

$ sp get smtp server
Server From Address
smtp.prefetch.net loopy@prefetch.net

Once the SMTP server(s) are configured, you will need to tell the service processor to generate email when an events occurs, and the address to send those events to. To generate email when informational, warning and critical events occur, the sp utility can be run with the “update smtp” option, the event notification level, and an address to send the alert to:

$ sp update smtp subscriber -n SMTP_Crit_Long -r zematty@prefetch.net

$ sp update smtp subscriber -n SMTP_Info_Long -r zematty@prefetch.net

$ sp update smtp subscriber -n SMTP_Warn_Long -r zematty@prefetch.net

Now each time an event occurs, the service processor will send a message with details on the event that occurred. If you want to generate a test event to make sure the SP email event notification facility is configured correctly, the sp utility can be run with the “create test events” options:

$ sp create test events

Tis all about getting notified when ze hardware fails.

Replacing failed disk drives in a ZFS pool

I had a disk drive fail in one of my ZFS pools over the weekend, and needed to swap it out to restore the pool to an optimal state. To begin the swap out, I used the zpool utility to see which disk drive was faulty:

$ zpool status -v

  pool: rz2pool
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://www.sun.com/msg/ZFS-8000-D3
 scrub: resilver completed with 0 errors on Tue Feb 13 14:12:37 2007
config:

        NAME          STATE     READ WRITE CKSUM
        rz2pool       DEGRADED     0     0     0
          raidz2      DEGRADED     0     0     0
            c1t9d0    ONLINE       0     0     0
            c1t10d0   ONLINE       0     0     0
            c1t12d0   ONLINE       0     0     0
            c2t1d0    ONLINE       0     0     0
            spare     DEGRADED     0     0     0
              c2t2d0  UNAVAIL      0     0     0  cannot open
              c2t3d0  ONLINE       0     0     0
        spares
          c2t3d0      INUSE     currently in use

Once I located the faulty device, I used cfgadm to add and remove the old and new disk drives from the system, and then ran zpool with the “replace” option to replace the failed drive in my pool:

$ zpool replace rz2pool c2t2d0 c2t2d0

After the replacement operation completed, I used zpool to monitor the resilvering of the replacement drive:

$ zpool status -v

  pool: rz2pool
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scrub: resilver in progress, 0.10% done, 0h31m to go

config:

        NAME                STATE     READ WRITE CKSUM
        rz2pool             DEGRADED     0     0     0
          raidz2            DEGRADED     0     0     0
            c1t9d0          ONLINE       0     0     0
            c1t10d0         ONLINE       0     0     0
            c1t12d0         ONLINE       0     0     0
            c2t1d0          ONLINE       0     0     0
            spare           DEGRADED     0     0     0
              replacing     DEGRADED     0     0     0
                c2t2d0s0/o  UNAVAIL      0     0     0  cannot open
                c2t2d0      ONLINE       0     0     0
              c2t3d0        ONLINE       0     0     0
        spares
          c2t3d0            INUSE     currently in use

errors: No known data errors

All of this was done online, and with minimal interruption to the applications running on the host.

The power of the SMF Apache2 service

I am in the process of migrating some old Solaris 8 web servers to Solaris 10, and plan to use SMF to stop and start Apache. Solaris ships with a relatively recent release of Apache (if only it included the LDAP authentication module and PHP), which is nicely integrated with SMF. If your a *NIX admin, you gotta love the fact that SMF will restart processes for you:

$ svcadm enable apache2

$ ps -ef | grep http

webservd 16663 16660   0 09:57:59 ?           0:00 /usr/apache2/bin/httpd -k start
webservd 16662 16660   0 09:57:59 ?           0:00 /usr/apache2/bin/httpd -k start
webservd 16661 16660   0 09:57:59 ?           0:00 /usr/apache2/bin/httpd -k start
webservd 16664 16660   0 09:58:00 ?           0:00 /usr/apache2/bin/httpd -k start
webservd 16665 16660   0 09:58:00 ?           0:00 /usr/apache2/bin/httpd -k start
    root 16660     1   2 09:57:58 ?           0:00 /usr/apache2/bin/httpd -k start

$ pkill httpd

$ ps -ef | grep http

webservd 16689 16686   0 09:58:08 ?           0:00 /usr/apache2/bin/httpd -k start
webservd 16690 16686   0 09:58:08 ?           0:00 /usr/apache2/bin/httpd -k start
webservd 16691 16686   0 09:58:08 ?           0:00 /usr/apache2/bin/httpd -k start
    root 16686     1   2 09:58:07 ?           0:00 /usr/apache2/bin/httpd -k start
webservd 16688 16686   0 09:58:08 ?           0:00 /usr/apache2/bin/httpd -k start
webservd 16687 16686   0 09:58:08 ?           0:00 /usr/apache2/bin/httpd -k start

Creating more /dev/md[0-9]+ entries

While upgrading my desktop this weekend to Fedora Core 6, I received the following error while attempting to start one of my md arrays:

$ /sbin/mdadm -A /dev/md3 /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh
mdadm: error opening /dev/md3: No such file or directory

To fix the issue, I had to cd into /dev and add some additional md entries with the MAKEDEV executable:

$ cd /dev && ./MAKEDEV md

Once I ran MAKDEV, mdadm was able to start up the array:

$ /sbin/mdadm -A /dev/md3 /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh
mdadm: /dev/md3 has been started with 6 drives.

*** UPDATE ***

Instead of going through the hassle of running MAKDEV, it looks like you can also use the mdadm “-a” option:

-a, –auto{=no,yes,md,mdp,part,p}{NN}
Instruct mdadm to create the device file if needed, possibly allocating an unused minor
number. “md” causes a non-partitionable array to be used. “mdp”, “part” or “p” causes
a partitionable array (2.6 and later) to be used. “yes” requires the named md device
to have a ’standard’ format, and the type and minor number will be determined from
this. See DEVICE NAMES below.