Blog O' Matty


DTrace out of memory errors

This article was posted by Matty on 2006-03-04 11:10:00 -0400 -0400

While profiling an application with DTrace this week, I received the following error:

$ dtrace -p 19019 -s follow.d

dtrace: failed to compile script follow.d: line 21: failed to create
probe in process 19019: Not enough space

After a bit of googling, it looks like I was hitting the upper bounds on the number of probes that can be enabled. To fix the problem, I increased fasttrap-max-probes in /kernel/drv/fasttrap.conf from 250,000 to 1,000,000. Once I made the change, I ran update_drv to force the fastrap kernel module to re-read it’s configuration:

$ update_drv fasttrap

Now everything is swell!

Smartmontools saves the day!

This article was posted by Matty on 2006-03-04 10:49:00 -0400 -0400

While booting up my x86 laptop this week, I noticed the following errors on the console:

Feb 26 18:16:54 zebox smartd[492]: Device: /dev/ad0, 1 Currently unreadable (pending) sectors
Feb 26 18:16:54 zebox smartd[492]: Device: /dev/ad0, 1 Offline uncorrectable sectors
Feb 26 18:46:55 zebox smartd[492]: Device: /dev/ad0, 1 Currently unreadable (pending) sectors
Feb 26 18:46:55 zebox smartd[492]: Device: /dev/ad0, 1 Offline uncorrectable sectors

Eeeeep – it looks like the disk drive is going bad. To verify this, I decided to run smartctl against the device:

$ smartctl -a /dev/ad0

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 100 253 006 Pre-fail Always - 0
3 Spin_Up_Time 0x0003 095 095 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 92
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 068 062 030 Pre-fail Always - 6508058
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 329
10 Spin_Retry_Count 0x0013 100 100 034 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 96
187 Unknown_Attribute 0x0032 094 094 000 Old_age Always - 6
189 Unknown_Attribute 0x003a 100 100 000 Old_age Always - 0
190 Unknown_Attribute 0x0022 065 055 045 Old_age Always - 622067747
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 56
193 Load_Cycle_Count 0x0032 090 090 000 Old_age Always - 21143
194 Temperature_Celsius 0x0022 035 045 000 Old_age Always - 35 (Lifetime Min/Max 0/15)
195 Hardware_ECC_Recovered 0x001a 078 054 000 Old_age Always - 177096143
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 1
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 1
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0
202 TA_Increase_Count 0x0032 100 253 000 Old_age Always - 0

Hmmm – the value of Seek_Error_Rate looks extremely high, so I decided to run smartctl a second time to see if the value of Seek_Error_Rate was climbing:

$ smartctl -a /dev/ad0

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 100 253 006 Pre-fail Always - 0
3 Spin_Up_Time 0x0003 095 095 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 92
5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 068 062 030 Pre-fail Always - 6508123
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 329
10 Spin_Retry_Count 0x0013 100 100 034 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 96
187 Unknown_Attribute 0x0032 094 094 000 Old_age Always - 6
189 Unknown_Attribute 0x003a 100 100 000 Old_age Always - 0
190 Unknown_Attribute 0x0022 065 055 045 Old_age Always - 622067747
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 56
193 Load_Cycle_Count 0x0032 090 090 000 Old_age Always - 21146
194 Temperature_Celsius 0x0022 035 045 000 Old_age Always - 35 (Lifetime Min/Max 0/15)
195 Hardware_ECC_Recovered 0x001a 078 054 000 Old_age Always - 177096157
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 1
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 1
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0000 100 253 000 Old_age Offline - 0
202 TA_Increase_Count 0x0032 100 253 000 Old_age Always - 0

Sure enough, the value was increasing at a staggering rate! Since I had just purchased the drive from NewEgg, I gave them a call, and they are going to send me a replacement. Viva la smartmontools!

Amusing UNIX commands

This article was posted by Matty on 2006-03-04 10:40:00 -0400 -0400

While surfing the net this week, I came across a website with a list of several amusing UNIX command lines. I think my personal favorite is “man: why did you get a divorce?":

$ man: why did you get a divorce?
man:: Too many arguments.

It’s all about a little humor! :)

Removing messages from POP3S servers

This article was posted by Matty on 2006-02-27 20:34:00 -0400 -0400

While checking my e-mail recently, I noticed that my inbox contained several new messages that were roughly 3MB in size. The quantity of messages was choking my poor text-based e-mail client, so I needed to figure out what has happening. After spending a few minutes reviewing my maillog, I noticed that my mail retrieval program was erroring out while popping the 3MB message (it transferred the message, but wasn’t issuing a dele when it finished). Since my mail retrieval program was having issues retrieivng the 3MB file, I decided to fire up openssl and delete the message myself:

$ openssl s_client -quiet -connect mail.prefetch.net:995

depth=0 /C=US/O=mail.prefetch.net/OU=https://services.choicepoint.net/get.jsp?1605445126/OU=See www.rapidssl.com/cps (c)04/OU=Domain Control Validated - StarterSSL(TM)/CN=mail.prefetch.net
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /C=US/O=mail.prefetch.net/OU=https://services.choicepoint.net/get.jsp?1605445126/OU=See www.rapidssl.com/cps (c)04/OU=Domain Control Validated - StarterSSL(TM)/CN=mail.prefetch.net
verify error:num=27:certificate not trusted
verify return:1
depth=0 /C=US/O=mail.prefetch.net/OU=https://services.choicepoint.net/get.jsp?1605445126/OU=See www.rapidssl.com/cps (c)04/OU=Domain Control Validated - StarterSSL(TM)/CN=mail.prefetch.net
verify error:num=21:unable to verify the first certificate
verify return:1

+OK mail.prefetch.net Cyrus POP3 v2.2.12 server ready <1499164100.1140385444@mail.prefetch.net>

user myaccount
+OK Name is a valid mailbox

pass MYSPECIALPASSWORD
+OK Mailbox locked and ready

list
+OK scan listing follows
1 1893014
2 599397
3 4219
.

dele 1
+OK message deleted

quit
+OK
read:errno=0

I really dig openssl, and find myself using it daily. Giddie up!

Alice in chains reunion!

This article was posted by Matty on 2006-02-27 20:19:00 -0400 -0400

While reading through pollstar.com, I read that Jerry Cantrell is reforming Alice In Chains with a new lead singer. This is one of the happiest moments in my life, and I am going to brave my fear of planes to wander to several AIC shows!! Rock on!