While developing a Perl script this weekend to summarize Solaris zone usage, I wanted to display some type of status while my script did it’s thing. My friend Clay came up with a cool way to do this, and I thought I would share it here in case others needed to do something similar. Here is the code:
#!/usr/bin/perl -w
# Initialize the index.
my $index=0;
# The characters to use as spinners.
my @spinners = ("-", "\", "|", "/");
my $totalspinners = $#spinners + 1;
# Turn on autoflush.
$ `| = 1;`
print "Factoring primes: ";
while (1) {
print "$spinners[$index++ % $totalspinners]";
sleep 1;
}
# If you were to exit the loop, turn off autoflush.
$ `| = 0;`
There may be a better way to do this (adding an additional module isn’t an option), but I have yet to find it. Niiiiiice!
I just posted my article Debugging Apache web server problems from the July ‘07 issue of SysAdmin magazine to my website. If the need arises for you to debug problems with a server running Apache, hopefully the items described in the article will prove helpful.
In my previous post, I discussed how jmap can be used to view the contents of the Java heap. Jmap is a wonderful utility for viewing heap utilization at a specific point in time, but it’s not the best utiltiy for answer questions like “which call stack is allocating objects of type foo?", or “how many objects are being allocated per second?", or “what is the average size of allocated objects?". To answer these types of questions, we need to look to a more thorough instrumentation framework.
One such framework is the DTrace hotspot provider which was introduced in Java 6, and provides a number of probes to dynamically instrument Java applications. These probes allow you to observe thread behavior, garbage collection, method calls, class loader activity, monitor contention and object allocation. The probe to observe object allocation is rightfully named “object-alloc,” and contains several arguments (since the probe arguments are documented extremely well in the Java community, I won’t describe them here) that can be used to understand how the heap is being used.
To use the object-alloc probe, you will need to start the JVM with the “-XX:+DTraceAllocProbes” option, or run the jinfo utility to enable it at runtime (enabling it at runtime is preferred):
$ jinfo -XX:+DTraceAllocProbespgrep java``
Once the object-alloc probe is enabled, you can begin prodding your application to see what it’s doing. If you are seeing lots of garbage collection events in jconsole, jstat or your garbage collection logs, the object allocation probe can be used to display object allocations by object type, the number of objects being allocated, and the size of these allocations. The newobjects.d DTrace script provides the necessary probe definitions to display this information:
$ newobjects.d
Class Objects created Bytes Allocated
[I 112 16064
Wasteful 17943 287088
java/lang/StringBuilder 17943 287088
java/lang/String 17943 430632
[C 107658 52393560
Class Objects created Bytes Allocated
[I 121 18520
Wasteful 19541 312656
java/lang/StringBuilder 19541 312656
java/lang/String 19542 469008
[C 117252 57062640
If you see tons of object allocations, you can use the DTrace quantize() action to create a histogram with object counts or object sizes. The objectsize.d DTrace script shows how to create a distribution that is keyed on object size:
$ objectsize.d
value ------------- Distribution ------------- count
8 | 0
16 |@@@@@@@@@@@@@ 95897
32 |@@@@ 31965
64 | 1
128 |@@@@ 32157
256 |@@@@ 31965
512 |@@@@@@@@@ 63932
1024 |@@@@ 31967
2048 | 0
Once we know how many objects are being allocated, it would useful to see which Java call stacks are responsible for allocating these objects. To generate a list of Java calls stacks with the number of objects allocated per call stack, we can run the whoallocatecount.d DTrace script:
$ whoallocatecount.d
< ..... >
libjvm.so`__1cNSharedRuntimeTdtrace_object_alloc6FpnHoopDesc__i_+0x4f
Wasteful.main([Ljava/lang/String;)V
StubRoutines (1)
libjvm.so`_pnGThread__v_+0x1a3
libjvm.so`jni_CallStaticVoidMethod+0x15d
java`JavaMain+0xd30
libc.so.1`_thr_setup+0x52
libc.so.1`_lwp_start
5678
In the output above (which was cleaned up to make it easy to view), we can see that 5678 objects where allocated by the main() method in the Wasteful class. If you would prefer to see call stacks by bytes allocated, you can run the whoallocatebybytes.d script:
$ whoallocatebytes.d
< ..... >
libjvm.so`__1cNSharedRuntimeTdtrace_object_alloc6FpnHoopDesc__i_+0x4f
Wasteful.main([Ljava/lang/String;)V
StubRoutines (1)
libjvm.so`_pnGThread__v_+0x1a3
libjvm.so`jni_CallStaticVoidMethod+0x15d
java`JavaMain+0xd30
libc.so.1`_thr_setup+0x52
libc.so.1`_lwp_start
817632
The hotspot provider is an incredible debugging tool, and it’s pretty amazing what you can do with just one probe!! If you decide to use these probes in production, be somewhat cautious. Selectively enable probes, and do so only for the duration of your troubleshooting session. Be safe, be careful, and most of all, have fun! Niiiiiiice!
A while back I wrote an article titled dynamically growing a Clariion LUN with Solaris. In the article I described how to update the VTOC on a UN that was resized on the storage array. One of my colleagues came to me a few weeks back and told me the procedure was not working, and he couldn’t assign a new size (200GB in this case) to the device in format.
This made me curious, so I started poking around. To see what Solaris thought the actual size of the LUN was, I ran luxadm with the “display” option:
$ luxadm display /dev/rdsk/c5t600601602B9318008CD071B0FF7BDB11d0s2
DEVICE PROPERTIES for disk: /dev/rdsk/c5t600601602B9318008CD071B0FF7BDB11d0s2
Vendor: DGC
Product ID: RAID 5
Revision: 0322
Serial Num: APM000612076
Unformatted capacity: 204800.000 MBytes
Read Cache: Enabled
Minimum prefetch: 0x0
Maximum prefetch: 0x0
Device Type: Disk device
Path(s):
/dev/rdsk/c5t600601602B9318008CD071B0FF7BDB11d0s2
/devices/scsi_vhci/disk@g600601602b9318008cd071b0ff7bdb11:c,raw
Controller /dev/cfg/c3
Device Address 5006016039a00e91,1
Host controller port WWN 10000000c9563354
Class secondary
State STANDBY
Controller /dev/cfg/c3
Device Address 5006016939a00e91,1
Host controller port WWN 10000000c9563354
Class primary
State ONLINE
Controller /dev/cfg/c4
Device Address 5006016139a00e91,1
Host controller port WWN 10000000c9563355
Class secondary
State STANDBY
Controller /dev/cfg/c4
Device Address 5006016839a00e91,1
Host controller port WWN 10000000c9563355
Class primary
State ONLINE
On the host we were having issues with, Solaris was able to see all 200GB, but for some reason the format “Auto Configure” option still thought the LUN was 50GB. When I chose the format type “DEFAULT,” the LUN showed up with a capacity of 200GB:
$ format -e
format> type
AVAILABLE DRIVE TYPES:
0. Auto configure
1. DEFAULT
2. DEFAULT
3. DEFAULT
4. other
Specify disk type (enter its number)[2]: 1
selecting c5t600601602B9318008CD071B0FF7BDB11d0
[disk formatted]
partition> p
Current partition table (original):
Total disk cylinders available: 6524 + 2 (reserved cylinders)
Part Tag Flag Cylinders Size Blocks
0 unassigned wm 1 - 26104 199.97GB (26104/0/0) 419360760
1 unassigned wm 0 0 (0/0/0) 0
2 backup wu 0 - 26104 199.97GB (26105/0/0) 419376825
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0
8 boot wu 0 - 0 7.84MB (1/0/0) 16065
9 unassigned wm 0 0 (0/0/0) 0
After chatting with some qualified Sun folks, this madness started to make sense. When you select “Auto Configure,” format will choose an entry from the format.dat file instead of determing the drive’s capacity from the SCSI READ CAPACITY command. The DEFAULT option causes this command to be sent, and therefore the new capacity is available. For future reference, running format with the “-e” option and choosing “DEFAULT” seems to work.
On most Linux hosts, the first field of the /etc/fstab file contains labels instead of disk partions. This simplifies file system management, since you don’t have to update the fstab file if you move a drive to a new controller, or add additional drives to a system. If you want to locate the partition that is associated with a label, you can use the findfs utility:
$ /sbin/findfs LABEL=/
/dev/hda1
You can also use the findfs utility to locate a partition by UUID
$ /sbin/findfs UUID=b4ce6d24-000c-45a3-8258-cbf9f826c0ce
/dev/hda1
The findfs utility is extremely useful, and is just one of a number of cool programs (others include blkid, e2label, partinfo and findsuper) in the e2fsprogs package! Giddie up!