Determining the capabilities of a NIC on a Solaris host

There are a myriad of NIC chipsets in use by the major server vendors (Broadcom, Intel, NVidia, etc.), and each chipset typically contains a unique set of capabilities (e.g., hardware offload support, some amount of on board cache devoted to RX / TX rings, hardware flow classification, etc.). To see which capabilities a given NIC chipset supports, you can usually read through the technical white papers and engineering documents that were published when the chipset shipped. To find the NIC chipsets that are in use on a Solaris host, the kstat utility can be run with the the name of a network driver, the instance of the driver, and the “chipid” name:

$ kstat -m bge -i 0 -n chipid | head -10

module: bge                             instance: 0     
name:   chipid                          class:    net
        asic_rev                        2416115712
        bus_size                        64 bit
        bus_speed                       fast
        bus_type                        PCI-X
        businfo                         4746
        cache_line_size                 16
        chip_type                       5715
        command                         342

This will display a number of pieces of information, including the type of BUS in use, whether it is 32- or 64-bit, and the chipset version. In the output above, we can see that a 64-bit PCI express Broadcom model 5715 adapter is in use by the server. Kstat rocks!

OpenSolaris storage wishlist

A number of opensolaris communities have asked their members for feedback, and the list of technologies they would like to see added in the future. The storage community received a ton of feedback when they asked the community for the list of features they would like to have added to opensolaris, and this feedback was recently posted to the genunix wiki. If there are features you are interested in that don’t appear on the list, I would highly recommend adding them. There are so many cool things underway (Comstar, CIFS client and server, NPIV support, pNFS, better remote replication, etc.) in the storage community, and it’s awesome to see the storage project leaders reaching out to the community to get their ideas!

Substituting text in the HTTP request body with mod_substitute

While doing a bit of research tonight I came across a reference to mod_substitute. This nifty module allows you to substitute text in the HTTP request body, which provides an easy way to do things similar to the following:

<Location /private>
    AddOutputFilterByType SUBSTITUTE text/html
    Substitute s/SECRET/XXXXX/ni
</Location>

I digs me some Apache!

Resources from my Java performance presentation

I gave a presentation last night on debugging Java performance problems. I got a couple of requests to post links to the performance analysis tools I discussed, so here you go:

DTraceToolkit

Garbage collection log visualization utility

Garbage collection visualization utility

Java hotspot DTrace provider

Java heap profiling agent

Monitoring garbage collection with jstat

Observing object allocation with DTrace

Trending Java performance

I would like to thank everyone for attending, and hope to see ya’ll at a future meeting!

Figuring out which package a Solaris utility belongs to

While reading through some old notes this weekend, I came across a page I created eons ago about managing Solaris packages. If you want to find out the file modes, the user and group ownership and the package a file belongs to, you can run the pkgchk utility with the “-l” and “-p” options and the name of a file to check:

$ pkgchk -l -p /usr/sfw/bin/snmpget

Pathname: /usr/sfw/bin/snmpget
Type: regular file
Expected mode: 0755
Expected owner: root
Expected group: bin
Expected file size (bytes): 20372
Expected sum(1) of contents: 48443
Expected last modification: Sep 20 17:41:36 2007
Referenced by the following packages:
        SUNWsmcmd      
Current status: installed

Pkgchk is a nifty utility!

Profiling Java methods with the heap profiling agent

The Java SDK comes with a number of tools and JVM options that can be used to analyze the performance of the Java runtime. One extremely useful tool is the heap profiler agent, which provides facilities to profile memory usage, CPU utilization and lock contention. To load the profiler agent to profile CPU utilization, you can add the “-agentlib:hprof=cpu=times” option to your java command line:

$ java -Xms256m -Xmx256m -verbose:gc -agentlib:hprof=cpu=times App

Once loaded, the agent will use byte code injection (BCI) to instrument each method’s entry and return points. This allows the agent to measure the number of times each method was called, the time spent in each method, and the call chain that led to the method being invoked. To utilize the agent to it’s full potential, you will need to exercise the application while the agent is active. Once the runtime has been exercised, you can hit cntrl+C to stop the process. This will cause the agent to write the data it has collected to the file java.hprof.txt, which can be viewed with your favorite pager or editor:

$ more java.hprof.txt

CPU TIME (ms) BEGIN (total = 40712194) Mon Jan 21 19:23:12 2008
rank   self  accum   count trace method   
   1 38.52% 38.52% 1036273 301143 java.util.Random.next   
   2 19.57% 58.09%  518136 301144 java.util.Random.nextDouble   
   3 11.87% 69.96%  518136 301145 java.lang.Math.random
   4 11.05% 81.01% 1036274 301141 java.util.concurrent.atomic.AtomicLong.get
   5 10.53% 91.54% 1036273 301142 java.util.concurrent.atomic.AtomicLong.compareAndSet
   6  8.14% 99.68%  259068 301146 TestMain.foo
   7  0.05% 99.73%       1 300969 java.security.Permissions.add
   8  0.04% 99.77%       1 301106 java.lang.Class.privateGetDeclaredFields
   9  0.03% 99.79%       1 301138 java.util.Random.
  10  0.02% 99.81%       2 300908 java.io.FilePermission$1.run
  11  0.01% 99.82%       1 301283 java.lang.ThreadGroup.remove
  12  0.01% 99.83%       1 300820 sun.net.www.protocol.file.Handler.createFileURLConnection
......



The java.hprof.txt file contains the number of times each method was invoked, as well as the amount of CPU time (as a percentage) that was spent in each method. To see how a method was called, you can search the profiler output for the trace identifier that is listed along side the profiling data. This will produce a Java stack trace similar to the following that shows how a given method was invoked:

TRACE 301143:
        java.util.Random.next(Random.java:Unknown line)
        java.util.Random.nextDouble(Random.java:Unknown line)
        java.lang.Math.random(Math.java:Unknown line)
        TestMain.foo(TestMain.java:Unknown line)



The BCI approach introduces a fair amount of overhead to the Java runtime. In cases were the overhead is hindering testing, you can use the agent’s “cpu=samples” option instead. This will cause the agent to sample the runtime environment at periodic intervals to see which methods are executing. While this approach is not as accurate as the BCI approach, it provides a good set of results with less runtime overhead. The java profiling agent is incredibly useful, and just one of the vast number of tools that are available to profile Java applications.