Jonathan Schwartz’s update on the recent NetApp patent litigation

I just read Jonathan Schwartz’s update on the Netapp patent litigation. I used to be a huge fan of Network Appliance, and recommended them whenever I got the chance. When I read Dave Hitz’s blog post about NetApp suing Sun for their ZFS implementation, my jaw dropped. I couldn’t believe NetApp would file litigation against an opensource technology, and the high regard I help for NetApp flew out the window. Not only would I never recommend NetApp again, but I would love nothing more than to replace our NetApp filers with thumpers (and I am actively working to persuade my boss to do so). Hopefully NetApp will come to the realization that litigation doesn’t benefit anyone, and they will drop their suit against Sun. Hopefully one day we will get to a litigation free world!

Viewing SCSI mode page data

I came across the sdparm utility while surfing the web last weekend. This super useful utility can be used to display and modify SCSI device parameters, and is the best tool I’ve found for dumping SCSI mode and VPD pages. I wish sdparm would have been around when I was reading through the SBC documentation on the T11 website. That would have been swell!

Using the Solaris process tools on a specific thread

In Solaris 10 the ptools (process tools) were enhanced to display data for specific threads in a process. This functionality can be tapped into by appending a slash and a thread id to the ptool command line:

$ pstack 739/2

739:    java TestApp
-----------------  lwp# 2 / thread# 2  --------------------
 feeb3025 lwp_cond_wait (806eeb8, 806eea0, 0, 0)
 feac58c1 __1cHMonitorEwait6Mbl_b_ (806ee48, 0, 0) + 2c1
 feb28310 __1cHThreadsKdestroy_vm6F_b_ (0, 0, 806a79c, 0, 10006, febb59c9) + 7c
 fe8e2de2 jni_DestroyJavaVM (fec46dc0) + 12a
 0805324c JavaMain (8047cb8) + fec
 feeae662 _thr_setup (fe650200) + 52
 feeae8c0 _lwp_start (fe650200, 0, 0, 0, 0, 0)

This is extremely useful when you need to interrogate a specific thread in a process (e.g., a thread that is consuming 100% of a CPU), or when a process has 100s or 1000s of threads (which is typically the case with Java applications). Nice!

Monitoring the ZFS ARC cache

The ZFS file system uses the adaptive replacement cache (ARC) to cache data in the kernel. Measuring ARC utilization is pretty straight forward, since ZFS populates a number of kstat values with usage data. Neelakanth Nadgir wrote a cool Perl script to summarize the ARC kstats, and a sample run is included below:

$ arcstat.pl

   Time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c  
12:31:28     0     0      0     0    0     0    0     0    0   283M  283M  
12:31:30     0     0      0     0    0     0    0     0    0   283M  283M  
12:31:32     3     1     42     1   42     0    0     1   42   283M  283M  
12:31:34     0     0      0     0    0     0    0     0    0   283M  283M  
12:31:36     4     1     25     1   25     0    0     1   33   283M  283M  
12:31:38     0     0      0     0    0     0    0     0    0   283M  283M  
12:31:40    24    23     95    23   95     0    0    23   95   284M  283M  
12:31:42    35    33     95    33   95     0    0    33   95   202K  283M  
12:31:44     0     0      0     0    0     0    0     0    0   202K  283M  
12:31:46     0     0      0     0    0     0    0     0    0   202K  283M  
12:31:48     0     0      0     0    0     0    0     0    0   202K  283M  
12:31:50     0     0      0     0    0     0    0     0    0   202K  283M  

Since numerous discussions have come up on zfs-discuss regarding ARC sizing (the size of the ARC is controlled by the zfs:zfs_arc_max tunable), folks will find the “arcsz” column extremely useful. Nice!

Enabling the DTrace hotspot provider after the JVM starts

While debugging a JVM performance issue a while back, I encountered the following error when I enabled the DTrace hotspot provider:

$ jinfo -flag +ExtendedDTraceProbes `pgrep java`
590: Unable to open door: target process not responding or HotSpot VM not loaded

After a bit of debugging, I figured out that the jinfo command needs to be run by the user the JVM runs as. Hopefully this will help others who encounter this annoying problem.

Stopping nfsmapid from querying DNS TXT records

With the introduction of NFSv4, user and group identifiers were changed to use the username@domain format. On Solaris hosts, the domain is determined using the following methods:

1. The NFSMAPID_DOMAIN variable is checked in /etc/default/nfs

2. DNS is queried for the _nfsv4idmapdomain TXT record

3. The configured DNS domain is used

4. The file /etc/defaultdomain is consulted

If a site doesn’t update the NFSMAPID_DOMAIN variable when deploying NFSv4, DNS will be queried for the domain to use. If the DNS server doesn’t contain a _nfsv4idmapdomain TXT record, you will see failed queries similar to the following:

host1 -> host2 ETHER Type=0800 (IP), size = 77 bytes
host1 -> host2 IP D=1.2.3.4 S=1.2.3.5 LEN=63, ID=19779, TOS=0x0, TTL=255
host1 -> host2 UDP D=53 S=52032 LEN=43
host1 -> host2 DNS C _nfsv4idmapdomain. Internet TXT ?
________________________________
host2 -> host1 ETHER Type=0800 (IP), size = 77 bytes
host2 -> host1 IP D=1.2.3.5 S=1.2.3.4 LEN=63, ID=26996, TOS=0x0, TTL=254
host2 -> host1 UDP D=52032 S=53 LEN=43
host2 -> host1 DNS R Error: 3(Name Error)

This can of course pose a problem for large sites, since the DNS server will be inundated with queries for records that don’t exist. If you want to stop these DNS queries from happening, you can add the domain to the NFSMAPID_DOMAIN variable in /etc/default/nfs. Shibby!