I recently gave a presentation at the local UNIX users group titledDebugging Java performance problems. The presentation describes various opensource tools and how they can be used to understand what is causing CPU, memory and lock contention issues inside a Java virtual machine. If there are additional tools not discussed in the presentation that you find useful for debugging Java performance problems, please let me know through the comment feature.
Slashdot and other sites had mention of the release of the 2.6.27 Linux kernel today. Some of the new features in the kernel take from here about improved SMP support for the page cache:
The page cache is the place where the kernel keeps in RAM a copy of a file to improve performance by avoiding disk I/O when the data that needs to be read is already on RAM. Each “mapping”, which is the data structure that keeps track of the correspondence between a file and the page cache, is SMP-safe thanks to its own lock. So when different processes in different CPUs access different files, there’s no lock contention, but if they access the same file (shared libraries or shared data files for example), they can hit some contention on that lock. In 2.6.27, thanks to some rules on how the page cache can be used and the usage of RCU, the page cache will be able to do lookups (ie., “read” the page cache) without needing to take the mapping lock, and hence improving scalability. But it will only be noticeable on systems with lots of cpus (page fault speedup of 250x on a 64 way system have been measured).*
Hasn’t Solaris been able to successfully scale vertically on 64+ CPU systems since the Solaris 2.5.1 days back on the E10k in 1996 without this type of contention? It also seems like this kernel version brings new enhancements to direct I/O. This also was implemented back in Solaris 6?
Its great that work is being done in the Linux kernel now to allow for vertical scalability, but it just seems to me that these are already mature kernel features that have been around in Solaris for years.
I have been a long time Cowboy Mouth fan, and still remember the first time I saw them play a concert in Delaware. They had an incredible amount of energy, and where one of the few bands I had seen that wanted the crowd involved in every aspect of the show. When I heard they were playing one of the local festivals here in town, I decided to venture out with a few friends to see them.
The show started off like all Cowboy Mouth shows with a ton of energy, and the lead singer (who also plays drums) jamming on drums to get the crowd into the show! I don’t recall the full set list, but the show peaked when they jammed out their classic hit “Jenny Says”! These guys are totally rad, and I can’t wait to see them again in the near future!
One really cool feature of ubuntu is the command-not-found script. If you try to execute a program, say, nmap – and that dpkg hasn’t been installed, the command-not-found script executes to parse a locally installed database to suggest, “hey, you should try executing # apt-get install nmap” or something similar.
This really makes ubuntu more user friendly. Fedora has the ability to use " # yum whatprovides <>” but yum is going to bomb out if you don’t have a working network connection. Yum’s first steps are to connect to online repositories to download filelists.sqlite.bz2, primary.sqlite,bz2, etc.
So what are these files? They are sqlite databases that contain data about what packages are available and the information about those packages… So lets say we’re looking for some binary called kde4-config, but I don’t have an active network connection…. What package contains that binary?
First, lets find all available instances of filelist.sqlite. Each fedora RPM repository is going to contain its own copy.
[cylon:/]
<svoboda>locate filelists.sqlite
/var/cache/yum/fedora/filelists.sqlite
/var/cache/yum/livna/filelists.sqlite
/var/cache/yum/updates/filelists.sqlite
/var/cache/yum/updates-newkey/filelists.sqlite
Cool – so there are 4 network repositories here. Lets mess around with the fedora repository first…
[cylon:/var/cache/yum/fedora]
<svoboda>ls -lh
total 89M
-rw-r--r-- 1 root root 0 2008-09-30 10:11 cachecookie
-rw-r--r-- 1 root root 58M 2008-09-17 13:46 filelists.sqlite
-rw-r--r-- 1 root root 2.0K 2008-09-30 07:19 mirrorlist.txt
drwxr-xr-x 2 root root 4.0K 2008-09-30 10:12 packages
-rw-r--r-- 1 root root 32M 2008-09-17 13:35 primary.sqlite
-rw-r--r-- 1 root root 2.4K 2008-09-30 10:11 repomd.xml
[cylon:/var/cache/yum/fedora]
<svoboda>file filelists.sqlite primary.sqlite
filelists.sqlite: SQLite 3.x database
primary.sqlite: SQLite 3.x database
Sure enough, they’re sqlite databases. Lets issue a select statement on filenames within the filelists.sqlite database and look for the kde4-config binary. First, lets take a peek at what we have available….
[cylon:/var/cache/yum/fedora]
<svoboda>sqlite3 filelists.sqlite
SQLite version 3.5.9
Enter ".help" for instructions
Before we get too deep, lets turn headers on and the output to column mode so things are a bit more readable…
sqlite> .headers on
sqlite> .mode column
Next, lets see what tables we have available…
sqlite> .tables
db_info filelist packages
Filelist looks interesting. What does it contain?
sqlite> .schema filelist
CREATE TABLE filelist ( pkgKey INTEGER, dirname TEXT, filenames
TEXT, filetypes TEXT);
CREATE INDEX dirnames ON filelist (dirname);
CREATE INDEX keyfile ON filelist (pkgKey);
sqlite> pragma table_info (filelist);
0|pkgKey|INTEGER|0||0
1|dirname|TEXT|0||0
2|filenames|TEXT|0||0
3|filetypes|TEXT|0||0
sqlite>
Cool. So lets extract the pkgKey and filenames column for any entries that contain the phrase “kde4-config” anywhere in the path…
sqlite> SELECT pkgKey,filenames FROM filelist
...> WHERE filenames LIKE '%kde4-config%';
pkgKey filenames
----------
---------------------------------------------------------------------------------------------
3703
preparetips/nepomuk-rcgen/meinproc4/kwrapper4/kunittestmodrunner/kshell4/kross/kjsconsole/kjscmd/kjs/kdeinit4_wrapper/kdeinit4_shutdown/kdeinit4/kded4/kde4automoc/kde4-doxygen.sh/kde4-config/kcookiejar4/kbuildsycoca4/checkXML
3703 kde4-config.1.gz/checkXML.1.gz
10746
preparetips/nepomuk-rcgen/meinproc4/kwrapper4/kunittestmodrunner/kshell4/kross/kjsconsole/kjscmd/kjs/kdeinit4_wrapper/kdeinit4_shutdown/kdeinit4/kded4/kde4automoc/kde4-doxygen.sh/kde4-config/kcookiejar4/kbuildsycoca4/checkXML
10746 kde4-config.1.gz/checkXML.1.gz
Awesome! We found something. It looks like we matched two packages – package 3703 and package 10746. Now, we just need to figure out what those packages are… Lets go peek in the second sqlite database file, primary.sqlite.
[cylon:/var/cache/yum/fedora]
<svoboda>sqlite3 primary.sqlite
SQLite version 3.5.9
Enter ".help" for instructions
sqlite> .header on
sqlite> .mode column
sqlite> .tables
conflicts db_info files obsoletes packages provides
requires
sqlite>
What does the packages table look like?
sqlite> pragma table_info (packages);
cid name type notnull dflt_value pk
---------- ---------- ---------- ---------- ---------- ----------
0 pkgKey INTEGER 0 1
1 pkgId TEXT 0 0
2 name TEXT 0 0
3 arch TEXT 0 0
4 version TEXT 0 0
5 epoch TEXT 0 0
6 release TEXT 0 0
7 summary TEXT 0 0
8 descriptio TEXT 0 0
9 url TEXT 0 0
10 time_file INTEGER 0 0
11 time_build INTEGER 0 0
12 rpm_licens TEXT 0 0
13 rpm_vendor TEXT 0 0
14 rpm_group TEXT 0 0
15 rpm_buildh TEXT 0 0
16 rpm_source TEXT 0 0
17 rpm_header INTEGER 0 0
18 rpm_header INTEGER 0 0
19 rpm_packag TEXT 0 0
20 size_packa INTEGER 0 0
21 size_insta INTEGER 0 0
22 size_archi INTEGER 0 0
23 location_h TEXT 0 0
24 location_b TEXT 0 0
25 checksum_t TEXT 0 0
Cool. We want to match the name of the package that has the pkgKey
entries 3703 or 10746. Lets adjust our width so we don't get truncated
values displayed to the screen...
sqlite> .width 10 50
And query aganist that pkgKey value.
sqlite> SELECT name,location_href FROM packages
...> WHERE pkgKey = 3703;
name location_href
---------- --------------------------------------------------
kdelibs Packages/kdelibs-4.0.3-7.fc9.i386.rpm
sqlite> SELECT name,location_href FROM packages
...> WHERE pkgKey = 10746;
name location_href
---------- --------------------------------------------------
kdelibs Packages/kdelibs-4.0.3-7.fc9.x86_64.rpm
Oh my gosh, this is the funniest (and coolest) site I have seen in a LONG LONG time!