Blog O' Matty


Finding out the file name associated with an inode

This article was posted by Matty on 2006-03-13 11:30:00 -0400 -0400

While perusing the logfiles on one of my personal systems a while back, I noticed that an error was generated with the inode of a troublesome file (I can’t recall the specifics). Since the error was rather vague and contained an inode number, I wanted to resolve the inode number to a file name to see which file was making the application angry. There are NUMEROUS way to resolve inode numbers to file names, but my favorite methods involve find and ncheck. To use find to locate the file name associated with an inode, you can pass the inode number to find’s “-inum” option:

$ find /export/home -inum 12345 -ls

If find doesn’t fit your fancy, you can also use the ncheck utility to resolve a file name to an inode:

$ ncheck -i 261

/dev/dsk/c0t0d0s0:
261 /usr/bin/cktime
261 /usr/sadm/bin/errtime
261 /usr/sadm/bin/helptime
261 /usr/sadm/bin/valtime

The find command tends to run significantly faster, but ncheck can be a viable alternative in some cases.

Viewing system call errors with DTrace

This article was posted by Matty on 2006-03-13 09:56:00 -0400 -0400

When an application needs to access one of the services provided by the kernel, the application typically invokes a system call. When a system call returns, the system call will typically provide a return value to indicate success or failure, and in the case of a failure, the global errno variable will be set to indicate the specific error that occurred. To see what errors are occurring on a system or within a specific process, the DTraceToolkit’s errinfo script can be used:

$ errinfo -n _app

EXEC SYSCALL ERR DESC
_app stat 2 No such file or directory
_app lseek 29 Illegal seek
_app pause 4 interrupted system call

This will print the system calls that returned with an error condition, the value of the errno, and the error number description from errno.h. This is a super useful script, and can make tracking down errors super easy.

Locating pipe endpoints

This article was posted by Matty on 2006-03-11 01:00:00 -0400 -0400

While debugging a performance problem this week, I noticed that one of our applications was issuing 1000s of read and write system calls each minute to file descriptor 19. To get a better idea of what file descriptor 19 was used for, I used the trusty Solaris pfiles utility:

$ pfiles 8988

[ ..... ]

19: S_IFIFO mode:0000 dev:291,0 ino:382227 uid:911116 gid:315 size:0 O_RDWR

Well this is interesting, the application is reading and writing to a pipe. But what resides on the remote side of the pipe? To answer this question, I passed the “ino” numeric argument to find’s “-inum” option:

$ find /proc -inum 382227 2>/dev/null

/proc/8996/fd/0
/proc/8995/fd/0
/proc/8988/fd/20

Once I located the process identifiers, I used the ps command to locate the pipe endpoints:

$ ps -ef | egrep '(8995|8996|8988)'

app 8996 8995 0 06:20:31 ? 0:00 /opt/VENDOR/logging.binary
app 8995 8988 0 06:20:31 ? 0:00 /bin/sh -c /opt/VENDOR/logging.binary
app 8988 1 1 06:20:31 ? 5:48 /opt/VENDOR/application

This definitely explains why they are issuing lots of writes, but why reads? Only the vendor can tell. :)

Tracing shared library calls

This article was posted by Matty on 2006-03-06 22:01:00 -0400 -0400

While debugging a problem over the weekend, I needed a way to trace method calls by library. This can be accomplished on Solaris systems with Dtrace, truss, or with the less well known sotruss shell script:

$ /bin/sotruss /usr/sbin/metastat

metastat -> libc.so.1:*atexit(0xff3c0220, 0x26400, 0x0)
metastat -> libc.so.1:*atexit(0x14d94, 0xfefecbc0, 0xa7bf4)
metastat -> libc.so.1:*setlocale(0x6, 0x14fd8, 0x0)
metastat -> libc.so.1:*textdomain(0x14fdc, 0xfefcf566, 0xff3a0280)
metastat -> libmeta.so.1:*sdssc_bind_library(0x27430, 0x0, 0x5ca8c)
metastat -> libmeta.so.1:*md_init(0x1, 0xffbffb1c, 0x0)
metastat -> libc.so.1:*getopt(0x1, 0xffbffb1c, 0x14ff0)
metastat -> libmeta.so.1:*metasetname(0x26598, 0xffbffa80, 0x14ff0)
metastat -> libc_psr.so.1:*memset(0xffbff178, 0x0, 0x860)
metastat -> libmeta.so.1:*metaioctl(0x560d, 0xffbff178, 0xffbff178)
metastat -> libc.so.1:*getenv(0x150c0, 0xffbffa80, 0x14ff0)
metastat -> libmeta.so.1:*meta_check_ownership(0x27448, 0xffbff9a8, 0x0)
metastat -> libmeta.so.1:*meta_print_all(0x27448, 0x0, 0xffbff9a0)
metastat -> libmeta.so.1:*meta_getalldevs(0x27448, 0xffbff9a0, 0x0)
metastat -> libmeta.so.1:*meta_create_non_dup_list(0x28680, 0xffbff93c, 0xffbff938)
metastat -> libmeta.so.1:*meta_create_non_dup_list(0x2c118, 0xffbff93c, 0x6)
metastat -> libmeta.so.1:*meta_create_non_dup_list(0x287a0, 0xffbff93c, 0x93394)
metastat -> libmeta.so.1:*meta_create_non_dup_list(0x2b770, 0xffbff93c, 0x93394)
metastat -> libmeta.so.1:*meta_print_devid(0x27448, 0x26a18, 0x27850)
metastat -> libmeta.so.1:*Free(0x27778, 0x26a18, 0x0)
metastat -> libmeta.so.1:*Free(0x27850, 0x1084, 0x93394)
metastat -> libmeta.so.1:*Free(0x27868, 0x1084, 0x93394)
metastat -> libmeta.so.1:*Free(0x27880, 0x1084, 0x93394)
metastat -> libmeta.so.1:*md_exit(0x27448, 0x0, 0x2eec0)

This is a cool script, and makes it super easy to dig through library calls.

Fixing typos with bash

This article was posted by Matty on 2006-03-04 13:09:00 -0400 -0400

I frequently find myself make silly typing mistakes when changing between directories. Since this is especially annoying with long directory names, I use the bash cdspell option to transparently fix my typing mistakes. To set this nifty option, you can add the following line to your .bash_profile:

$ grep cdspell .bash_profile
shopt -s cdspell

This is a super useful option, and makes life a bit easier for folks with no typing skillz. Niiiiice!