One of my friends recently reached out with a fun problem. His monitoring system was periodically not firing when file systems grew past the thresholds he defined. When we hopped on one of his EC2 instances to debug the issue, I noticed that we were getting a permission denied (EACCES) errno when running df as their monitoring user:
$ df -h /vault/data
df: ‘/vault/data’: Permission denied
When we ran the same command as trusty UID 0, everything worked as expected:
$ sudo df -h /vault/data
Filesystem Size Used Avail Use% Mounted on
/dev/nvme1n1 20G 1G 20G 1% /vault/data
A quick check with strace verified this as well:
$ strace -e trace=statfs df -h 2>&1 | grep vault
statfs("/vault/data", 0x7ffe00af8aa0) = -1 EACCES (Permission denied)
If you aren’t familiar with statfs(2), it returns information about a mounted file system in a statfs structure. Here is a blurb from the manual page describing which information is returned:
The function statfs() returns information about a mounted file system. path is the pathname of any file within the mounted file system. buf is a pointer to a statfs structure defined approximately as follows:
struct statfs {
__SWORD_TYPE f_type; /* type of file system (see below) */
__SWORD_TYPE f_bsize; /* optimal transfer block size */
fsblkcnt_t f_blocks; /* total data blocks in file system */
fsblkcnt_t f_bfree; /* free blocks in fs */
fsblkcnt_t f_bavail; /* free blocks available to
unprivileged user */
fsfilcnt_t f_files; /* total file nodes in file system */
fsfilcnt_t f_ffree; /* free file nodes in fs */
fsid_t f_fsid; /* file system id */
__SWORD_TYPE f_namelen; /* maximum length of filenames */
__SWORD_TYPE f_frsize; /* fragment size (since Linux 2.6) */
__SWORD_TYPE f_spare[5];
};
I thought that df was setuid root like the mount uility, so when I initially saw the permission denied error I thought it was something unrelated to permissions. But low and behold it was indeed due to df not being setuid root:
$ ls -la /usr/bin/df
-rwxr-xr-x 1 root root 100856 Jan 23 2020 /usr/bin/df
So when df tried to statfs() this file system as an unprivileged user, it got the permission denied error. We found a simple wrokaround to get things working, and I learned something new in the process. Neato!