I have on a couple of occasions been asked to restore files that were deleted. I’ve been fortunate up to this point that I have always been able to get the deleted file back, either through file system manipulation, dd’ing of a device or by restoring the file from a previous back up. One thing I’ve never quite understood is why UNIX and Linux Operating Systems don’t ship with a undelete utility for each file system type. Assuming you don’t zero out the metadata and the data blocks haven’t been re-used, restoring a file is pretty straight forward.
My luck almost came to an end the other day when I accidentally deleted a Nagios configuration file on my desktop. This was only a test installation, so I didn’t take the time to back everything up to a remote destination. I talked about one way to undelete a file on an EXT file system in the past, and was curious if any additional tools had been written to recover files. I found the amazingly cool extundelete utility, and after several tests it appears to be true to its name!
Extundelete requires an unmounted file system to operate on, so you will need to unmount the file system that contains the deleted files prior to recovering these files. To use the tool, you will need to pass extundelete one or more options and the block device associated with the filesystem. The options will tell extundelete which files to undelete, and will allow you to recover a single file, a directory and its contents, or ALL of the files that have been deleted. Be careful with the last one. ;)
So say we have a file system named /mnt, and we accidentally deleted a file named /services:
$ cd /mnt && rm services
If we use the “–restore-file” option, we can restore the file named /services:
$ umount /mnt
$ extundelete --restore-file '/services' /dev/sdc1
WARNING: Extended attributes are not restored.
Loading filesystem metadata ... 13 groups loaded.
Loading journal descriptors ... 22 descriptors loaded.
Writing output to directory RECOVERED_FILES/
Restored inode 12 to file RECOVERED_FILES/services
$ ls -la RECOVERED_FILES/
total 20
drwxr-xr-x. 2 root root 4096 Apr 15 14:49 .
dr-xr-x---. 16 root root 4096 Apr 15 14:49 ..
-rw-r--r--. 1 root root 2241973580461 Apr 15 14:49 services
$ more RECOVERED_FILES/services
# /etc/services:
# $Id: services,v 1.51 2010/11/12 12:45:32 ovasik Exp $
#
# Network services, Internet style
# IANA services version: last updated 2010-11-09
........
Now let’s say we have three critical files in /mnt named 1, 2 and 3 that are blown away:
$ cd /mnt && ls -la
total 1943
drwxr-xr-x. 3 root root 1024 Apr 15 14:50 .
dr-xr-xr-x. 28 root root 4096 Apr 15 14:48 ..
-rw-r--r--. 1 root root 651949 Apr 15 14:50 1
-rw-r--r--. 1 root root 651949 Apr 15 14:50 2
-rw-r--r--. 1 root root 651949 Apr 15 14:50 3
drwx------. 2 root root 12288 Apr 15 14:44 lost+found
$ cd /mnt && rm 1 2 3
Yikes! No need to fear though, we can run extundelete with the “–restore-all” option to restore all of the files deleted in the file system:
$ umount /mnt
$ extundelete --restore-all /dev/sdc1
WARNING: Extended attributes are not restored.
Loading filesystem metadata ... 13 groups loaded.
Loading journal descriptors ... 34 descriptors loaded.
Searching for recoverable inodes in directory / ...
3 recoverable inodes found.
Looking through the directory structure for deleted files ...
Restored inode 12 to file RECOVERED_FILES/1
Restored inode 13 to file RECOVERED_FILES/2
Restored inode 14 to file RECOVERED_FILES/3
0 recoverable inodes still lost.
$ ls -la RECOVERED_FILES/
total 44
drwxr-xr-x. 2 root root 4096 Apr 15 14:51 .
dr-xr-xr-x. 28 root root 4096 Apr 15 14:48 ..
-rw-r--r--. 1 root root 2241973580461 Apr 15 14:51 1
-rw-r--r--. 1 root root 2241973580461 Apr 15 14:51 2
-rw-r--r--. 1 root root 2241973580461 Apr 15 14:51 3
You may have noticed that the recovered files are showing up as 2.1TB in size, and for reasons I am not yet clear on the files are restored as HUGE sparse files (I’ll post an update once I figure this out). They aren’t actually 2.1TB:
$ cd RECOVERED_FILES && du -sh *
12K 1
12K 2
12K 3
As with any file system undelete utility, it can’t work if the file system is active and re-using metadata and data blocks. This utility was able to restore all of the files I removed, but then again I didn’t test it on a system that is performing 100s or 1000s of file operations per minute. This is no way a replacement for a quality backup system, but a tool you bring in once all other measures fail. I would like to thank Nic for the awesome work he did on this utility. I haven’t studied the internal of the program yet, so use at your own risk (I haven’t seen any adverse effects from it, but that’s not to say there aren’t any!).