A great introduction to ZFS de-duplication

I’ve been looking into deploying ZFS de-duplication, and I have one application in particular (backup staging) that would greatly benefit from it. George Wilson did an awesome introduction to ZFS de-duplication video, and it’s a great place to get started. I’m planning to start testing out de-duplication as soon as my SSDs are ordered, and hopefully I will have some positive results to report!

3 thoughts on “A great introduction to ZFS de-duplication”

  1. ZFS dedupe works extremely well for backups, especially if you use something like rsync to store individual files in separate ZFS filesystems.

    The big thing is to not skimp on RAM. The more RAM you can stick in the server, the better. 16 GB is a good starting point. We started with 16, and have upgraded to 20 and 24 GB in two servers, and 32 GB in another.

    We do backups for just under 150 Linux and FreeBSD servers each night using rsync. Then replicate the backups to another server across town via rsync. Works extremely well.

    We have a combined dedupe + compression ratio over 5x on one server and just over 3x on another. Both are increasing slowly each week.

  2. Be sure to have lots of memory. And then double that!

    Also SSD-based L2ARC. But of course, you need memory (DDT) to properly address the L2ARC… :)

    We found dedupe wasn’t worth the hassle currently (at least’s ZFS’s currently implementation).

  3. What about this feature in the next coming BTRFS in Linux ? Same pros/cons than ZFS ?
    Great video. :)

Leave a Reply

Your email address will not be published. Required fields are marked *