On Jul 31, 2012, at 10:07 AM, Nigel W wrote: > On Tue, Jul 31, 2012 at 9:36 AM, Ray Arachelian <r...@arachelian.com> wrote: >> On 07/31/2012 09:46 AM, opensolarisisdeadlongliveopensolaris wrote: >>> Dedup: First of all, I don't recommend using dedup under any >>> circumstance. Not that it's unstable or anything, just that the >>> performance is so horrible, it's never worth while. But particularly >>> with encrypted data, you're guaranteed to have no duplicate data >>> anyway, so it would be a pure waste. Don't do it. >>> _______________________________________________ zfs-discuss mailing >>> list zfs-discuss@opensolaris.org >>> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >> >> One thing you can do is enable dedup when you copy all your data from >> one zpool to another, then, when you're done, disable dedup. It will no >> longer waste a ton of memory, and your new volume will have a high dedup >> ratio. (Obviously anything you add after you turn dedup off won't be >> deduped.) You can keep the old pool as a backup, or wipe it or whatever >> and later on do the same operation in the other direction. > > Once something is written deduped you will always use the memory when > you want to read any files that were written when dedup was enabled, > so you do not save any memory unless you do not normally access most > of your data. > > Also don't let the system crash :D or try to delete too much from the > deduped dataset :D (including snapshots or the dataset itself) because > then you have to reload all (most) of the DDT in order to delete the > files. This gets a lot of people in trouble (including me at $work > :|) because you need to have the ram available at all times to load > the most (>75% to grab a number out of the air) in case the server > crashes. Otherwise you are stuck with a machine trying to verify its > filesystem for hours. I have one test system that has 4 GB of RAM and > 2 TB of deduped data, when it crashes (panic, powerfailure, etc) it > would take 8-12 hours to boot up again. It now has <1TB of data and > will boot in about 5 minutes or so.
I believe what you meant to say was "dedup with HDDs sux." If you had used fast SSDs instead of HDDs, you will find dedup to be quite fast. -- richard -- ZFS Performance and Training richard.ell...@richardelling.com +1-760-896-4422
_______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
