>As others have noted, different vendors dedup at different levels of >granularity.
I think I'd put it slightly differently. I'd say that they each approach it differently. Those different approaches may have advantages and disadvantages with different data types. >When I spoke to Diligent at the Gartner conference over >a year ago, they were very tight-lipped about their actual >algorithm. The patent was filed. It's not that secret. ;) They are quite different in their approach, and it's a little different to grock. But based on what I know about their approach, the scenario that started the discussion may indeed be a limitation. (Or all the vendors may have this limitation; I have some questions out to them.) >The[y] would, however, state that they were able to dedup >parts of two files that had similar data, but were not >identical. I.e., if data was inserted at the beginning of the file, >some parts of the end of the file could still be deduped. Neat trick >if it's true. Any de-dupe vendor is able to claim that. If it wasn't true, they wouldn't see the de-dupe rates they're seeing. They can also identify blocks that are common between a file in the file system and the same file emailed via Exchange. >Other vendors dedup at the file or block (or chunk) level. If a vendor doesn't do subfile de-dupe, then they're not a de-dupe vendor; they're a CAS vendor. File-level de-dupe is CAS (i.e. Centerra, Archivas), and the de-dupe is not really pitched as the main feature. It's about using the signature as a way to provide immutability of data stored in the CAS array. >I've not been able to gather much more detail about the specific >dedup algorithms, but hope to get some more info this fall, as take a >closer look at these products. If anyone has more details, I'd love >to hear them. I wrote this article that may help: http://tinyurl.com/3588fb . I also blog about de-dupe quite a bit at www.backupcentral.com.
