> can you guess? wrote: ...
If you include > 'image files of various > > sorts', as he did (though this also raises the > question of whether we're > > still talking about 'consumers'), then you also > have to specify exactly > > how damaging single-bit errors are to those various > 'sorts' (one might > > guess not very for the uncompressed formats that > might well be taking up > > most of the space). And since the CERN study seems > to suggest that the > > vast majority of errors likely to be encountered at > this level of > > incidence (and which could be caught by ZFS) are > *detectable* errors, > > they'll (in the unlikely event that you encounter > them at all) typically > > only result in requiring use of a RAID (or backup) > copy (surely one > > wouldn't be entrusting data of any real value to a > single disk). > > > I have to comment here. As a bloke with a bit of a > photography > habit - I have a 10Mpx camera and I shoot in RAW mode > - it is > very, very easy to acquire 1Tb of image files in > short order. So please respond to the question that I raised above (and that you yourself quoted): just how much damage will a single-bit error do to such a RAW file? > > Each of the photos I take is between 8 and 11Mb, and > if I'm > at a sporting event or I'm travelling for work or > pleasure, > it is *incredibly* easy to amass several hundred Mb > of photos > every single day. Even assuming that you meant 'MB' rather than 'Mb' above, that suggests that it would take you well over a decade to amass 1 TB of RAW data (assuming that, as you suggest both above and later, you didn't accumulate several hundred MB of pictures *every* day but just on those days when you were traveling, at a sporting event, etc.). > > I'm by no means a professional photographer (so I'm > not out > taking photos every single day), although a very > close friend > of mine is. My photo storage is protected by ZFS with > mirroring > and backups to dvd media. My profotog friend has 3 > copies of > all her data - working set, immediate copy on > usb-attached disk, > and second backup also on usb-attached disk but > disconnected. Sounds wise on both your parts - and probably makes ZFS's extra protection pretty irrelevant (I won't bother repeating why here). > > Even if you've got your original file archived, you > still need > your working copies available, and Adobe Photoshop > can turn that > RAW file into a PSD of nearly 60Mb in some cases. If you really amass all your pictures this way (rather than, e.g., use Photoshop on some of them and then save the result in a less verbose format), I'll suggest that this takes you well beyond the 'consumer' range of behavior. > > It is very easy for the storage medium to acquire > some degree > of corruption - whether it's a CF or SD card, they > all use > FAT32. I have been in the position of losing photos > due to > this. Not many - perhaps a dozen over the course of > 12 months. So in those cases you didn't maintain multiple copies. Bad move, and usually nothing that using ZFS could help with. While I'm not intimately acquainted with flash storage, my impression is that data loss usually occurs due to bad writes (since once written the data just sits there persistently and AFAIK is not subject ot the kinds of 'bit rot' that disk and tape data can experience). So if the loss occurs to the original image captured on flash before it can be copied elsewhere, you're just SOL and nothing ZFS offers could help you. > > That flipped bit which you seem to be dismissing as > "hardly... > a disaster" can in fact make your photo file totally > useless, > because not only will you probably not be able to get > the file > off the media card, but whatever software you're > using to keep > track of your catalog will also be unable to show you > the > entire contents. That might be the image itself, or > it might > be the equally important EXIF information. Here come those pesky numbers again, I'm afraid. Because given that the size difference between your image data and the metadata (including EXIF information, if that's what I suspect it is) is at least several orders of magnitude, the chance that the bad bit will be in something other than the image data is pretty negligible. So even if you can format your card to use ZFS (can you? if not, what possible relevance does your comment above have to this discussion?), doing so won't help at all: the affected file will still be inaccessible (unless you use ZFS to create a redundant pool across multiple such cards: is that really what you're suggesting should be done?) both to normal extraction (though couldn't dd normally get off everything but the bad sector?) and to your cataloging software. > > I don't depend on FAT32-formatted media cards to make > my > living, fortunately, but if I did I imagine I'd > probably end > up only using each card for about a month before > exercising > caution and purchasing a new one rather than > depending on the > card itself to be reliable any more. The 'wear leveling' algorithms on current cards are supposedly quite good, so that degree of caution may no longer be necessary (at least if you get a decent card). > > 1Tb of photos shot on a 10MPx camera in the camera's > native > RAW format is around 100,000 photos. It's not > difficult to > imagine a "consumer" having that sort of storage > requirement. Well, as I noted above in your own case it would take well over a decade to generate 'that sort of storage requirement'. Furthermore, it *is* difficult to imagine a 'consumer' keeping all of them in RAW format - and even if they did, the question still remains: just how much will a single-bit error affect such a photo? - bill This message posted from opensolaris.org _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
