How badly compressed are they? Problems in the codec, or in the data that comes out of the codec?
We've had some lzo corruption problems, and so far have simply been dealing with that by doing correctness tests in our log mover pipeline before moving into the "data warehouse" area. Skipping bad files silently seems like asking for trouble (at some point the problem quietly grows and you wind up skipping most of your data), so I've been avoiding putting something like that in so that when things are badly broken, we get some early pain rather than lots of late pain. D On Tue, Jan 25, 2011 at 2:54 PM, Kim Vogt <k...@simplegeo.com> wrote: > Hi, > > I'm processing gzipped compressed files in a directory, but some files are > corrupted and can't be decompressed. Is there a way to skip the bad files > with a custom load func? > > -Kim >
