Dmitri, Snappy tends to be much faster than gzip with a slightly lower compression ratio. Not sure how stable it is though.
Thanks, Jun On Fri, Nov 30, 2012 at 1:33 PM, Dmitri Priimak < prii...@highwire.stanford.edu> wrote: > Well, the files that I am sending are themselves not compressed, but I do > use gzip compression when > sending data. kafka client is configured with option > > compression.codec=1 > > which implies gzip compression. Setting that value to 2 means "Snappy > compression". What compression > would you recommend? Is "Snappy compression" any good? > > -- > Dmitri Priimak > > On 11/30/2012 01:17 PM, Paul Sutter wrote: > > Any chance the stream was created by the gzip command? If so, that just > happens sometime. The only > > way we found is to confirm that the error happened at the expected end > of the file and not a > > too-short file (we learned this dealing with petabytes of gziped > logfiles) > > > > Apologies if thats irrelevant. > > > > Begin forwarded message: > > > >> *From:* Dmitri Priimak <prii...@highwire.stanford.edu <mailto: > prii...@highwire.stanford.edu>> > >> *Date:* November 30, 2012 12:26:10 PM PST > >> *To:* users@kafka.apache.org <mailto:users@kafka.apache.org> > >> *Subject:* *Re: Unexpected end of ZLIB input stream* > >> *Reply-To:* users@kafka.apache.org <mailto:users@kafka.apache.org> > >> > >> Well, it does not look like I can reproduce it anymore. While I was > able to consistently crash it > >> yesterday, today kafka broker runs without any problems. > >> > >> -- > >> Dmitri Priimak > >> > >> On 11/29/2012 02:10 PM, Dmitri Priimak wrote: > >>> On 11/29/2012 02:06 PM, Neha Narkhede wrote: > >>>> That just meant that we couldn't reproduce it during our testing, but > >>>> we occasionally do see it in production, which is much harder to > >>>> reproduce. > >>>> It will be great if you can reproduce the issue and attach the test > case. > >>> I see. I will try to isolate this problem. I am hoping that I can > locate exact sequence of messages > >>> that trigger this problem, but at the moment I cannot easily do that. > >>> > >>> -- > >>> Dmitri Priimak > >
