The LZ4 implementation "works" but has a framing bug that can make third party client use difficult. See KAFKA-3160. If you only plan to use the official Java client then that issue shouldn't be a problem.
-Dana On Mar 21, 2016 12:26 PM, "Pete Wright" <pwri...@rubiconproject.com> wrote: > > > On 03/17/2016 04:03 PM, Virendra Pratap Singh wrote: > >> More like getting a feel from the community about using lz4 for >> compression? Has anyone used in the kafka setup. >> I am aware that gzip and snappy are more older implementation and >> regressed. Given that lz4 has better compression/decompression cycles >> (though slightly less compression ratio), was thinking to leveraging the >> same. >> Regards,Virendra >> >> > i use the lz4 compression algorithm quite extensively in conjunction with > ZFS (ZFS can configure a filesystem to do transparent compression) and have > not had any issues with it under load. i've also found that it does a > better job than snappy with negligible overhead that i've been able to > observe. > > i tend to avoid gzip in production as i have measured more overhead using > this algorithm, and generally speaking i've found lz4 to compact data > better. > > i am not super familiar with the lz4 code as implemented in kafka, but i > would assume the java implementation is pretty solid. > > hope this helps, > -pete > > -- > Pete Wright > Lead Systems Architect > Rubicon Project > pwri...@rubiconproject.com > 310.309.9298 >
