On 03/17/2016 04:03 PM, Virendra Pratap Singh wrote:
More like getting a feel from the community about using lz4 for compression? Has anyone used in the kafka setup. I am aware that gzip and snappy are more older implementation and regressed. Given that lz4 has better compression/decompression cycles (though slightly less compression ratio), was thinking to leveraging the same. Regards,Virendra
i use the lz4 compression algorithm quite extensively in conjunction with ZFS (ZFS can configure a filesystem to do transparent compression) and have not had any issues with it under load. i've also found that it does a better job than snappy with negligible overhead that i've been able to observe.
i tend to avoid gzip in production as i have measured more overhead using this algorithm, and generally speaking i've found lz4 to compact data better.
i am not super familiar with the lz4 code as implemented in kafka, but i would assume the java implementation is pretty solid.
hope this helps, -pete -- Pete Wright Lead Systems Architect Rubicon Project pwri...@rubiconproject.com 310.309.9298
