Re: Kafka 2.0.0 - How to verify if Kafka compression is working

2021-05-13 Thread Scott Carey
ting over and > over again. > > At least with that it could have compressed. But to no avail! > > I also applied batch.size=1 and linger.ms=10 still it didnt work. > > On Thu, May 13, 2021 at 11:32 AM Scott Carey > wrote: > > > Snappy and lz4 do not have entropy

Re: Kafka 2.0.0 - How to verify if Kafka compression is working

2021-05-12 Thread Scott Carey
Snappy and lz4 do not have entropy encoders ( https://en.wikipedia.org/wiki/Entropy_encoding). If your data is random text they will not compress. If your text is a string of all zeros or any repeating pattern, it will compress significantly. If its something like JSON, or XML it will compress.

Re: duplicat and lost message in Kafka

2013-04-22 Thread Scott Carey
On 4/22/13 12:03 PM, "Yu, Libo " wrote: >Hi, >I have read both the Kafka design document at >http://kafka.apache.org/design.html >and the paper. And I have two questions about lost and duplicate message. > >"Without acknowledging the producer, there is no guarantee that every >published >message