We are using official Apache Kafka stuff. Nothing different.
On Fri, 18 Sep 2020 at 03:30, Shaohan Yin <shaohan....@gmail.com> wrote: > Hi, > > > > Are you using the official Java client or any third party clients? > > > > On Thu, 17 Sep 2020 at 22:53, M. Manna <manme...@gmail.com> wrote: > > > > > Hey Shaohan, > > > > > > Thanks for your reply, much appreciated. We are using Kafka 2.4.1. > Although > > > we use the Confluent platform under Confluent Licence, I don't think this > > > matters. > > > > > > We are using compression.type=GZIP for all our producers. > > > > > > We create all topics with compression.type=GZIP i.e. per topic level. > When > > > I do a "describe" of a kafka topic, I see metadata confirming that it's > > > enabled for GZIP compressed data. > > > > > > That's all I could say. > > > > > > Regards, > > > > > > > > > > > > > > > On Thu, 17 Sep 2020 at 15:01, Shaohan Yin <shaohan....@gmail.com> wrote: > > > > > > > Hi, > > > > > > > > Could you specify your version and the protocol of your broker? > > > > > > > > From the client of 2.5.1 I didn't see any changes that could be made to > > > the > > > > client compression. > > > > Maybe you could check if the compression.type is set on the topic level > > > or > > > > the broker side? > > > > > > > > Cheers > > > > > > > > On Thu, 17 Sep 2020 at 20:19, M. Manna <manme...@gmail.com> wrote: > > > > > > > > > Hello, > > > > > > > > > > I am trying to understand the compression.type settings and the > growth > > > > > impact from this. With a compression type (valid) settings, this > > > ensures > > > > > that the message in the topic stays compressed and has to be > > > decompressed > > > > > by the consumer. > > > > > > > > > > However, it may be possible that the compression will not happen i.e. > > > > NONE > > > > > if the uncompressed size is the same as compressed. I am trying to > use > > > > this > > > > > command to verify: > > > > > > > > > > ./kafka-run-class kafka.tools.DumpLogSegments --files > > > > > <log-dir>/filename.log > > > > > --print-data-log | grep -iE "compresscodec: NONE" > > > > > > > > > > I can see a lot of entries with NONE. But all my producers are > writing > > > > with > > > > > compression.type=GZIP. Is this expected? > > > > > > > > > > Regards, > > > > > > > > > > > > > >
