Assuming you’re using the new producer (org.apache.kafka.clients.producer) the property is called compression.type I believe.
Double check it’s being passed correctly to the process. The producer logs the properties it uses if you set the logging level to info. B > On 18 Mar 2016, at 16:10, R P <hadoo...@outlook.com> wrote: > > Hey Ben, Thanks again for your response. > > I checked log files using DumpLogSegments --print-data-log. And compression > codec used is showing NoCompressionCodec ( compresscodec: NoCompressionCodec > ) . > > I am guessing my configuration is not correct. I am adding following line in > Kafka producer.properties config file. I am using Kafka 0.8.2 > > compression.codec=gzip > # compression.codec=1 (tried with old config value too) > > And for this experiment I am sending data via kafka-console-producer.sh. > Still I don't see any compression being used. > > What am I missing? > > Thanks, > R P > > > On 3/18/16 8:24 AM, R P wrote: >> Thanks for the response Ben. >> I am wondering why is "du" command not showing reduced size when compression >> is used. >> I ran an experiment with compression enabled on a topic and without >> compression enabled sending same amount of data in both cases. I used single >> node 1 replication factor Kafka instance on Mac OS. >> I didn't see any difference in the data size stored on disk. In both cases >> data stored on disk in log files had same size equals to the data sent to >> Kafka. >> How do I verify that compression is being used and data stored on disk has >> savings in space due to compression? >> Thanks, >> R P >> >> _____________________________ >> From: Ben Stopford <b...@confluent.io >> <mailto:b...@confluent.io><mailto:b...@confluent.io> >> <mailto:b...@confluent.io>> >> Sent: Friday, March 18, 2016 7:50 AM >> Subject: Re: Question regarding compression of topics in Kafka >> To: <users@kafka.apache.org >> <mailto:users@kafka.apache.org><mailto:users@kafka.apache.org> >> <mailto:users@kafka.apache.org>> >> >> >> Yes it will compress the data stored on the file system if you specify >> compression in the producer. You can check whether the data is compressed on >> disk by running the following command in the data directory. >> kafka-run-class kafka.tools.DumpLogSegments --print-data-log --files >> latest-log-file.log >> >>> > On 17 Mar 2016, at 23:59, R P <hadoo...@outlook.com >>> > <mailto:hadoo...@outlook.com><mailto:hadoo...@outlook.com> >>> > <mailto:hadoo...@outlook.com>> wrote: >>> > >>> > Hello All, >>> > Does kafka support compressing storage logs stored in log dir? >>> > What does compression.type=(gzip/snappy) in server.properties do? >>> > >>> > Based on documents I am assuming that it will compress the logs on local >>> > file system. >>> > I ran a quick experiment and found that my logs stored on local disk are >>> > not getting compressed. >>> > Size of data stored on disk is same with or without compression. >>> > >>> > I am using following configuration properties in server.properties >>> > config file. >>> > >>> > compression.type=gzip >>> > compressed.topics="gzip-topic" >>> > >>> > Thanks for reading and appreciate any responses. >>> > >>> > Thanks, >>> > R P >> >> >
