Thanks for the response Ben. I am wondering why is "du" command not showing reduced size when compression is used. I ran an experiment with compression enabled on a topic and without compression enabled sending same amount of data in both cases. I used single node 1 replication factor Kafka instance on Mac OS. I didn't see any difference in the data size stored on disk. In both cases data stored on disk in log files had same size equals to the data sent to Kafka. How do I verify that compression is being used and data stored on disk has savings in space due to compression? Thanks, R P
_____________________________ From: Ben Stopford <b...@confluent.io<mailto:b...@confluent.io>> Sent: Friday, March 18, 2016 7:50 AM Subject: Re: Question regarding compression of topics in Kafka To: <users@kafka.apache.org<mailto:users@kafka.apache.org>> Yes it will compress the data stored on the file system if you specify compression in the producer. You can check whether the data is compressed on disk by running the following command in the data directory. kafka-run-class kafka.tools.DumpLogSegments --print-data-log --files latest-log-file.log > On 17 Mar 2016, at 23:59, R P > <hadoo...@outlook.com<mailto:hadoo...@outlook.com>> wrote: > > Hello All, > Does kafka support compressing storage logs stored in log dir? > What does compression.type=(gzip/snappy) in server.properties do? > > Based on documents I am assuming that it will compress the logs on local > file system. > I ran a quick experiment and found that my logs stored on local disk are > not getting compressed. > Size of data stored on disk is same with or without compression. > > I am using following configuration properties in server.properties > config file. > > compression.type=gzip > compressed.topics="gzip-topic" > > Thanks for reading and appreciate any responses. > > Thanks, > R P
