Re: Question regarding compression of topics in Kafka

2016-03-19 Thread R P
, - R P From: Ben Stopford Sent: Friday, March 18, 2016 9:21 AM To: users@kafka.apache.org Subject: Re: Question regarding compression of topics in Kafka Assuming you’re using the new producer (org.apache.kafka.clients.producer) the property is called

Re: Question regarding compression of topics in Kafka

2016-03-19 Thread Ben Stopford
gt; _____ >> From: Ben Stopford > <mailto:b...@confluent.io><mailto:b...@confluent.io> >> <mailto:b...@confluent.io>> >> Sent: Friday, March 18, 2016 7:50 AM >> Subject: Re: Question regarding compression of topics in Kafka >>

Re: Question regarding compression of topics in Kafka

2016-03-19 Thread Ben Stopford
Yes it will compress the data stored on the file system if you specify compression in the producer. You can check whether the data is compressed on disk by running the following command in the data directory. kafka-run-class kafka.tools.DumpLogSegments --print-data-log --files latest-log-file.l

Re: Question regarding compression of topics in Kafka

2016-03-19 Thread R P
on? Thanks, R P _ From: Ben Stopford mailto:b...@confluent.io>> Sent: Friday, March 18, 2016 7:50 AM Subject: Re: Question regarding compression of topics in Kafka To: mailto:users@kafka.apache.org>> Yes it will compress the data stored on the file system if

Question regarding compression of topics in Kafka

2016-03-18 Thread R P
Hello All, Does kafka support compressing storage logs stored in log dir? What does compression.type=(gzip/snappy) in server.properties do? Based on documents I am assuming that it will compress the logs on local file system. I ran a quick experiment and found that my logs stored on local disk