We changed the policy to "delete" dynamically for the __consumer_offsets topic and it was a better option than doing a cluster restart after enabling log compaction. Also, we found problems when you are replicating to a log compacted topic from a non-compacted topic (which is leader).
On Mon, Mar 14, 2016 at 12:07 PM, Kunal Gupta <kunal.gu...@cube26.com> wrote: > Thanks @Stevo Slavić > > > *Thanks, Kunal* > *+91-9958189589* > *Data Analyst* > *First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798 > <http://dl.acm.org/citation.cfm?id=2790798> * > *Blog:- **http://learnhardwithkunalgupta.blogspot.in > <http://learnhardwithkunalgupta.blogspot.in> * > > > > > > On Mon, Mar 14, 2016 at 11:57 AM, Stevo Slavić <ssla...@gmail.com> wrote: > > > You are affected by this 0.9.0.0 bug > > https://issues.apache.org/jira/browse/KAFKA-2988 > > > > It was fixed for 0.9.0.1. > > > > You could just apply same fix to your 0.9.0.0 cluster but I'd recommend > > upgrading to 0.9.0.1. > > > > Kind regards, > > Stevo Slavic. > > On Mon, Mar 14, 2016, 07:10 Kunal Gupta <kunal.gu...@cube26.com> wrote: > > > > > Hi everyone, > > > > > > I am new here, recently join the group. I faced a problem in Kafka > > Cluster, > > > a problem is described below. > > > > > > I am using Kafka version 0.9.0.0 > > > > > > We have established a Kafka Cluster of 3 machines where 2 machines are > > > utilized for Kafka broker and same 3 machines utilized for zookeeper. > > When > > > we start our consumer written in Kafka-Python, Kafka brokers create a > new > > > topic name __consumer_offsets with 50 partitions and spread over 100 % > in > > > Kafka broker with replica 2. > > > > > > I know why topic is get created it is because offsets are stored in > Kafka > > > broker instead of zookeeper but problem is each partition of > > > __consumer_offsets take a lot of space as compared to topic data and > > > yesterday I have restarted my cluster gain due to lack of space. > > > > > > Can you exactly help me why __consumer_offsets are taking a lot of > > space? > > > and how would I get rid of this. > > > > > > Topic data space is 2 GB and while __consumer_offsets are taking 92 GB. > > My > > > server size is 98 GB with 4 GB RAM. > > > > > > > > > > > > > > > *Thanks, Kunal* > > > *+91-9958189589* > > > *Data Analyst* > > > *First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798 > > > <http://dl.acm.org/citation.cfm?id=2790798> * > > > *Blog:- **http://learnhardwithkunalgupta.blogspot.in > > > <http://learnhardwithkunalgupta.blogspot.in> * > > > > > > -- Regards Vamsi Subhash
