You are affected by this 0.9.0.0 bug https://issues.apache.org/jira/browse/KAFKA-2988
It was fixed for 0.9.0.1. You could just apply same fix to your 0.9.0.0 cluster but I'd recommend upgrading to 0.9.0.1. Kind regards, Stevo Slavic. On Mon, Mar 14, 2016, 07:10 Kunal Gupta <kunal.gu...@cube26.com> wrote: > Hi everyone, > > I am new here, recently join the group. I faced a problem in Kafka Cluster, > a problem is described below. > > I am using Kafka version 0.9.0.0 > > We have established a Kafka Cluster of 3 machines where 2 machines are > utilized for Kafka broker and same 3 machines utilized for zookeeper. When > we start our consumer written in Kafka-Python, Kafka brokers create a new > topic name __consumer_offsets with 50 partitions and spread over 100 % in > Kafka broker with replica 2. > > I know why topic is get created it is because offsets are stored in Kafka > broker instead of zookeeper but problem is each partition of > __consumer_offsets take a lot of space as compared to topic data and > yesterday I have restarted my cluster gain due to lack of space. > > Can you exactly help me why __consumer_offsets are taking a lot of space? > and how would I get rid of this. > > Topic data space is 2 GB and while __consumer_offsets are taking 92 GB. My > server size is 98 GB with 4 GB RAM. > > > > > *Thanks, Kunal* > *+91-9958189589* > *Data Analyst* > *First Paper Publication : **http://dl.acm.org/citation.cfm?id=2790798 > <http://dl.acm.org/citation.cfm?id=2790798> * > *Blog:- **http://learnhardwithkunalgupta.blogspot.in > <http://learnhardwithkunalgupta.blogspot.in> * >
