Hi, My 3 node cluster (2x4 core xeon, 8GB ram per machine, striped HDD) had crashed with OOM when I had some 5-6 topics with 256 partitions per topic. dont remember the heap size I used, I think it was the default one that is there in the 0.8.2.1 bundle
I have been trying to come up with the numbers around this, as to how many topics and partitions can a cluster of n nodes support. I have used the producer performance script to pump data into the cluster. But, that script has some issues and some parameters that are passed form command line are not exactly accepted. I'll try and take out some time on it and give out the numbers Regards, Prabhjot On Thu, Nov 5, 2015 at 2:58 PM, Muqtafi Akhmad <muqt...@traveloka.com> wrote: > Hello Prabhjot, > > Actually what I meant in previous email is that there are 200 topics, with > 1 partition each so there are 200 total partitions. Is there any rule of > thumb regarding this matter? Can you share your configuration (including > spec, jvm memory, etc) for kafka cluster? > > Thank you, > > On Thu, Nov 5, 2015 at 4:07 PM, Prabhjot Bharaj <prabhbha...@gmail.com> > wrote: > > > Hi, > > > > Not sure. But, I had hit OOM when using too many partitions and many > topics > > with a lesser heap size assigned to the jvm > > > > Regards, > > Prabhjot > > > > On Thu, Nov 5, 2015 at 10:40 AM, Muqtafi Akhmad <muqt...@traveloka.com> > > wrote: > > > > > Hello all, > > > Recently I got incident with kafka cluster, I found OutOfMemoryError in > > > kafka server log > > > > > > > WARN [ReplicaFetcherThread-0-2], Error in fetch Name: FetchRequest; > > > > Version: 0; CorrelationId: 7041716; ClientId: > ReplicaFetcherThread-0-2; > > > > ReplicaId: 0; MaxWait: 500 ms; MinBytes: 1 bytes; RequestInfo:[. . . > . > > > .]. > > > > Possible cause: java.lang.OutOfMemoryError: Java heap space > > > > (kafka.server.ReplicaFetcherThread) > > > > > > And at the same time there are socket timeout exceptions in kafka > > consumer. > > > Is there any hypothesis about what might be wrong? > > > > > > Here is the cluster spec : > > > 3 hosts, @ with memory 16 G > > > partitions : ~200 > > > replication factor : 3 > > > rate in : 300 K/s > > > rate out : 800 K/s > > > > > > Thank you, > > > > > > -- > > > Muqtafi Akhmad > > > Software Engineer > > > Traveloka > > > > > > > > > > > -- > > --------------------------------------------------------- > > "There are only 10 types of people in the world: Those who understand > > binary, and those who don't" > > > > > > -- > Muqtafi Akhmad > Software Engineer > Traveloka > -- --------------------------------------------------------- "There are only 10 types of people in the world: Those who understand binary, and those who don't"
