Re: GC problem on broker startup under minimal load

2015-03-30 Thread Timo Ahokas
Sorry - some final details on this for today. I stopped the broker being added to the cluster (the one with GC taking 1min & both cores), dropped the max heap to 1GB (out of 4GB) and started it up again. It joined nicely, caught up and went ISR. Haven't yet run into the increasing GC cycles (which

Re: GC problem on broker startup under minimal load

2015-03-30 Thread Timo Ahokas
A few more details before the looming crash. The broker replicating data and in sync (before GC cycle starts) - GC cycles taking nearly 1min real: 2015-03-30T21:08:39.502+0300: 3805.931: [GC2015-03-30T21:08:39.502+0300: 3805.932: [ParNew: 142199K->6219K(153344K), 51.6581120 secs] 678471K->543078K(

Re: GC problem on broker startup under minimal load

2015-03-30 Thread Timo Ahokas
Hi, A related issue which results in a similar increasing GC cycles (but doesn't recover)... We're trying to bootstrap a new broker into the cluster (replacing one of the 3 main ones described previously). It connects fine, starts replicating data and becomes ISR, but starts a similar GC cycle on

GC problem on broker startup under minimal load

2015-03-26 Thread Timo Ahokas
Hi, We are seeing really weird GC behaviour on our Kafka cluster. After a clean stop/start of the whole cluster, the servers go into a loop of increasingly slow GC cycles for an hour and then seem to recover to a normal state without any external changes to the servers or traffic. The load to the