We don¹t see zookeeper expiration. However I noticed that our servers system time is NOT synced. Hence server1 and server2 had 30+sec delay. Do you think that could cause leadership change or any other issue.
On 12/31/14, 4:03 PM, "Jun Rao" <[email protected]> wrote: >A typical cause of frequent leadership changes is GC-induced soft failure. >Do you see ZK session expiration on the broker? If so, you may want to >enable GC log to see the GC time. > >Thanks, > >Jun > >On Tue, Dec 23, 2014 at 2:06 PM, Birla, Lokesh <[email protected]> >wrote: > >> >> I was already using 4GB heap memory. I even changed to 8 GB heap memory >> and could see leadership changing very often. In my 5 minute run, I saw >> leadership changed from 1,2,3 to 3,3,3, to 1,1,1. >> Also my message rate is just: 7k and total msg count is only 2,169,001. >> >> Does anyone has cline on leadership change? >> >> ‹Lokesh >> >> >> >> From: Thunder Stumpges <[email protected]<mailto:[email protected]>> >> Date: Monday, December 22, 2014 at 6:31 PM >> To: "[email protected]<mailto:[email protected]>" < >> [email protected]<mailto:[email protected]>> >> Cc: "Birla, Lokesh" <[email protected]<mailto: >> [email protected]>> >> Subject: RE: Kafka 0.8.1.1 eadership changes are happening very often >> >> Did you check the GC logs in the server? We ran into this and the >>default >> setting of 1G max heap on the broker process was nowhere near enough. We >> currently have set to 4G. >> -T >> >> -----Original Message----- >> From: Birla, Lokesh [[email protected]<mailto: >> [email protected]>] >> Received: Monday, 22 Dec 2014, 5:27PM >> To: [email protected]<mailto:[email protected]> [ >> [email protected]<mailto:[email protected]>] >> CC: Birla, Lokesh [[email protected]<mailto: >> [email protected]>] >> Subject: Kafka 0.8.1.1 eadership changes are happening very often >> >> Hello, >> >> I am running 3 brokers, one zookeeper and producer all on separate >> machine. I am also sending very low load around 6K msg/sec. Each msg is >> around 150 bytes only. >> I ran the load for only 5 minutes and during this time, I see leadership >> chained very often. >> >> I created 3 partitions. >> >> Here leadership for each partitions changed. All 3 brokers are running >> perfectly fine. No broker is down. Could someone let me know why kafka >> leadership changed very often. >> >> Initially: >> >> Topic:mmetopic1PartitionCount:3 ReplicationFactor:3 Configs: >> >> Topic: mmetopic1Partition: 0 Leader: 2Replicas: 2,3,1 Isr: 2,3,1 >> >> Topic: mmetopic1Partition: 1 Leader: 3Replicas: 3,1,2 Isr: 3,1,2 >> >> Topic: mmetopic1Partition: 2 Leader: 1Replicas: 1,2,3 Isr: 1,2,3 >> >> >> Changed to: >> >> >> Topic:mmetopic1PartitionCount:3 ReplicationFactor:3 Configs: >> >> Topic: mmetopic1Partition: 0 Leader: 3Replicas: 2,3,1 Isr: 3,1,2 >> >> Topic: mmetopic1Partition: 1 Leader: 3Replicas: 3,1,2 Isr: 3,1,2 >> >> Topic: mmetopic1Partition: 2 Leader: 1Replicas: 1,2,3 Isr: 1,3,2 >> >> >> Changed to: >> >> >> Topic:mmetopic1PartitionCount:3 ReplicationFactor:3 Configs: >> >> Topic: mmetopic1Partition: 0 Leader: 1Replicas: 2,3,1 Isr: 1,2,3 >> >> Topic: mmetopic1Partition: 1 Leader: 1Replicas: 3,1,2 Isr: 1,2,3 >> >> Topic: mmetopic1Partition: 2 Leader: 2Replicas: 1,2,3 Isr: 2,1,3 >> >> Changed to: >> >> >> Topic:mmetopic1PartitionCount:3 ReplicationFactor:3 Configs: >> >> Topic: mmetopic1Partition: 0 Leader: 3Replicas: 2,3,1 Isr: 3,1,2 >> >> Topic: mmetopic1Partition: 1 Leader: 3Replicas: 3,1,2 Isr: 3,1,2 >> >> Topic: mmetopic1Partition: 2 Leader: 1Replicas: 1,2,3 Isr: 1,3,2 >> >> >> Thanks, >> Lokesh >>
