Well, I made the problem go away, but I'm not sure why it works :-/ Previously I used a time out value of 100 for Consumer.poll(). Increasing it to 10.000 makes the problem go away completely?! I tried other values as well: - 0 problem remained - 3000, same as heartbeat.interval, problem remained, but less frequent
Not really sure what is going on, but happy that the problem went away :-) Martin On 30 November 2015 at 15:33, Martin Skøtt <martin.sko...@falconsocial.com> wrote: > Hi Guozhang, > > I have done some testing with various values of heartbeat.interval.ms and > they don't seem to have any influence on the error messages. Running > kafka-consumer-groups also continues to return that the consumer groups > does not exists or is rebalancing. Do you have any suggestions to how I > could debug this further? > > Regards, > Martin > > > On 25 November 2015 at 18:37, Guozhang Wang <wangg...@gmail.com> wrote: > >> Hello Martin, >> >> It seems your consumer's heartbeat.interval.ms config value is too small >> (default is 3 seconds) for your environment, consider increasing it and >> see >> if this issue goes away. >> >> At the same time, we have some better error handling fixes in trunk which >> will be included in the next point release. >> >> https://issues.apache.org/jira/browse/KAFKA-2860 >> >> Guozhang >> >> >> >> On Wed, Nov 25, 2015 at 6:54 AM, Martin Skøtt < >> martin.sko...@falconsocial.com> wrote: >> >> > Hi, >> > >> > I'm experiencing some very strange issues with 0.9. I get these log >> > messages from the new consumer: >> > >> > [main] ERROR >> > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Error >> > ILLEGAL_GENERATION occurred while committing offsets for group >> > aaa-bbb-reader >> > [main] WARN >> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator >> > - Auto offset commit failed: Commit cannot be completed due to group >> > rebalance >> > [main] ERROR >> > org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Error >> > ILLEGAL_GENERATION occurred while committing offsets for group >> > aaa-bbb-reader >> > [main] WARN >> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator >> > - Auto offset commit failed: >> > [main] INFO >> org.apache.kafka.clients.consumer.internals.AbstractCoordinator >> > - Attempt to join group aaa-bbb-reader failed due to unknown member id, >> > resetting and retrying. >> > >> > And this in the broker log: >> > [2015-11-25 15:41:01,542] INFO [GroupCoordinator 0]: Preparing to >> > restabilize group aaa-bbb-reader with old generation 1 >> > (kafka.coordinator.GroupCoordinator) >> > [2015-11-25 15:41:01,544] INFO [GroupCoordinator 0]: >> > Group aaa-bbb-reader generation 1 is dead and removed >> > (kafka.coordinator.GroupCoordinator) >> > [2015-11-25 15:41:13,474] INFO [GroupCoordinator 0]: Preparing to >> > restabilize group aaa-bbb-reader with old generation 0 >> > (kafka.coordinator.GroupCoordinator) >> > [2015-11-25 15:41:13,475] INFO [GroupCoordinator 0]: Stabilized >> > group aaa-bbb-reader generation 1 (kafka.coordinator.GroupCoordinator) >> > [2015-11-25 15:41:13,477] INFO [GroupCoordinator 0]: Assignment received >> > from leader for group aaa-bbb-reader for generation 1 >> > (kafka.coordinator.GroupCoordinator) >> > [2015-11-25 15:41:43,478] INFO [GroupCoordinator 0]: Preparing to >> > restabilize group aaa-bbb-reader with old generation 1 >> > (kafka.coordinator.GroupCoordinator) >> > [2015-11-25 15:41:43,478] INFO [GroupCoordinator 0]: >> > Group aaa-bbb-reader generation 1 is dead and removed >> > (kafka.coordinator.GroupCoordinator) >> > >> > When this happens the kafka-consumer-groups describe command keeps >> saying >> > that the group no longer exists or is rebalancing. What is probably even >> > worse is that my consumers appears to be looping constantly through >> > everything written to the topics!? >> > >> > Does anyone have any input on what might be happening? >> > >> > I'm running 0.9 locally on my laptop using one Zookeeper and one broker, >> > both using the configuration provided in the distribution. I have 13 >> topics >> > with two partitions each and a replication factor of 1. I run one >> producer >> > and once consumer also on the same machine. >> > >> > -- >> > Martin Skøtt >> > >> >> >> >> -- >> -- Guozhang >> > >
