Hey Gligor, Sorry for the rough edges. I think there are a variety of rough edges in error messages here we can improve:
1. "Error ILLEGAL_GENERATION occurred while committing offsets for group MetadataConsumerSpout" is obviously NOT the most intuitive error message, it doesn't really explain what happened or what to do to fix it. We should improve that. 2. The error when you exceed your max session timeout doesn't tell you that there is a config on the broker that controls this. There isn't an easy way to figure this out on your own. 3. The default max on the broker should probably be higher? 4. The docs on group.max.session.timeout.ms should explain why this exists, why you'd want to change it, etc. 5. We are working on a way to control the maximum number of messages per poll request which will help reduce the session timeout you need, but I think that is orthogonal to the confusion. Thanks for walking through the steps you had to go through in figuring it out, that will help us to round off some of the corners. -Jay On Wed, Mar 2, 2016 at 6:09 AM, Gligor Vanessa <gligor.vane...@gmail.com> wrote: > Hello, > > I am using Kafka higher consumer 0.9.0. I am not using the auto commit for > the offsets, so after I consume the messaged (poll from Kafka) I will have > to commit the offsets manually. > > The issue that I have is actually that the processing of the messages takes > longer than 30s (and I cannot call poll again, before these messages are > processed) and when I try to commit the offset a exception is thrown: > ERROR o.a.k.c.c.i.ConsumerCoordinator - Error ILLEGAL_GENERATION occurred > while committing offsets for group MetadataConsumerSpout. > (I have found on stackoverflow this explanation: so if you wait for longer > that the timeout request then the coordinator for the topic will kickout > the consumer because it will think is dead and it will rebalance the group) > > In order to get rid of this I have thought about a couple of solutions: > > 1. The configuration session.timeout.ms has a maximum value, so if I try > to > set it to 60 seconds, also I get an exception, because this value is not in > the valid interval. > > 2. I have tried to find a solution to get a paginated request when the > polling method is called - no success. > > 3. I have tried to send a heart beat from the outside of the poll (because > this method sends the heartbeats) - no success. > > > Thank you. > > Vanessa. >
