Hi Everyone, I am facing some issues with new Kafka consumer API. My consumer polls and processes records in same thread. I commit explicitly through commitSync(ConsumerRecord) method. i have seen following issues
1) Sometimes when processing time is too high i see error "UNKNOWN_MEMBER_ID" on commit and also see "REBALANCE_IN_PROGRESS" error sometimes. 2) Above problem suggested that probably session is timing out which i didn't expected as hearbeat task should still be executing time to time. But to ensure that session doesn't tiomeout I started commiting processed offsets after 75% time of session time is elapsed and above issue was resolved. Is this a normal practice with new consumer API? 3) What is significance of consumer config "metadata.max.age.ms"? I see that if i am processing records after a processing and a metadata request is issued by NetworkClient then i start getting error "ILLEGAL_GENERATION" on each commit. 4) What should a slow consumer do in case of a rebalance? if a consumer is processing and committing and meanwhile a rebalance happen then will this not cause any issue like "UNKNOWN_MEMBER_ID" or "REBALANCE_IN_PROGRESS"? what is ideal way to deal with this? Any pointers will be much appreciated. Regards, Vinay Sharma
