We had a consumer crash and restart itself a bunch of times, and then the group got stuck in state CompletingRebalance. All of the consumers were dead (I checked that there are no processes running on the host in the output of kafka-consumer-groups.sh --describe --group group-name --members), but the group was still rebalancing. If I tried to start up a consumer, it just timed out trying to join the group.
I tried to delete the group, but that fails with * Group 'trackers-etl' could not be deleted due to: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.GroupNotEmptyException: The group is not empty. I ended up restarting the broker that was the leader for that group; that fixed the issue but is obviously a pretty heavy-weight solution. There was nothing interesting in the logs on that broker; the last thing it logged about that group was: [2020-04-13 21:37:20,308] INFO [GroupCoordinator 6]: Stabilized group trackers-etl generation 4987 (__consumer_offsets-29) (kafka.coordinator.group.GroupCoordinator) Has anyone seen this before? Should I file a JIRA ticket? Was there a better process than restarting the broker? -- James Brown Systems Engineer
