Ritika Muduganti created KAFKA-15079:
----------------------------------------
Summary: Offset expirations in topics with less event generation
results in messages that take months to be delivered.
Key: KAFKA-15079
URL: https://issues.apache.org/jira/browse/KAFKA-15079
Project: Kafka
Issue Type: Bug
Components: group-coordinator
Reporter: Ritika Muduganti
It was observed that messages were taking months to be delivered in certain
pipelines. The offsets had been reset and consumption restarted from messages
that were a couple months old.
On further investigation the team found that once the offsets.retention.minutes
(7 days) elapses and there's no subscribers to the topic, the offsets are
subject to immediate expiration. This can be a problem when we close the
current consumer and the retention period has already passed due to the
infrequent nature of offset commits. The offsets then expire before a new
consumer can come up and subscribe to the topic again. Since only one consumer
could be subscribed to the topic it will be removed from the subscription list
as soon as the consumer leaves.
Two things to keep in mind here are :-
# The consumer group has different consumers subscribed to different topics.
# Events aren't generated very frequently to these topics.
[https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/coordinator/group/GroupMetadata.scala#LL766C1-L825C4]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
