[
https://issues.apache.org/jira/browse/KAFKA-16622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842119#comment-17842119
]
Greg Harris commented on KAFKA-16622:
-------------------------------------
To clarify your specific points:
> but the problem here is that if the consumer never fully catches up once, we
> will never have a checkpoint.
If the consumer never catches up later than the most recent MM2 checkpoint task
restart, it will not have a checkpoint. In the above example, it needed to get
past 9999.
> If as initial state the
>{color:#000000}OffsetSyncStore.{color}{color:#871094}offsetSyncs
>{color}contained a distribution of {color:#000000}OffsetSync rather than just
>multiple copies of the last {color}{color:#000000}OffsetSync , Checkpoints
>would be computed earlier I think{color}
{color:#000000}This is the effect a solution to KAFKA-15905 would have. If we
can restore the state of the `checkpointsPerConsumerGroup` variable, then it
will be safe to keep those offset syncs in the sync store.{color}
> Mirromaker2 first Checkpoint not emitted until consumer group fully catches
> up once
> -----------------------------------------------------------------------------------
>
> Key: KAFKA-16622
> URL: https://issues.apache.org/jira/browse/KAFKA-16622
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
> Affects Versions: 3.7.0, 3.6.2, 3.8.0
> Reporter: Edoardo Comar
> Priority: Major
> Attachments: connect.log.2024-04-26-10.zip,
> edo-connect-mirror-maker-sourcetarget.properties
>
>
> We observed an excessively delayed emission of the MM2 Checkpoint record.
> It only gets created when the source consumer reaches the end of a topic.
> This does not seem reasonable.
> In a very simple setup :
> Tested with a standalone single process MirrorMaker2 mirroring between two
> single-node kafka clusters(mirromaker config attached) with quick refresh
> intervals (eg 5 sec) and a small offset.lag.max (eg 10)
> create a single topic in the source cluster
> produce data to it (e.g. 10000 records)
> start a slow consumer - e.g. fetching 50records/poll and pausing 1 sec
> between polls which commits after each poll
> watch the Checkpoint topic in the target cluster
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9192 \
> --topic source.checkpoints.internal \
> --formatter org.apache.kafka.connect.mirror.formatters.CheckpointFormatter \
> --from-beginning
> -> no record appears in the checkpoint topic until the consumer reaches the
> end of the topic (ie its consumer group lag gets down to 0).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
