[
https://issues.apache.org/jira/browse/FLINK-31006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser updated FLINK-31006:
-----------------------------------
Release Note:
A bounded Kafka source now finishes after the job has been restored from a
savepoint or a retained checkpoint, or after a JobManager failover. Previously
the restored readers waited indefinitely for a no-more-splits signal that was
never sent, because the recreated enumerator found every partition already
assigned and skipped the step that marks the discovery as finished.
A bounded source whose topic pattern currently matches no partition now
finishes immediately instead of hanging. A source configured with an explicit
topic list still fails when the topic does not exist, which is unchanged.
> job is not finished when using pipeline mode to run bounded source like kafka
> -----------------------------------------------------------------------------
>
> Key: FLINK-31006
> URL: https://issues.apache.org/jira/browse/FLINK-31006
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
> Affects Versions: 2.1.0
> Reporter: Jacky Lau
> Assignee: Martijn Visser
> Priority: Major
> Labels: pull-request-available
> Fix For: kafka-5.1.0, kafka-4.0.2, kafka-5.0.1, kafka-3.4.1
>
> Attachments: image-2023-02-10-13-20-52-890.png,
> image-2023-02-10-13-23-38-430.png, image-2023-02-10-13-24-46-929.png,
> image-2023-03-04-01-04-18-658.png, image-2023-03-04-01-05-25-335.png,
> image-2023-03-04-01-07-04-927.png, image-2023-03-04-01-07-36-168.png,
> image-2023-03-04-01-08-29-042.png, image-2023-03-04-01-09-24-199.png
>
>
> when i do failover works like kill jm/tm when using pipeline mode to run
> bounded source like kafka, i found job is not finished, when every partition
> data has consumed.
>
> After dig into code, i found this logical not run when JM recover. the
> partition infos are not changed. so noMoreNewPartitionSplits is not set to
> true. then this will not run
>
> !image-2023-02-10-13-23-38-430.png!
>
> !image-2023-02-10-13-24-46-929.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
