MartijnVisser commented on PR #309: URL: https://github.com/apache/flink-connector-kafka/pull/309#issuecomment-5618046511
Correcting my own nit on `readersSignalledNoMoreSplits`: I said clearing the set when sub-enumerators are recreated was fine as a follow-up. It is not, once FLINK-31006 lands. The chain is #291 plus this set. #291 deliberately replays no-more-splits only on the reader's first metadata update and leaves a later metadata change to the enumerator signalling again, which is the re-signal FLINK-31006 restores. On that metadata change `onHandleSubscribedStreamsFetch` recreates every sub-enumerator, each one signals, but the readers never re-registered, so the set still holds them, `readersToSignal` is empty and `handleNoMoreSplits` returns without signalling. `DynamicKafkaSourceReader` has meanwhile closed and recreated every sub-reader, each back to `noMoreSplitsAssignment == false`. A bounded job hangs after a metadata change. The set is keyed on the reader alone; it needs the enumerator generation as well, or clearing wherever the sub-enumerators are recreated. Please take it in this PR rather than as a follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
