Martijn Visser created FLINK-40659:
--------------------------------------
Summary: SingleThreadFetcherManager reaps and recreates its
fetcher on every idle gap
Key: FLINK-40659
URL: https://issues.apache.org/jira/browse/FLINK-40659
Project: Flink
Issue Type: Improvement
Components: Connectors / Common
Reporter: Martijn Visser
{{SingleThreadFetcherManager#addSplits}} creates a new {{SplitFetcher}}
whenever the fetcher map is
empty, and {{SourceReaderBase#finishedOrAvailableLater}} calls
{{maybeShutdownFinishedFetchers()}}
every time the element queue drains, which reaps the fetcher as soon as it is
idle. For a source that
finishes a split and then requests the next one, such as the file source with
continuous discovery,
that is one fetcher per split.
Each one costs a {{SplitReader}} construction and close, a pool thread, a
{{FetchTask}}, a
{{CountDownLatch}}, an extra empty batch through the element queue and five
INFO log lines. The job
reported in FLINK-40657 does this about 19 times a second per TaskManager for
17 hours, and there is
no option to hold the fetcher open. It would be better to keep an idle fetcher
for a short while than
to reap it on every gap. FLINK-36146 reports a race in {{getRunningFetcher()}}
caused by the same
cycle.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
