[
https://issues.apache.org/jira/browse/FLINK-38335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hongshun Wang updated FLINK-38335:
----------------------------------
Description:
SplitEnumeratorContext natively only supports fixed rate scheduling for
asynchronous calls, which can lead to task accumulation if individual calls
take too long to complete.
For example, In KafkaSourceEnumerator, if once
getSubscribedTopicPartitions cost two much time, then a lot of
getSubscribedTopicPartitions will be added into work thread. Then
initializePartitionSplits can only be executed after a long time since this
paririon will be found.
It will case that the assigned and read time of a partition is far lag from
created time.
was:Currently, SplitEnumeratorContext#callAsync only supports fixed rate but
not supports fixed delay. That means if the io call cost two much time, io
thread will will backlog a lot.
> SplitEnumeratorContext supports fixed delay async call.
> -------------------------------------------------------
>
> Key: FLINK-38335
> URL: https://issues.apache.org/jira/browse/FLINK-38335
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
> Affects Versions: 2.1
> Reporter: Hongshun Wang
> Priority: Major
> Fix For: 2.2
>
>
> SplitEnumeratorContext natively only supports fixed rate scheduling for
> asynchronous calls, which can lead to task accumulation if individual calls
> take too long to complete.
>
> For example, In KafkaSourceEnumerator, if once
> getSubscribedTopicPartitions cost two much time, then a lot of
> getSubscribedTopicPartitions will be added into work thread. Then
> initializePartitionSplits can only be executed after a long time since this
> paririon will be found.
> It will case that the assigned and read time of a partition is far lag from
> created time.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
