[
https://issues.apache.org/jira/browse/FLINK-28838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17577422#comment-17577422
]
Aitozi edited comment on FLINK-28838 at 8/9/22 3:47 PM:
--------------------------------------------------------
[~renqs] Thanks for your inputs. Actually, it's observed in our internal source
connector, I think the blocking mode do not conflict with the cool down
mechanism, because if kafka block until there is data then it will never have
the chance to trigger the SleepTask I think.
Besides, I lean to provide an inner cool down manager which will be disabled by
default, and if the {{fetch}} is non-blocking mode, user can enable it by
setting a suitable threshold and sleep interval.
Otherwise, the current work flow will rely on a blocking {{fetch}} to avoid the
busy loop in this case.
was (Author: aitozi):
[~renqs]Thanks for your inputs. Actually, it's observed in our internal source
connector, I think the blocking mode do not conflict with the cool down
mechanism. We can provide an inner cool down manager which can be disabled by
default, and if the {{fetch}} is non-blocking mode, it can enable it by setting
a suitable threshold and sleep interval.
Otherwise, the current work flow will rely on a blocking {{fetch}} to avoid the
busy loop in this case.
> Avoid to notify the elementQueue consumer when the fetch result is empty
> ------------------------------------------------------------------------
>
> Key: FLINK-28838
> URL: https://issues.apache.org/jira/browse/FLINK-28838
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
> Affects Versions: 1.15.0, 1.15.1
> Reporter: Aitozi
> Priority: Major
> Fix For: 1.16.0
>
> Attachments: 20220805165441.jpg
>
>
> When using the new source api, I found that if the source has no data, it
> still brings high cpu usage.
> The reason behind this is that it will always return the
> {{RecordsWithSplitIds}} from the {{splitReader.fetch}} in FetchTask and it
> will be added to the elementQueue. It will make the consumer be notified to
> wake up frequently.
> This causes the thread to keep busy to run and wake up, which leads to the
> high sys and user cpu usage.
> I think not all the SplitReader#fetch will block until there is data, if it
> returns immediately when there is no data, then this problem will happen
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
