[
https://issues.apache.org/jira/browse/FLINK-34400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17815245#comment-17815245
]
Rui Fan commented on FLINK-34400:
---------------------------------
{quote}I thought alignment implicitly enabled some form of idleness
{quote}
The idleness is disabled by default, it must be enabled by users. Because user
needs to set how long it mark the source to be idle after no new data is
received.
{quote}Moreover, why would it only stall one of the parallel source streams
instead of both?
{quote}
As you said, one source is always empty. So I think one source doesn't have lag
because it doesn't have data.
Other one has lag because it's waiting for the empty source. So only one source
is blocked.
Please correct my if my understanding is wrong, thanks~
> Kafka sources with watermark alignment sporadically stop consuming
> ------------------------------------------------------------------
>
> Key: FLINK-34400
> URL: https://issues.apache.org/jira/browse/FLINK-34400
> Project: Flink
> Issue Type: Bug
> Affects Versions: 1.18.1
> Reporter: Alexis Sarda-Espinosa
> Priority: Major
> Attachments: alignment_lags.png, logs.txt
>
>
> I have 2 Kafka sources that read from different topics. I have assigned them
> to the same watermark alignment group, and I have _not_ enabled idleness
> explicitly in their watermark strategies. One topic remains pretty much empty
> most of the time, while the other receives a few events per second all the
> time. Parallelism of the active source is 2, for the other one it's 1, and
> checkpoints are once every minute.
> This works correctly for some time (10 - 15 minutes in my case) but then 1 of
> the active sources stops consuming, which causes lag to increase. Weirdly,
> after another 15 minutes or so, all the backlog is consumed at once, and then
> everything stops again.
> I'm attaching some logs from the Task Manager where the issue appears. You
> will notice that the Kafka network client reports disconnections (a long time
> after the deserializer stopped reporting that events were being consumed),
> I'm not sure if this is related.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
