Alexis Sarda-Espinosa created FLINK-34400:
---------------------------------------------
Summary: Kafka sources with watermark alignment sporadically stop
consuming
Key: FLINK-34400
URL: https://issues.apache.org/jira/browse/FLINK-34400
Project: Flink
Issue Type: Bug
Affects Versions: 1.18.1
Reporter: Alexis Sarda-Espinosa
Attachments: logs.txt
I have 2 Kafka sources that read from different topics. I have assigned them to
the same watermark alignment group, and I have _not_ enabled idleness
explicitly in their watermark strategies. One topic remains pretty much empty
most of the time, while the other receives a few events per second all the
time. Parallelism of the active source is 2.
This works correctly for some time (10 - 15 minutes in my case) but then 1 of
the active sources stops consuming, which causes lag to increase. Weirdly,
after another 15 minutes or so, all the backlog is consumed at once, and then
everything stops again.
I'm attaching some logs from the Task Manager where the issue appears. You will
notice that the Kafka network client reports disconnections - this is because
my Kafka cluster was indeed restarted (one broker at a time), I'm not sure if
this is related.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
