Hi Seth, I think the global watermark / mark as idle is one of the reasons why the FLIP-27 [1] interfaces were created, while FLIP-180 [2] is also relevant here. Unfortunately, the FlinkKinesisConsumer is still using the old interface. I'm not sure it can be properly fixed with the old interfaces. There are tickets for migrating the Kinesis source to new interfaces [3] [4] but I don't think they are planned yet.
I believe that the only other source that used global watermark / idleness in Flink was the FlinkKafkaConsumer, which has been deprecated already in favour of the KafkaSource. You could consider having a look at how the handling was done for the FlinkKafkaConsumer to see if that can also be applied for FlinkKinesisConsumer. Best regards, Martijn [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-180%3A+Adjust+StreamStatus+and+Idleness+definition [3] https://issues.apache.org/jira/browse/FLINK-24438 [4] https://issues.apache.org/jira/browse/FLINK-25729 Op ma 29 aug. 2022 om 23:25 schreef Seth Saperstein via user < u...@flink.apache.org>: > Hi > > I wanted to bring awareness to this Jira > <https://issues.apache.org/jira/browse/FLINK-29099> describing a deadlock > state we've experienced for a single subtask in the FlinkKinesisConsumer. > > This occurs when we've reached the following conditions in the subtask: > - reached the max lookahead so the RecordEmitter does not emit > - are marked idle and the global watermark does not refresh > > More details are in the ticket, however, we intend to resolve this by > updating the global watermark for idle subtasks, as noted here > <https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcher.java#L1268>. > This will allow us to emit records again in the RecordEmitter and bring us > out of the idle state. > > I'm curious to hear if anyone has run into similar issues or sees getting > the global watermark when idle as problematic. We plan to test an > implementation of this internally in the coming weeks. > > Thanks, > Seth >
