Hi I wanted to bring awareness to this Jira <https://issues.apache.org/jira/browse/FLINK-29099> describing a deadlock state we've experienced for a single subtask in the FlinkKinesisConsumer.
This occurs when we've reached the following conditions in the subtask: - reached the max lookahead so the RecordEmitter does not emit - are marked idle and the global watermark does not refresh More details are in the ticket, however, we intend to resolve this by updating the global watermark for idle subtasks, as noted here <https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/KinesisDataFetcher.java#L1268>. This will allow us to emit records again in the RecordEmitter and bring us out of the idle state. I'm curious to hear if anyone has run into similar issues or sees getting the global watermark when idle as problematic. We plan to test an implementation of this internally in the coming weeks. Thanks, Seth
