Tzu-Li (Gordon) Tai created FLINK-11164:
-------------------------------------------
Summary: Refreshing expired shard iterator fails if no records
were retrieved from shard since ShardConsumer was started in
FlinkKinesisConsumer
Key: FLINK-11164
URL: https://issues.apache.org/jira/browse/FLINK-11164
Project: Flink
Issue Type: Bug
Components: Kinesis Connector
Reporter: Tzu-Li (Gordon) Tai
Originally reported by:
[http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Encountered-the-following-Expired-Iterator-exception-in-getRecords-using-FlinkKinesisConsumer-td25093.html]
In {{ShardConsumer.getRecords(...)}}, we catch any expired iterator exceptions
so that in the case no data was written to the Kinesis shard, we last iterator
which eventually expires doesn't just fail the job, by doing:
`
shardItr = kinesis.getShardIterator(subscribedShard,
ShardIteratorType.AFTER_SEQUENCE_NUMBER.toString(),
lastSequenceNum.getSequenceNumber());
`.
The problem is that, if absolutely no records were retrieved at all from the
shard since the {{ShardConsumer}} was started, then {{lastSequenceNumber}}
would still be a sentinel value (e.g. `EARLIEST_SEQUENCE_NUMBER`,
`LATEST_SEQUENCE_NUMBER`, etc.) instead of an actual value. This isn't
recognized by Kinesis, and would fail the job.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
