[
https://issues.apache.org/jira/browse/FLINK-4514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15445445#comment-15445445
]
ASF GitHub Bot commented on FLINK-4514:
---------------------------------------
GitHub user tzulitai opened a pull request:
https://github.com/apache/flink/pull/2432
[FLINK-4514][kinesis-connector] Handle unexpected ExpiredIteratorExceptions
Handle any unexpected {{ExpiredIteratorException}}s on {{getRecords()}}
calls be refreshing the failing shard iterator with a new one.
A user reported this issue when replaying Kinesis data over a wide time
span, but then the consumer was back to normal after the consumer caught up
with the latest data. I tried to reproduce the exception, but have come short
to be able to reproduce. The behaviour seems to be inconsistent.
Therefore, this change treats the exception as "unexpected" by simply
catching the exception and refreshing the iterator. There's actually no
guarantee of how much time had passed between each getRecords() request
anyways, so this is a simple way to handle this.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tzulitai/flink FLINK-4514
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2432.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2432
----
commit df833ddbca9971b5f03417efb65527408a8ad9c4
Author: Tzu-Li (Gordon) Tai <[email protected]>
Date: 2016-08-29T09:30:39Z
[FLINK-4514][kinesis-connector] Handle unexpected ExpiredIteratorExceptions
----
> ExpiredIteratorException in Kinesis Consumer on long catch-ups to head of
> stream
> --------------------------------------------------------------------------------
>
> Key: FLINK-4514
> URL: https://issues.apache.org/jira/browse/FLINK-4514
> Project: Flink
> Issue Type: Bug
> Components: Kinesis Connector
> Affects Versions: 1.1.0, 1.1.1
> Reporter: Tzu-Li (Gordon) Tai
> Assignee: Tzu-Li (Gordon) Tai
> Fix For: 1.2.0, 1.1.2
>
>
> Original mailing thread for the reported issue:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kinesis-connector-Iterator-expired-exception-td8711.html
> Normally, the exception is thrown when the consumer uses the same shard
> iterator after 5 minutes since it was retrieved. I've still yet to clarify &
> reproduce the root cause of the {{ExpiredIteratorException}}, because from
> the code this seems to be impossible. I'm leaning towards suspecting this is
> a Kinesis-side issue (from the description in the ML, the behaviour also
> seems indeterminate).
> Either way, the exception can be fairly easily handled so that the consumer
> doesn't just fail. When caught, we request a new shard iterator from Kinesis
> with the last sequence number.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
