[
https://issues.apache.org/jira/browse/FLINK-33260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17776198#comment-17776198
]
Emre Kartoglu edited comment on FLINK-33260 at 10/17/23 5:49 PM:
-----------------------------------------------------------------
There appears to be an existing mechanism to distinguish between retriable
errors and recoverable errors:
[https://github.com/awslabs/amazon-kinesis-connector-flink/blob/master/amazon-kinesis-connector-flink/src/main/java/software/amazon/kinesis/connectors/flink/internals/publisher/fanout/FanOutRecordPublisher.java#L152]
* *Recoverable* errors are retried indefinitely
* *Retriable* errors use the retry configuration (backoff, no of retries etc)
One way to deliver this feature fast/small is to focus on
FanOutRecordPublisher, and have 2 configurations along the lines
{code:java}
custom.recoverable_errors: java.lang.Exception; com.other.CustomException;
custom.retriable_errors: some.AccessDeniedException; some.OtherException
{code}
Where *retriable* errors would make use of the retry configurations currently
present in the codebase, and *recoverable* errors would be retried indefinitely
(structure for this is already in place).
I'm inclined towards this as the structure for it is already in place allowing
faster delivery. And potentially avoiding waste i.e. implementing a more
sophisticated approach (previous comment) that users might not actually need.
I will make an attempt at implementing above, focusing initially on Fan Out.
was (Author: iemre):
There appears to be an existing mechanism to distinguish between retriable
errors and recoverable errors:
[https://github.com/awslabs/amazon-kinesis-connector-flink/blob/master/amazon-kinesis-connector-flink/src/main/java/software/amazon/kinesis/connectors/flink/internals/publisher/fanout/FanOutRecordPublisher.java#L152]
* *Recoverable* errors that are retried indefinitely
* *Retriable* errors use the retry configuration (backoff, no of retries etc)
One way to deliver this feature fast/small is to focus on
FanOutRecordPublisher, and have 2 configurations along the lines
{code:java}
custom.recoverable_errors: java.lang.Exception; com.other.CustomException;
custom.retriable_errors: some.AccessDeniedException; some.OtherException
{code}
Where *retriable* errors would make use of the retry configurations currently
present in the codebase, and *recoverable* errors would be retried indefinitely
(structure for this is already in place).
I'm inclined towards this as the structure for it is already in place allowing
faster delivery. And potentially avoiding waste i.e. implementing a more
sophisticated approach (previous comment) that users might not actually need.
I will make an attempt at implementing above, focusing initially on Fan Out.
> Custom Error Handling for Kinesis Consumer
> ------------------------------------------
>
> Key: FLINK-33260
> URL: https://issues.apache.org/jira/browse/FLINK-33260
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kinesis
> Reporter: Danny Cranmer
> Assignee: Emre Kartoglu
> Priority: Major
> Fix For: aws-connector-4.2.0
>
>
> Background
> The Kinesis Consumer exposes various configuration that allows the user to
> define retry and backoff strategies when dealing with errors. However, the
> configuration does not allow the user to configure which errors are
> retryable, or different strategies for different errors. The error handling
> logic is hard coded within the connector. Over time we discover errors that
> should be retryable that are not, for example KDS throwing 500 on
> SubscribeToShare or transient DNS issues. When these arise the user can
> either fork-fix the connector or log an issue and wait for the next version.
> h3. Scope
> Add the ability for the user to define retry/backoff strategy per error. This
> could be achieved using flexible configuration keys, or allowing the user to
> register their own retry strategies on the connector
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
