[
https://issues.apache.org/jira/browse/SPARK-20140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-20140:
------------------------------------
Assignee: Apache Spark
> Remove hardcoded kinesis retry wait and max retries
> ---------------------------------------------------
>
> Key: SPARK-20140
> URL: https://issues.apache.org/jira/browse/SPARK-20140
> Project: Spark
> Issue Type: Bug
> Components: DStreams
> Affects Versions: 2.1.0
> Reporter: Yash Sharma
> Assignee: Apache Spark
> Labels: kinesis, recovery
>
> The pull requests proposes to remove the hardcoded values for Amazon Kinesis
> - MIN_RETRY_WAIT_TIME_MS, MAX_RETRIES.
> This change is critical for kinesis checkpoint recovery when the kinesis
> backed rdd is huge.
> Following happens in a typical kinesis recovery :
> - kinesis throttles large number of requests while recovering
> - retries in case of throttling are not able to recover due to the small wait
> period
> - kinesis throttles per second, the wait period should be configurable for
> recovery
> The patch picks the spark kinesis configs from:
> - spark.streaming.kinesis.retry.wait.time
> - spark.streaming.kinesis.retry.max.attempts
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
