Yash Sharma created SPARK-20140:
-----------------------------------
Summary: Remove hardcoded kinesis retry wait and max retries
Key: SPARK-20140
URL: https://issues.apache.org/jira/browse/SPARK-20140
Project: Spark
Issue Type: Bug
Components: DStreams
Affects Versions: 2.1.0
Reporter: Yash Sharma
The pull requests proposes to remove the hardcoded values for Amazon Kinesis -
MIN_RETRY_WAIT_TIME_MS, MAX_RETRIES.
This change is critical for kinesis checkpoint recovery when the kinesis backed
rdd is huge.
Following happens in a typical kinesis recovery :
- kinesis throttles large number of requests while recovering
- retries in case of throttling are not able to recover due to the small wait
period
- kinesis throttles per second, the wait period should be configurable for
recovery
The patch picks the spark kinesis configs from:
- spark.streaming.kinesis.retry.wait.time
- spark.streaming.kinesis.retry.max.attempts
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
