[
https://issues.apache.org/jira/browse/SPARK-11613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heji Kim closed SPARK-11613.
----------------------------
Resolution: Fixed
> Kinesis ASL should allow caller to set ClientConfiguration for socket
> timeouts and other connection setting
> -----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-11613
> URL: https://issues.apache.org/jira/browse/SPARK-11613
> Project: Spark
> Issue Type: Improvement
> Components: DStreams
> Affects Versions: 1.5.1
> Reporter: Heji Kim
>
> Currently the Kinesis ASL does not allow configuration of Kinesis settings
> such as maxRecords fetched per getRecords or the connection settings on
> AmazonKinesisClient such as the socket timeout.
> Currently we get intermittent multiple SocketTimeoutExceptions on restart
> from Kinesis which holds up the driver processing(2-10 minutes) and introduce
> a pileup. After we engaged AWS support, they asked us to lower the
> maxRecords per getRecords to 1,000- current default is 10,000. There appeared
> no way to configure this externally so I made a custom build and this
> resolved some of our issues.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
