Is the exception on the driver or the executor?
To be clear, you're going to see a task fail if a partition changes
leader while the task is running, regardless of configuration
settings. The task should be retried up the maxFailures though.
What are maxRetries and maxFailures set to? How long
Cody,
We already set the maxRetries. We're still seeing issue - when leader is
shifted, for example, it does not appear that direct stream reader
correctly handles this. We're running 1.6.1.
Bryan Jeffrey
On Mon, Jun 13, 2016 at 10:37 AM, Cody Koeninger wrote:
> http://spark.apache.org/docs/
http://spark.apache.org/docs/latest/configuration.html
spark.streaming.kafka.maxRetries
spark.task.maxFailures
On Mon, Jun 13, 2016 at 8:25 AM, Bryan Jeffrey wrote:
> All,
>
> We're running a Spark job that is consuming data from a large Kafka cluster
> using the Direct Stream receiver. We're
