You can try running the driver in the cluster manager with --supervise, but
that's basically the same as restarting it when it fails.
There is no reasonable automatic "recovery" when something is fundamentally
wrong with your kafka cluster.
On Wed, Oct 21, 2015 at 12:46 AM, swetha kasireddy <
swe
Hi Cody,
What other options do I have other than monitoring and restarting the job?
Can the job recover automatically?
Thanks,
Sweth
On Thu, Oct 1, 2015 at 7:18 AM, Cody Koeninger wrote:
> Did you check you kafka broker logs to see what was going on during that
> time?
>
> The direct stream wi
: swetha
Cc: "user@spark.apache.org<mailto:user@spark.apache.org>"
Subject: Re: Lost leader exception in Kafka Direct for Streaming
Did you check you kafka broker logs to see what was going on during that time?
The direct stream will handle normal leader loss / rebalance by retrying
Did you check you kafka broker logs to see what was going on during that
time?
The direct stream will handle normal leader loss / rebalance by retrying
tasks.
But the exception you got indicates that something with kafka was wrong,
such that offsets were being re-used.
ie. your job already proce
