The direct stream shouldn't silently lose data in the case of a leader loss. Loss of a leader is handled like any other failure, retrying up to spark.task.maxFailures times.
But really if you're losing leaders and taking that long to rebalance.... you should figure out what's wrong with your kafka cluster and fix it, regardless of what consumer you're using. On Tue, Nov 24, 2015 at 10:55 PM, SRK <swethakasire...@gmail.com> wrote: > Hi, > > Does receiver based approach lose any data in case of a leader/broker loss > in Spark Streaming? We currently use Kafka Direct for Spark Streaming and > it > seems to be failing out when there is a leader loss and we can't really > guarantee that there won't be any leader loss due rebalancing. > > If we go with receiver based approach, would it be able to overcome that > situation? > > > Thanks, > Swetha > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Does-receiver-based-approach-lose-any-data-in-case-of-a-leader-broker-loss-in-Spark-Streaming-tp25470.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
