What's the actual stacktrace / exception you're getting related to
commit failure?
On Tue, Sep 27, 2016 at 9:37 AM, Matthias Niehoff
<matthias.nieh...@codecentric.de> wrote:
> Hi everybody,
>
> i am using the new Kafka Receiver for Spark Streaming for my Job. When
> running with old consumer it runs fine.
>
> The Job consumes 3 Topics, saves the data to Cassandra, cogroups the topic,
> calls mapWithState and stores the results in cassandra. After that I
> manually commit the Kafka offsets using the commitAsync method of the
> KafkaDStream.
>
> With the new consumer I experience the following problem:
>
> After a certain amount of time (about 4-5 minutes, might be more or less)
> there are exceptions that the offset commit failed. The processing takes
> less than the batch interval. I also adjusted the session.timeout and
> request.timeout as well as the max.poll.records setting which did not help.
>
> After the first offset commit failed the time it takes from kafka until the
> microbatch is started increases, the processing time is constantly below the
> batch interval. Moreover further offset commits also fail and as result the
> delay time builds up.
>
> Has anybody made this experience as well?
>
> Thank you
>
> Relevant Kafka Parameters:
>
> "session.timeout.ms" -> s"${1 * 60 * 1000}",
> "request.timeout.ms" -> s"${2 * 60 * 1000}",
> "auto.offset.reset" -> "largest",
> "enable.auto.commit" -> "false",
> "max.poll.records" -> "1000"
>
>
>
> --
> Matthias Niehoff | IT-Consultant | Agile Software Factory | Consulting
> codecentric AG | Zeppelinstr 2 | 76185 Karlsruhe | Deutschland
> tel: +49 (0) 721.9595-681 | fax: +49 (0) 721.9595-666 | mobil: +49 (0)
> 172.1702676
> www.codecentric.de | blog.codecentric.de | www.meettheexperts.de |
> www.more4fi.de
>
> Sitz der Gesellschaft: Solingen | HRB 25917| Amtsgericht Wuppertal
> Vorstand: Michael Hochgürtel . Mirko Novakovic . Rainer Vehns
> Aufsichtsrat: Patric Fedlmeier (Vorsitzender) . Klaus Jäger . Jürgen Schütz
>
> Diese E-Mail einschließlich evtl. beigefügter Dateien enthält vertrauliche
> und/oder rechtlich geschützte Informationen. Wenn Sie nicht der richtige
> Adressat sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie
> bitte sofort den Absender und löschen Sie diese E-Mail und evtl. beigefügter
> Dateien umgehend. Das unerlaubte Kopieren, Nutzen oder Öffnen evtl.
> beigefügter Dateien sowie die unbefugte Weitergabe dieser E-Mail ist nicht
> gestattet
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
