Kafka Clients are blocking spark streaming jobs and after a time streaming job queue increases.
-----Original Message----- From: Cody Koeninger [mailto:c...@koeninger.org] Sent: Tuesday, December 26, 2017 6:47 PM To: Diogo Munaro Vieira <diogo.mun...@corp.globo.com> Cc: Serkan TAS <serkan....@enerjisa.com>; user <user@spark.apache.org> Subject: Re: Which kafka client to use with spark streaming Do not add a dependency on kafka-clients, the spark-streaming-kafka library has appropriate transitive dependencies. Either version of the spark-streaming-kafka library should work with 1.0 brokers; what problems were you having? On Mon, Dec 25, 2017 at 7:58 PM, Diogo Munaro Vieira <diogo.mun...@corp.globo.com> wrote: > Hey Serkan, it depends of your Kafka version... Is it 0.8.2? > > Em 25 de dez de 2017 06:17, "Serkan TAS" <serkan....@enerjisa.com> escreveu: >> >> Hi, >> >> >> >> Working on spark 2.2.0 cluster and 1.0 kafka brokers. >> >> >> >> I was using the library >> >> "org.apache.spark" % "spark-streaming-kafka-0-10_2.11" % "2.2.0" >> >> >> >> and had lots of problems during streaming process then downgraded to >> >> "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" >> % "2.2.0" >> >> >> >> And i know there is also another path which is using kafka-clients >> jars which has the latest version of 1.0.0 >> >> >> >> <!-- >> https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients --> >> >> <dependency> >> >> <groupId>org.apache.kafka</groupId> >> >> <artifactId>kafka-clients</artifactId> >> >> <version>1.0.0</version> >> >> </dependency> >> >> >> >> I am confused which path is the right one >> >> >> >> Thanks… >> >> >> >> >> >> >> ________________________________ >> >> Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken >> bilgiler içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu >> iletiyi çoğaltmak ve dağıtmak yasaktır. Bu mesajı yanlışlıkla alan >> kişi, bu durumu derhal gönderene telefonla ya da e-posta ile >> bildirmeli ve bilgisayarından silmelidir. Bu iletinin içeriğinden >> yalnızca iletiyi gönderen kişi sorumludur. >> >> This communication may contain information that is legally >> privileged, confidential or exempt from disclosure. If you are not >> the intended recipient, please note that any dissemination, >> distribution, or copying of this communication is strictly >> prohibited. Anyone who receives this message in error should notify >> the sender immediately by telephone or by return communication and >> delete it from his or her computer. Only the person who has sent this >> message is responsible for its content. ________________________________ Bu ileti hukuken korunmuş, gizli veya ifşa edilmemesi gereken bilgiler içerebilir. Şayet mesajın gönderildiği kişi değilseniz, bu iletiyi çoğaltmak ve dağıtmak yasaktır. Bu mesajı yanlışlıkla alan kişi, bu durumu derhal gönderene telefonla ya da e-posta ile bildirmeli ve bilgisayarından silmelidir. Bu iletinin içeriğinden yalnızca iletiyi gönderen kişi sorumludur. This communication may contain information that is legally privileged, confidential or exempt from disclosure. If you are not the intended recipient, please note that any dissemination, distribution, or copying of this communication is strictly prohibited. Anyone who receives this message in error should notify the sender immediately by telephone or by return communication and delete it from his or her computer. Only the person who has sent this message is responsible for its content. --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
