Re: using Spark Streaming with Kafka 0.9/0.10

Thanks for the link and info Cody ! Regards, Aakash On Tue, Nov 15, 2016 at 7:47 PM, Cody Koeninger wrote: > Generating / defining an RDDis not the same thing as running the > compute() method of an rdd . The direct stream definitely runs kafka > consumers on the executors. > > If you want m

Re: using Spark Streaming with Kafka 0.9/0.10

Generating / defining an RDDis not the same thing as running the compute() method of an rdd . The direct stream definitely runs kafka consumers on the executors. If you want more info, the blog post and video linked from https://github.com/koeninger/kafka-exactly-once refers to the 0.8 implementa

Re: using Spark Streaming with Kafka 0.9/0.10

> You can use the 0.8 artifact to consume from a 0.9 broker We are currently using "Camus " in production and one of the main goal to move to Spark is to use new Kafka Consumer API of Kafka 0.9 and in our case we need the security provisions ava

Re: using Spark Streaming with Kafka 0.9/0.10

It'd probably be worth no longer marking the 0.8 interface as experimental. I don't think it's likely to be subject to active development at this point. You can use the 0.8 artifact to consume from a 0.9 broker Where are you reading documentation indicating that the direct stream only runs on th