Re: SSL between Kafka and Spark Streaming API

e SSL between Spark 1.4.1 and Kafka 0.9.0.0, can someone throw some lite on this pls ? here are few links I have referred http://grokbase.com/p/kafka/users/158wy0wtxk/ssl-between-kafka-and-spark-streaming-api <http://grokbase.com/p/kafka/users/158wy0wtxk/ssl-between-kafka-and-spark-streaming-

Re: SSL between Kafka and Spark Streaming API

SSL is also for 0.8.3... if you are on 0.8.2.1, the normal Spark instructions will work for you. On Fri, Aug 28, 2015 at 1:32 PM, Cassa L wrote: > Just to confirm, is this what you are mentioning about? Is there any > example on how to set it? I believe it is for 0.8.3 version? > > > https://cwi

Re: SSL between Kafka and Spark Streaming API

Just to confirm, is this what you are mentioning about? Is there any example on how to set it? I believe it is for 0.8.3 version? https://cwiki.apache.org/confluence/display/KAFKA/Multiple+Listeners+for+Kafka+Brokers On Fri, Aug 28, 2015 at 12:52 PM, Sriharsha Chintalapani wrote: > You can con

Re: SSL between Kafka and Spark Streaming API

Yeah, the direct api uses the simple consumer On Fri, Aug 28, 2015 at 1:32 PM, Cassa L wrote: > Hi I am using below Spark jars with Direct Stream API. > spark-streaming-kafka_2.10 > > When I look at its pom.xml, Kafka libraries that its pulling in is >org.apache.kafka >kafka_${scal

Re: SSL between Kafka and Spark Streaming API

You can configure PLAINTEXT listener as well with the broker and use that port for spark. --  Harsha On August 28, 2015 at 12:24:45 PM, Sourabh Chandak (sourabh3...@gmail.com) wrote: Can we use the existing kafka spark streaming jar to connect to a kafka server running in SSL mode? We are f

Re: SSL between Kafka and Spark Streaming API

Can we use the existing kafka spark streaming jar to connect to a kafka server running in SSL mode? We are fine with non SSL consumer as our kafka cluster and spark cluster are in the same network Thanks, Sourabh On Fri, Aug 28, 2015 at 12:03 PM, Gwen Shapira wrote: > I can't speak for the Sp

Re: SSL between Kafka and Spark Streaming API

I can't speak for the Spark Community, but checking their code, DirectKafkaStream and KafkaRDD use the SimpleConsumer API: https://github.com/apache/spark/blob/master/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala https://github.com/apache/spark/blob/m

Re: SSL between Kafka and Spark Streaming API

Hi I am using below Spark jars with Direct Stream API. spark-streaming-kafka_2.10 When I look at its pom.xml, Kafka libraries that its pulling in is org.apache.kafka kafka_${scala.binary.version} 0.8.2.1 I believe this DirectStream API uses SimpleConsumer API. Can someone from

Re: SSL between Kafka and Spark Streaming API

SSL is supported for new producer and consumer api and old api (simple consumer and high-level consumer) is not supported. I think spark uses simple consumer? if so its not supported. Thanks, Harsha On August 28, 2015 at 11:00:30 AM, Cassa L (lcas...@gmail.com) wrote: Hi, I was going through

SSL between Kafka and Spark Streaming API

Hi, I was going through SSL setup of Kafka. https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka However, I am also using Spark-Kafka streaming to read data from Kafka. Is there a way to activate SSL for spark streaming API or not possible at all? Thanks, LCassa