have you read
https://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html
?
On Mon, Sep 17, 2018 at 4:46 AM hagersaleh <hager.saleh....@gmail.com>
wrote:
> I write code to connect kafka with spark using python and I run code on
> jupyer
> my code
> import os
> #os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars
>
> /home/hadoop/Desktop/spark-program/kafka/spark-streaming-kafka-0-8-assembly_2.10-2.0.0-preview.jar
> pyspark-shell'
> os.environ['PYSPARK_SUBMIT_ARGS'] = "--packages
> org.apache.spark:spark-sql-kafka-0-10_2.11:2.3.0 pyspark-shell"
>
> os.environ['PYSPARK_SUBMIT_ARGS'] = "--packages
> org.apache.spark:spark-streaming-kafka-0-8_2.11:2.3.0 pyspark-shell"
>
> import pyspark
> from pyspark.streaming.kafka import KafkaUtils
> from pyspark.streaming import StreamingContext
>
> #sc = SparkContext()
> ssc = StreamingContext(sc,1)
>
> broker = "iotmsgs"
> directKafkaStream = KafkaUtils.createDirectStream(ssc, ["test1"],
> {"metadata.broker.list": broker})
> directKafkaStream.pprint()
> ssc.start()
>
> error display
> Spark Streaming's Kafka libraries not found in class path. Try one of the
> following.
>
> 1. Include the Kafka library and its dependencies with in the
> spark-submit command as
>
> $ bin/spark-submit --packages
> org.apache.spark:spark-streaming-kafka-0-8:2.3.0 ...
>
> 2. Download the JAR of the artifact from Maven Central
> http://search.maven.org/,
> Group Id = org.apache.spark, Artifact Id =
> spark-streaming-kafka-0-8-assembly, Version = 2.3.0.
> Then, include the jar in the spark-submit command as
>
> $ bin/spark-submit --jars <spark-streaming-kafka-0-8-assembly.jar> ...
>
>
>
>
> --
> Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
>
>
