Piotr Nestorow created SPARK-19579:
--------------------------------------
Summary: spark-submit fails to run Kafka Stream python script
Key: SPARK-19579
URL: https://issues.apache.org/jira/browse/SPARK-19579
Project: Spark
Issue Type: Bug
Components: Spark Submit
Affects Versions: 2.1.0
Environment: Linux Ubuntu 16.10 64bit
Reporter: Piotr Nestorow
Kafka Stream python script is executed but it fails with:
TypeError: 'JavaPackage' object is not callable
The Spark Kafka streaming jar is provided:
spark-streaming-kafka-0-10_2.11-2.1.0.jar
In the conf/spark-defaults.conf:
spark.jars.packages org.apache.spark:spark-streaming-kafka-0-10_2.11:2.1.0
Also at runtime:
bin/spark-submit --jars
/usr/local/spark/jars/spark-streaming-kafka-0-10_2.11-2.1.0.jar
--driver-class-path
/usr/local/spark/jars/spark-streaming-kafka-0-10_2.11-2.1.0.jar
/home/sysveradmin/work/Programs/ApacheSpark/ex_kafka_stream.py localhost:2181
example_topic
Nothing helps...
Detailed output:
_________________
Spark Streaming's Kafka libraries not found in class path. Try one of the
following.
1. Include the Kafka library and its dependencies with in the
spark-submit command as
$ bin/spark-submit --packages
org.apache.spark:spark-streaming-kafka-0-8:2.1.0 ...
2. Download the JAR of the artifact from Maven Central
http://search.maven.org/,
Group Id = org.apache.spark, Artifact Id =
spark-streaming-kafka-0-8-assembly, Version = 2.1.0.
Then, include the jar in the spark-submit command as
$ bin/spark-submit --jars <spark-streaming-kafka-0-8-assembly.jar> ...
________________________________________________________________________________________________
Traceback (most recent call last):
File "/home/sysveradmin/work/Programs/ApacheSpark/ex_kafka_stream.py", line
18, in <module>
kvs = KafkaUtils.createStream(ssc, zkQuorum, "spark-streaming-consumer",
{topic: 1})
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/streaming/kafka.py",
line 69, in createStream
File "/usr/local/spark/python/lib/pyspark.zip/pyspark/streaming/kafka.py",
line 195, in _get_helper
TypeError: 'JavaPackage' object is not callable
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
