Thank you,I can't find spark-streaming-kafka_2.10 jar for spark2 from maven center. so I try the version 1.6.2,it not work ,it need class org.apache.spark.Logging, which can't find in spark2. so I build spark-streaming-kafka_2.10 jar for spark2 from the source code. it's work now.
2016-07-26 2:12 GMT+08:00 Cody Koeninger <c...@koeninger.org>: > For 2.0, the kafka dstream support is in two separate subprojects > depending on which version of Kafka you are using > > spark-streaming-kafka-0-10 > or > spark-streaming-kafka-0-8 > > corresponding to brokers that are version 0.10+ or 0.8+ > > On Mon, Jul 25, 2016 at 12:29 PM, Reynold Xin <r...@databricks.com> wrote: > > The presentation at Spark Summit SF was probably referring to Structured > > Streaming. The existing Spark Streaming (dstream) in Spark 2.0 has the > same > > production stability level as Spark 1.6. There is also Kafka 0.10 > support in > > dstream. > > > > On July 25, 2016 at 10:26:49 AM, Andy Davidson > > (a...@santacruzintegration.com) wrote: > > > > Hi Kevin > > > > Just a heads up at the recent spark summit in S.F. There was a > presentation > > about streaming in 2.0. They said that streaming was not going to > production > > ready in 2.0. > > > > I am not sure if the older 1.6.x version will be supported. My project > will > > not be able to upgrade with streaming support. We also use kafka > > > > Andy > > > > From: Marco Mistroni <mmistr...@gmail.com> > > Date: Monday, July 25, 2016 at 2:33 AM > > To: kevin <kiss.kevin...@gmail.com> > > Cc: "user @spark" <u...@spark.apache.org>, "dev.spark" > > <dev@spark.apache.org> > > Subject: Re: where I can find spark-streaming-kafka for spark2.0 > > > > Hi Kevin > > you should not need to rebuild everything. > > Instead, i believe you should launch spark-submit by specifying the kafka > > jar file in your --packages... i had to follow same when integrating > spark > > streaming with flume > > > > have you checked this link ? > > https://spark.apache.org/docs/latest/streaming-kafka-integration.html > > > > > > hth > > > > > > > > On Mon, Jul 25, 2016 at 10:20 AM, kevin <kiss.kevin...@gmail.com> wrote: > >> > >> I have compile it from source code > >> > >> 2016-07-25 12:05 GMT+08:00 kevin <kiss.kevin...@gmail.com>: > >>> > >>> hi,all : > >>> I try to run example > org.apache.spark.examples.streaming.KafkaWordCount , > >>> I got error : > >>> Exception in thread "main" java.lang.NoClassDefFoundError: > >>> org/apache/spark/streaming/kafka/KafkaUtils$ > >>> at > >>> > org.apache.spark.examples.streaming.KafkaWordCount$.main(KafkaWordCount.scala:57) > >>> at > >>> > org.apache.spark.examples.streaming.KafkaWordCount.main(KafkaWordCount.scala) > >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >>> at > >>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >>> at > >>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>> at java.lang.reflect.Method.invoke(Method.java:498) > >>> at > >>> > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:724) > >>> at > >>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) > >>> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) > >>> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119) > >>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > >>> Caused by: java.lang.ClassNotFoundException: > >>> org.apache.spark.streaming.kafka.KafkaUtils$ > >>> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > >>> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > >>> ... 11 more > >>> > >>> so where I can find spark-streaming-kafka for spark2.0 > >> > >> > > >
