Hi Karan, Why do you try to submit class which does not exist? Don't you want to create it first? Jozef
Sent from [ProtonMail](https://protonmail.ch), encrypted email based in Switzerland. > -------- Original Message -------- > Subject: Re: Kafka-Spark Integration - build failing with sbt > Local Time: July 10, 2017 8:32 PM > UTC Time: July 10, 2017 6:32 PM > From: karan.al...@gmail.com > To: Jozef.koval <jozef.ko...@protonmail.ch> > users@kafka.apache.org <users@kafka.apache.org> > > Hi Jozef, all, > I seem to be running into another issue .. here is what i did .. > I'm running Spark Streaming - Kafka integration using Spark 2.x & Kafka 10. > I compiled the program using sbt, and the compilation went through fine. > I was able able to import this into Eclipse & run the program from Eclipse. > However, when i run the program using spark-submit, i'm getting the following > error : > ---------------------------------- > >> $SPARK_HOME/bin/spark-submit --class >> "structuredStreaming.kafka.StructuredKafkaWordCount1" --master local[2] >> /Users/karanalang/Documents/Technology/Coursera_spark_scala/structuredStreamingKafka/target/scala-2.11/StructuredStreamingKafka-assembly-1.0.jar > >> java.lang.ClassNotFoundException: >> structuredStreaming.kafka.StructuredKafkaWordCount1 >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >> at java.lang.Class.forName0(Native Method) >> at java.lang.Class.forName(Class.java:348) >> at org.apache.spark.util.Utils$.classForName(Utils.scala:229) >> at >> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:695) >> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) >> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > ------------------------------ > > I've put the jar in the classpath, but i still get the error -> > > echo $CLASSPATH > >> .:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/jopt-simple-3.2.jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/kafka-clients-0.9.0.1.jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/kafka_2.11-0.9.0.1.jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/log4j-1.2.17.jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/metrics-core-2.2.0.jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/scala-library-2.11.7.jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/slf4j-api-1.7.6.jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/slf4j-log4j12-1.7.6.jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/snappy-java-1.1.1.7.jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/zkclient-0.7.jar:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/zookeeper-3.4.6.jar:/Users/karanalang/Documents/Technology/ApacheSpark-v2.1/spark-2.1.0-bin-hadoop2.7/jars/*.jar:/Users/karanalang/Documents/Technology/kafka/mirrormaker_topic_rename-master/target/mmchangetopic-1.0-SNAPSHOT.jar:/Users/karanalang/Documents/Technology/Coursera_spark_scala/structuredStreamingKafka/target/scala-2.11/StructuredStreamingKafka-assembly-1.0.jar > > When i look inside the jar - StructuredStreamingKafka-assembly-1.0.jar, i > don't see the file "StructuredKafkaWordCount1.class" > > Attaching my build.sbt. > > Any ideas on what i need to do ? > > On Mon, Jun 19, 2017 at 1:39 PM, Jozef.koval <jozef.ko...@protonmail.ch> > wrote: > >> Hi Karan, >> You assume incorrectly, KafkaUtils is part of (already mentioned) >> spark-streaming-kafka library. I have not seen Eclipse in ages, but I would >> suggest to import sbt project there, could make it easier for you. Or >> alternatively, using ENSIME with your favorite editor will make your life >> much easier. >> Jozef >> >> Sent from [ProtonMail](https://protonmail.ch), encrypted email based in >> Switzerland. >> >>> -------- Original Message -------- >>> Subject: Re: Kafka-Spark Integration - build failing with sbt >>> >>> Local Time: June 19, 2017 8:02 PM >>> UTC Time: June 19, 2017 6:02 PM >>> From: karan.al...@gmail.com >>> To: Jozef.koval <jozef.ko...@protonmail.ch> >>> users@kafka.apache.org <users@kafka.apache.org> >>> >>> Hi Jozef - i do have a additional basic question .. >>> When i tried to compile the code in Eclipse, i was not able to do that >>> eg. >>> import org.apache.spark.streaming.kafka.KafkaUtils >>> gave errors saying KafaUtils was not part of the package. >>> However, when i used sbt to compile - the compilation went through fine >>> So, I assume additional libraries are being downloaded when i provide the >>> appropriate packages in LibraryDependencies ? >>> which ones would have helped compile this ? >>> >>> On Sat, Jun 17, 2017 at 2:52 PM, karan alang <karan.al...@gmail.com> wrote: >>> >>>> Hey Jozef, >>>> Thanks for the quick response .. >>>> yes, you are right .. spark-sql dependency was missing .. added that & it >>>> worked fine. >>>> regds, >>>> Karan Alang >>>> >>>> On Sat, Jun 17, 2017 at 2:24 PM, Jozef.koval <jozef.ko...@protonmail.ch> >>>> wrote: >>>> >>>>> Hey Karan, >>>>> I believe you are missing spark-sql dependency. >>>>> Jozef >>>>> >>>>> Sent from [ProtonMail](https://protonmail.ch), encrypted email based in >>>>> Switzerland. >>>>> >>>>>> -------- Original Message -------- >>>>>> Subject: Re: Kafka-Spark Integration - build failing with sbt >>>>>> Local Time: June 17, 2017 10:52 PM >>>>>> UTC Time: June 17, 2017 8:52 PM >>>>>> From: karan.al...@gmail.com >>>>>> To: users@kafka.apache.org, Jozef.koval <jozef.ko...@protonmail.ch> >>>>>> >>>>>> Thanks, i was able to get this working. >>>>>> here is what i added in build.sbt file >>>>>> ---------------------------------------------------------------------------------------------- >>>>>> >>>>>> scalaVersion := "2.11.7" >>>>>> >>>>>> val sparkVers = "2.1.0" >>>>>> >>>>>> // Base Spark-provided dependencies >>>>>> >>>>>> libraryDependencies ++= Seq( >>>>>> >>>>>> "org.apache.spark" %% "spark-core" % sparkVers % "provided", >>>>>> >>>>>> "org.apache.spark" %% "spark-streaming" % sparkVers % "provided", >>>>>> >>>>>> "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % sparkVers) >>>>>> >>>>>> ------------------------------------------------------------------------------------------------ >>>>>> >>>>>> However, i'm running into addition issue when compiling the file using >>>>>> sbt .. it gives errors as shown below : >>>>>> >>>>>> Pls note - I've added the jars in eclipse, and it works .. however when >>>>>> i use sbt to compile, it is failing. >>>>>> >>>>>> What needs to be done ? >>>>>> >>>>>> I've also tried added the jars in CLASSPATH, but still get the same >>>>>> error. >>>>>> >>>>>> ------------------------------------------------------------------------------------------------- >>>>>> >>>>>> [info] Compiling 1 Scala source to >>>>>> /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/target/scala-2.11/classes... >>>>>> >>>>>> [error] >>>>>> /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:6: >>>>>> object SQLContext is not a member of package org.apache.spark.sql >>>>>> >>>>>> [error] import org.apache.spark.sql.SQLContext >>>>>> >>>>>> [error] ^ >>>>>> >>>>>> [error] >>>>>> /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:10: >>>>>> object SparkSession is not a member of package org.apache.spark.sql >>>>>> >>>>>> [error] import org.apache.spark.sql.SparkSession >>>>>> >>>>>> [error] ^ >>>>>> >>>>>> [error] >>>>>> /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:11: >>>>>> object types is not a member of package org.apache.spark.sql >>>>>> >>>>>> [error] import org.apache.spark.sql.types.StructField >>>>>> >>>>>> [error] ^ >>>>>> >>>>>> [error] >>>>>> /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:12: >>>>>> object types is not a member of package org.apache.spark.sql >>>>>> >>>>>> [error] import org.apache.spark.sql.types.StringType >>>>>> >>>>>> [error] ^ >>>>>> >>>>>> [error] >>>>>> /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:13: >>>>>> object types is not a member of package org.apache.spark.sql >>>>>> >>>>>> [error] import org.apache.spark.sql.types.StructType >>>>>> >>>>>> [error] ^ >>>>>> >>>>>> [error] >>>>>> /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:14: >>>>>> object Row is not a member of package org.apache.spark.sql >>>>>> >>>>>> [error] import org.apache.spark.sql.Row >>>>>> >>>>>> [error] ^ >>>>>> >>>>>> On Sat, Jun 17, 2017 at 12:35 AM, Jozef.koval >>>>>> <jozef.ko...@protonmail.ch> wrote: >>>>>> >>>>>>> Hi Karan, >>>>>>> spark-streaming-kafka is for old spark (version < 1.6.3) >>>>>>> spark-streaming-kafka-0.8 is for current spark (version > 2.0) >>>>>>> Jozef >>>>>>> n.b. there is also version for kafka 0.10+ see >>>>>>> [this](https://spark.apache.org/docs/latest/streaming-kafka-integration.html) >>>>>>> Sent from [ProtonMail](https://protonmail.ch), encrypted email based in >>>>>>> Switzerland. >>>>>>> >>>>>>> -------- Original Message -------- >>>>>>> Subject: Kafka-Spark Integration - build failing with sbt >>>>>>> Local Time: June 17, 2017 1:50 AM >>>>>>> UTC Time: June 16, 2017 11:50 PM >>>>>>> From: karan.al...@gmail.com >>>>>>> To: users@kafka.apache.org >>>>>>> I"m trying to compile kafka & Spark Streaming integration code i.e. >>>>>>> reading >>>>>>> from Kafka using Spark Streaming, >>>>>>> and the sbt build is failing with error - >>>>>>> [error] (*:update) sbt.ResolveException: unresolved dependency: >>>>>>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found >>>>>>> Scala version -> 2.10.7 >>>>>>> Spark Version -> 2.1.0 >>>>>>> Kafka version -> 0.9 >>>>>>> sbt version -> 0.13 >>>>>>> Contents of sbt files is as shown below -> >>>>>>> 1) >>>>>>> vi spark_kafka_code/project/plugins.sbt >>>>>>> addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2") >>>>>>> 2) >>>>>>> vi spark_kafka_code/sparkkafka.sbt >>>>>>> import AssemblyKeys._ >>>>>>> assemblySettings >>>>>>> name := "SparkKafka Project" >>>>>>> version := "1.0" >>>>>>> scalaVersion := "2.11.7" >>>>>>> val sparkVers = "2.1.0" >>>>>>> // Base Spark-provided dependencies >>>>>>> libraryDependencies ++= Seq( >>>>>>> "org.apache.spark" %% "spark-core" % sparkVers % "provided", >>>>>>> "org.apache.spark" %% "spark-streaming" % sparkVers % "provided", >>>>>>> "org.apache.spark" %% "spark-streaming-kafka" % sparkVers) >>>>>>> mergeStrategy in assembly := { >>>>>>> case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard >>>>>>> case m if m.toLowerCase.startsWith("META-INF") => MergeStrategy.discard >>>>>>> case "reference.conf" => MergeStrategy.concat >>>>>>> case m if m.endsWith("UnusedStubClass.class") => MergeStrategy.discard >>>>>>> case _ => MergeStrategy.first >>>>>>> } >>>>>>> i launch sbt, and then try to create an eclipse project, complete error >>>>>>> is as shown below - >>>>>>> --------------------- >>>>>>> sbt >>>>>>> [info] Loading global plugins from /Users/karanalang/.sbt/0.13/plugins >>>>>>> [info] Loading project definition from >>>>>>> /Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/project >>>>>>> [info] Set current project to SparkKafka Project (in build >>>>>>> file:/Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/) >>>>>>>> eclipse >>>>>>> [info] About to create Eclipse project files for your project(s). >>>>>>> [info] Updating >>>>>>> {file:/Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/}spark_kafka_code... >>>>>>> [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 ... >>>>>>> [warn] module not found: >>>>>>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0 >>>>>>> [warn] ==== local: tried >>>>>>> [warn] >>>>>>> /Users/karanalang/.ivy2/local/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >>>>>>> [warn] ==== activator-launcher-local: tried >>>>>>> [warn] >>>>>>> /Users/karanalang/.activator/repository/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >>>>>>> [warn] ==== activator-local: tried >>>>>>> [warn] >>>>>>> /Users/karanalang/Documents/Technology/SCALA/activator-dist-1.3.10/repository/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >>>>>>> [warn] ==== public: tried >>>>>>> [warn] >>>>>>> https://repo1.maven.org/maven2/org/apache/spark/spark-streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom >>>>>>> [warn] ==== typesafe-releases: tried >>>>>>> [warn] >>>>>>> http://repo.typesafe.com/typesafe/releases/org/apache/spark/spark-streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom >>>>>>> [warn] ==== typesafe-ivy-releasez: tried >>>>>>> [warn] >>>>>>> http://repo.typesafe.com/typesafe/ivy-releases/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >>>>>>> [info] Resolving jline#jline;2.12.1 ... >>>>>>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >>>>>>> [warn] :: UNRESOLVED DEPENDENCIES :: >>>>>>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >>>>>>> [warn] :: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found >>>>>>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >>>>>>> [warn] >>>>>>> [warn] Note: Unresolved dependencies path: >>>>>>> [warn] org.apache.spark:spark-streaming-kafka_2.11:2.1.0 >>>>>>> (/Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/sparkkafka.sbt#L12-16) >>>>>>> [warn] +- sparkkafka-project:sparkkafka-project_2.11:1.0 >>>>>>> [trace] Stack trace suppressed: run last *:update for the full output. >>>>>>> [error] (*:update) sbt.ResolveException: unresolved dependency: >>>>>>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found >>>>>>> [info] Updating >>>>>>> {file:/Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/}spark_kafka_code... >>>>>>> [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 ... >>>>>>> [warn] module not found: >>>>>>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0 >>>>>>> [warn] ==== local: tried >>>>>>> [warn] >>>>>>> /Users/karanalang/.ivy2/local/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >>>>>>> [warn] ==== activator-launcher-local: tried >>>>>>> [warn] >>>>>>> /Users/karanalang/.activator/repository/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >>>>>>> [warn] ==== activator-local: tried >>>>>>> [warn] >>>>>>> /Users/karanalang/Documents/Technology/SCALA/activator-dist-1.3.10/repository/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >>>>>>> [warn] ==== public: tried >>>>>>> [warn] >>>>>>> https://repo1.maven.org/maven2/org/apache/spark/spark-streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom >>>>>>> [warn] ==== typesafe-releases: tried >>>>>>> [warn] >>>>>>> http://repo.typesafe.com/typesafe/releases/org/apache/spark/spark-streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom >>>>>>> [warn] ==== typesafe-ivy-releasez: tried >>>>>>> [warn] >>>>>>> http://repo.typesafe.com/typesafe/ivy-releases/org.apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >>>>>>> [info] Resolving jline#jline;2.12.1 ... >>>>>>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >>>>>>> [warn] :: UNRESOLVED DEPENDENCIES :: >>>>>>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >>>>>>> [warn] :: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found >>>>>>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >>>>>>> [warn] >>>>>>> [warn] Note: Unresolved dependencies path: >>>>>>> [warn] org.apache.spark:spark-streaming-kafka_2.11:2.1.0 >>>>>>> (/Users/karanalang/Documents/Technology/Coursera_spark_scala/spark_kafka_code/sparkkafka.sbt#L12-16) >>>>>>> [warn] +- sparkkafka-project:sparkkafka-project_2.11:1.0 >>>>>>> [trace] Stack trace suppressed: run last *:update for the full output. >>>>>>> [error] (*:update) sbt.ResolveException: unresolved dependency: >>>>>>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found >>>>>>> [error] Could not create Eclipse project files: >>>>>>> [error] Error evaluating task "scalacOptions": error >>>>>>> [error] Error evaluating task "externalDependencyClasspath": error >>>>>>>>
