Hi Jozef - the class does exist ( but in different location i,e. under src/main) btw, i was able to resolve the issue ..
sbt by default considers src/main/scala as default source location, I'd changed the location to a different one. I changed the build.sbt to point to the required location, that fixed the issue regds, Karan Alang On Wed, Jul 12, 2017 at 1:01 PM, Jozef.koval <jozef.ko...@protonmail.ch> wrote: > Hi Karan, > Why do you try to submit class which does not exist? Don't you want to > create it first? > Jozef > > Sent from [ProtonMail](https://protonmail.ch), encrypted email based in > Switzerland. > > > -------- Original Message -------- > > Subject: Re: Kafka-Spark Integration - build failing with sbt > > Local Time: July 10, 2017 8:32 PM > > UTC Time: July 10, 2017 6:32 PM > > From: karan.al...@gmail.com > > To: Jozef.koval <jozef.ko...@protonmail.ch> > > users@kafka.apache.org <users@kafka.apache.org> > > > > Hi Jozef, all, > > I seem to be running into another issue .. here is what i did .. > > I'm running Spark Streaming - Kafka integration using Spark 2.x & Kafka > 10. > > I compiled the program using sbt, and the compilation went through fine. > > I was able able to import this into Eclipse & run the program from > Eclipse. > > However, when i run the program using spark-submit, i'm getting the > following error : > > ---------------------------------- > > > >> $SPARK_HOME/bin/spark-submit --class > >> "structuredStreaming.kafka.StructuredKafkaWordCount1" > --master local[2] /Users/karanalang/Documents/Technology/Coursera_spark_ > scala/structuredStreamingKafka/target/scala-2.11/StructuredStreamingKafka- > assembly-1.0.jar > > > >> java.lang.ClassNotFoundException: structuredStreaming.kafka. > StructuredKafkaWordCount1 > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > >> at java.lang.Class.forName0(Native Method) > >> at java.lang.Class.forName(Class.java:348) > >> at org.apache.spark.util.Utils$.classForName(Utils.scala:229) > >> at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$ > deploy$SparkSubmit$$runMain(SparkSubmit.scala:695) > >> at org.apache.spark.deploy.SparkSubmit$.doRunMain$1( > SparkSubmit.scala:187) > >> at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) > >> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) > >> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > > >> at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > > > > ------------------------------ > > > > I've put the jar in the classpath, but i still get the error -> > > > > echo $CLASSPATH > > > >> .:/Users/karanalang/Documents/Technology/kafka/kafka_2.11-0. > 9.0.1/lib/jopt-simple-3.2.jar:/Users/karanalang/Documents/ > Technology/kafka/kafka_2.11-0.9.0.1/lib/kafka-clients-0.9.0. > 1.jar:/Users/karanalang/Documents/Technology/kafka/ > kafka_2.11-0.9.0.1/lib/kafka_2.11-0.9.0.1.jar:/Users/karanalang/Documents/ > Technology/kafka/kafka_2.11-0.9.0.1/lib/log4j-1.2.17.jar:/ > Users/karanalang/Documents/Technology/kafka/kafka_2.11-0. > 9.0.1/lib/metrics-core-2.2.0.jar:/Users/karanalang/ > Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/scala- > library-2.11.7.jar:/Users/karanalang/Documents/ > Technology/kafka/kafka_2.11-0.9.0.1/lib/slf4j-api-1.7.6.jar: > /Users/karanalang/Documents/Technology/kafka/kafka_2.11-0. > 9.0.1/lib/slf4j-log4j12-1.7.6.jar:/Users/karanalang/ > Documents/Technology/kafka/kafka_2.11-0.9.0.1/lib/snappy- > java-1.1.1.7.jar:/Users/karanalang/Documents/ > Technology/kafka/kafka_2.11-0.9.0.1/lib/zkclient-0.7.jar:/ > Users/karanalang/Documents/Technology/kafka/kafka_2.11-0. > 9.0.1/lib/zookeeper-3.4.6.jar:/Users/karanalang/Documents/ > Technology/ApacheSpark-v2.1/spark-2.1.0-bin-hadoop2.7/ > jars/*.jar:/Users/karanalang/Documents/Technology/kafka/ > mirrormaker_topic_rename-master/target/mmchangetopic-1. > 0-SNAPSHOT.jar:/Users/karanalang/Documents/Technology/Coursera_spark_ > scala/structuredStreamingKafka/target/scala-2.11/StructuredStreamingKafka- > assembly-1.0.jar > > > > When i look inside the jar - StructuredStreamingKafka-assembly-1.0.jar, > i don't see the file "StructuredKafkaWordCount1.class" > > > > Attaching my build.sbt. > > > > Any ideas on what i need to do ? > > > > On Mon, Jun 19, 2017 at 1:39 PM, Jozef.koval <jozef.ko...@protonmail.ch> > wrote: > > > >> Hi Karan, > >> You assume incorrectly, KafkaUtils is part of (already mentioned) > spark-streaming-kafka library. I have not seen Eclipse in ages, but I would > suggest to import sbt project there, could make it easier for you. Or > alternatively, using ENSIME with your favorite editor will make your life > much easier. > >> Jozef > >> > >> Sent from [ProtonMail](https://protonmail.ch), encrypted email based > in Switzerland. > >> > >>> -------- Original Message -------- > >>> Subject: Re: Kafka-Spark Integration - build failing with sbt > >>> > >>> Local Time: June 19, 2017 8:02 PM > >>> UTC Time: June 19, 2017 6:02 PM > >>> From: karan.al...@gmail.com > >>> To: Jozef.koval <jozef.ko...@protonmail.ch> > >>> users@kafka.apache.org <users@kafka.apache.org> > >>> > >>> Hi Jozef - i do have a additional basic question .. > >>> When i tried to compile the code in Eclipse, i was not able to do that > >>> eg. > >>> import org.apache.spark.streaming.kafka.KafkaUtils > >>> gave errors saying KafaUtils was not part of the package. > >>> However, when i used sbt to compile - the compilation went through fine > >>> So, I assume additional libraries are being downloaded when i provide > the appropriate packages in LibraryDependencies ? > >>> which ones would have helped compile this ? > >>> > >>> On Sat, Jun 17, 2017 at 2:52 PM, karan alang <karan.al...@gmail.com> > wrote: > >>> > >>>> Hey Jozef, > >>>> Thanks for the quick response .. > >>>> yes, you are right .. spark-sql dependency was missing .. added that > & it worked fine. > >>>> regds, > >>>> Karan Alang > >>>> > >>>> On Sat, Jun 17, 2017 at 2:24 PM, Jozef.koval < > jozef.ko...@protonmail.ch> wrote: > >>>> > >>>>> Hey Karan, > >>>>> I believe you are missing spark-sql dependency. > >>>>> Jozef > >>>>> > >>>>> Sent from [ProtonMail](https://protonmail.ch), encrypted email > based in Switzerland. > >>>>> > >>>>>> -------- Original Message -------- > >>>>>> Subject: Re: Kafka-Spark Integration - build failing with sbt > >>>>>> Local Time: June 17, 2017 10:52 PM > >>>>>> UTC Time: June 17, 2017 8:52 PM > >>>>>> From: karan.al...@gmail.com > >>>>>> To: users@kafka.apache.org, Jozef.koval <jozef.ko...@protonmail.ch> > >>>>>> > >>>>>> Thanks, i was able to get this working. > >>>>>> here is what i added in build.sbt file > >>>>>> ------------------------------------------------------------ > ---------------------------------- > >>>>>> > >>>>>> scalaVersion := "2.11.7" > >>>>>> > >>>>>> val sparkVers = "2.1.0" > >>>>>> > >>>>>> // Base Spark-provided dependencies > >>>>>> > >>>>>> libraryDependencies ++= Seq( > >>>>>> > >>>>>> "org.apache.spark" %% "spark-core" % sparkVers % "provided", > >>>>>> > >>>>>> "org.apache.spark" %% "spark-streaming" % sparkVers % "provided", > >>>>>> > >>>>>> "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % sparkVers) > >>>>>> > >>>>>> ------------------------------------------------------------ > ------------------------------------ > >>>>>> > >>>>>> However, i'm running into addition issue when compiling the file > using sbt .. it gives errors as shown below : > >>>>>> > >>>>>> Pls note - I've added the jars in eclipse, and it works .. however > when i use sbt to compile, it is failing. > >>>>>> > >>>>>> What needs to be done ? > >>>>>> > >>>>>> I've also tried added the jars in CLASSPATH, but still get the same > error. > >>>>>> > >>>>>> ------------------------------------------------------------ > ------------------------------------- > >>>>>> > >>>>>> [info] Compiling 1 Scala source to /Users/karanalang/Documents/ > Technology/Coursera_spark_scala/spark_kafka_code/target/ > scala-2.11/classes... > >>>>>> > >>>>>> [error] /Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:6: > object SQLContext is not a member of package org.apache.spark.sql > >>>>>> > >>>>>> [error] import org.apache.spark.sql.SQLContext > >>>>>> > >>>>>> [error] ^ > >>>>>> > >>>>>> [error] /Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:10: > object SparkSession is not a member of package org.apache.spark.sql > >>>>>> > >>>>>> [error] import org.apache.spark.sql.SparkSession > >>>>>> > >>>>>> [error] ^ > >>>>>> > >>>>>> [error] /Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:11: > object types is not a member of package org.apache.spark.sql > >>>>>> > >>>>>> [error] import org.apache.spark.sql.types.StructField > >>>>>> > >>>>>> [error] ^ > >>>>>> > >>>>>> [error] /Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:12: > object types is not a member of package org.apache.spark.sql > >>>>>> > >>>>>> [error] import org.apache.spark.sql.types.StringType > >>>>>> > >>>>>> [error] ^ > >>>>>> > >>>>>> [error] /Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:13: > object types is not a member of package org.apache.spark.sql > >>>>>> > >>>>>> [error] import org.apache.spark.sql.types.StructType > >>>>>> > >>>>>> [error] ^ > >>>>>> > >>>>>> [error] /Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:14: > object Row is not a member of package org.apache.spark.sql > >>>>>> > >>>>>> [error] import org.apache.spark.sql.Row > >>>>>> > >>>>>> [error] ^ > >>>>>> > >>>>>> On Sat, Jun 17, 2017 at 12:35 AM, Jozef.koval < > jozef.ko...@protonmail.ch> wrote: > >>>>>> > >>>>>>> Hi Karan, > >>>>>>> spark-streaming-kafka is for old spark (version < 1.6.3) > >>>>>>> spark-streaming-kafka-0.8 is for current spark (version > 2.0) > >>>>>>> Jozef > >>>>>>> n.b. there is also version for kafka 0.10+ see [this]( > https://spark.apache.org/docs/latest/streaming-kafka-integration.html) > >>>>>>> Sent from [ProtonMail](https://protonmail.ch), encrypted email > based in Switzerland. > >>>>>>> > >>>>>>> -------- Original Message -------- > >>>>>>> Subject: Kafka-Spark Integration - build failing with sbt > >>>>>>> Local Time: June 17, 2017 1:50 AM > >>>>>>> UTC Time: June 16, 2017 11:50 PM > >>>>>>> From: karan.al...@gmail.com > >>>>>>> To: users@kafka.apache.org > >>>>>>> I"m trying to compile kafka & Spark Streaming integration code > i.e. reading > >>>>>>> from Kafka using Spark Streaming, > >>>>>>> and the sbt build is failing with error - > >>>>>>> [error] (*:update) sbt.ResolveException: unresolved dependency: > >>>>>>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found > >>>>>>> Scala version -> 2.10.7 > >>>>>>> Spark Version -> 2.1.0 > >>>>>>> Kafka version -> 0.9 > >>>>>>> sbt version -> 0.13 > >>>>>>> Contents of sbt files is as shown below -> > >>>>>>> 1) > >>>>>>> vi spark_kafka_code/project/plugins.sbt > >>>>>>> addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2") > >>>>>>> 2) > >>>>>>> vi spark_kafka_code/sparkkafka.sbt > >>>>>>> import AssemblyKeys._ > >>>>>>> assemblySettings > >>>>>>> name := "SparkKafka Project" > >>>>>>> version := "1.0" > >>>>>>> scalaVersion := "2.11.7" > >>>>>>> val sparkVers = "2.1.0" > >>>>>>> // Base Spark-provided dependencies > >>>>>>> libraryDependencies ++= Seq( > >>>>>>> "org.apache.spark" %% "spark-core" % sparkVers % "provided", > >>>>>>> "org.apache.spark" %% "spark-streaming" % sparkVers % "provided", > >>>>>>> "org.apache.spark" %% "spark-streaming-kafka" % sparkVers) > >>>>>>> mergeStrategy in assembly := { > >>>>>>> case m if m.toLowerCase.endsWith("manifest.mf") => > MergeStrategy.discard > >>>>>>> case m if m.toLowerCase.startsWith("META-INF") => > MergeStrategy.discard > >>>>>>> case "reference.conf" => MergeStrategy.concat > >>>>>>> case m if m.endsWith("UnusedStubClass.class") => > MergeStrategy.discard > >>>>>>> case _ => MergeStrategy.first > >>>>>>> } > >>>>>>> i launch sbt, and then try to create an eclipse project, complete > error > >>>>>>> is as shown below - > >>>>>>> --------------------- > >>>>>>> sbt > >>>>>>> [info] Loading global plugins from /Users/karanalang/.sbt/0.13/ > plugins > >>>>>>> [info] Loading project definition from > >>>>>>> /Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/project > >>>>>>> [info] Set current project to SparkKafka Project (in build > >>>>>>> file:/Users/karanalang/Documents/Technology/Coursera_ > spark_scala/spark_kafka_code/) > >>>>>>>> eclipse > >>>>>>> [info] About to create Eclipse project files for your project(s). > >>>>>>> [info] Updating > >>>>>>> {file:/Users/karanalang/Documents/Technology/Coursera_ > spark_scala/spark_kafka_code/}spark_kafka_code... > >>>>>>> [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 > ... > >>>>>>> [warn] module not found: > >>>>>>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0 > >>>>>>> [warn] ==== local: tried > >>>>>>> [warn] > >>>>>>> /Users/karanalang/.ivy2/local/org.apache.spark/spark- > streaming-kafka_2.11/2.1.0/ivys/ivy.xml > >>>>>>> [warn] ==== activator-launcher-local: tried > >>>>>>> [warn] > >>>>>>> /Users/karanalang/.activator/repository/org.apache.spark/ > spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml > >>>>>>> [warn] ==== activator-local: tried > >>>>>>> [warn] > >>>>>>> /Users/karanalang/Documents/Technology/SCALA/activator- > dist-1.3.10/repository/org.apache.spark/spark-streaming- > kafka_2.11/2.1.0/ivys/ivy.xml > >>>>>>> [warn] ==== public: tried > >>>>>>> [warn] > >>>>>>> https://repo1.maven.org/maven2/org/apache/spark/spark- > streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom > >>>>>>> [warn] ==== typesafe-releases: tried > >>>>>>> [warn] > >>>>>>> http://repo.typesafe.com/typesafe/releases/org/apache/ > spark/spark-streaming-kafka_2.11/2.1.0/spark-streaming- > kafka_2.11-2.1.0.pom > >>>>>>> [warn] ==== typesafe-ivy-releasez: tried > >>>>>>> [warn] > >>>>>>> http://repo.typesafe.com/typesafe/ivy-releases/org. > apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml > >>>>>>> [info] Resolving jline#jline;2.12.1 ... > >>>>>>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: > >>>>>>> [warn] :: UNRESOLVED DEPENDENCIES :: > >>>>>>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: > >>>>>>> [warn] :: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not > found > >>>>>>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: > >>>>>>> [warn] > >>>>>>> [warn] Note: Unresolved dependencies path: > >>>>>>> [warn] org.apache.spark:spark-streaming-kafka_2.11:2.1.0 > >>>>>>> (/Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/sparkkafka.sbt#L12-16) > >>>>>>> [warn] +- sparkkafka-project:sparkkafka-project_2.11:1.0 > >>>>>>> [trace] Stack trace suppressed: run last *:update for the full > output. > >>>>>>> [error] (*:update) sbt.ResolveException: unresolved dependency: > >>>>>>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found > >>>>>>> [info] Updating > >>>>>>> {file:/Users/karanalang/Documents/Technology/Coursera_ > spark_scala/spark_kafka_code/}spark_kafka_code... > >>>>>>> [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 > ... > >>>>>>> [warn] module not found: > >>>>>>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0 > >>>>>>> [warn] ==== local: tried > >>>>>>> [warn] > >>>>>>> /Users/karanalang/.ivy2/local/org.apache.spark/spark- > streaming-kafka_2.11/2.1.0/ivys/ivy.xml > >>>>>>> [warn] ==== activator-launcher-local: tried > >>>>>>> [warn] > >>>>>>> /Users/karanalang/.activator/repository/org.apache.spark/ > spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml > >>>>>>> [warn] ==== activator-local: tried > >>>>>>> [warn] > >>>>>>> /Users/karanalang/Documents/Technology/SCALA/activator- > dist-1.3.10/repository/org.apache.spark/spark-streaming- > kafka_2.11/2.1.0/ivys/ivy.xml > >>>>>>> [warn] ==== public: tried > >>>>>>> [warn] > >>>>>>> https://repo1.maven.org/maven2/org/apache/spark/spark- > streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom > >>>>>>> [warn] ==== typesafe-releases: tried > >>>>>>> [warn] > >>>>>>> http://repo.typesafe.com/typesafe/releases/org/apache/ > spark/spark-streaming-kafka_2.11/2.1.0/spark-streaming- > kafka_2.11-2.1.0.pom > >>>>>>> [warn] ==== typesafe-ivy-releasez: tried > >>>>>>> [warn] > >>>>>>> http://repo.typesafe.com/typesafe/ivy-releases/org. > apache.spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml > >>>>>>> [info] Resolving jline#jline;2.12.1 ... > >>>>>>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: > >>>>>>> [warn] :: UNRESOLVED DEPENDENCIES :: > >>>>>>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: > >>>>>>> [warn] :: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not > found > >>>>>>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: > >>>>>>> [warn] > >>>>>>> [warn] Note: Unresolved dependencies path: > >>>>>>> [warn] org.apache.spark:spark-streaming-kafka_2.11:2.1.0 > >>>>>>> (/Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/sparkkafka.sbt#L12-16) > >>>>>>> [warn] +- sparkkafka-project:sparkkafka-project_2.11:1.0 > >>>>>>> [trace] Stack trace suppressed: run last *:update for the full > output. > >>>>>>> [error] (*:update) sbt.ResolveException: unresolved dependency: > >>>>>>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found > >>>>>>> [error] Could not create Eclipse project files: > >>>>>>> [error] Error evaluating task "scalacOptions": error > >>>>>>> [error] Error evaluating task "externalDependencyClasspath": error > >>>>>>>> >
