Hey Jozef, Thanks for the quick response .. yes, you are right .. spark-sql dependency was missing .. added that & it worked fine.
regds, Karan Alang On Sat, Jun 17, 2017 at 2:24 PM, Jozef.koval <jozef.ko...@protonmail.ch> wrote: > Hey Karan, > I believe you are missing spark-sql dependency. > > Jozef > > Sent from ProtonMail <https://protonmail.ch>, encrypted email based in > Switzerland. > > > -------- Original Message -------- > Subject: Re: Kafka-Spark Integration - build failing with sbt > Local Time: June 17, 2017 10:52 PM > UTC Time: June 17, 2017 8:52 PM > From: karan.al...@gmail.com > To: users@kafka.apache.org, Jozef.koval <jozef.ko...@protonmail.ch> > > > Thanks, i was able to get this working. > here is what i added in build.sbt file > ------------------------------------------------------------ > ---------------------------------- > > scalaVersion := "2.11.7" > > val sparkVers = "2.1.0" > > // Base Spark-provided dependencies > > libraryDependencies ++= Seq( > > "org.apache.spark" %% "spark-core" % sparkVers % "provided", > > "org.apache.spark" %% "spark-streaming" % sparkVers % "provided", > > "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % sparkVers) > > ------------------------------------------------------------ > ------------------------------------ > > However, i'm running into addition issue when compiling the file using sbt > .. it gives errors as shown below : > > *Pls note - I've added the jars in eclipse, and it works .. however when i > use sbt to compile, it is failing.* > > *What needs to be done ? * > > *I've also tried added the jars in CLASSPATH, but still get the same > error. * > > ------------------------------------------------------------ > ------------------------------------- > > [info] Compiling 1 Scala source to /Users/karanalang/Documents/ > Technology/Coursera_spark_scala/spark_kafka_code/target/ > scala-2.11/classes... > > [error] /Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:6: > object SQLContext is not a member of package org.apache.spark.sql > > [error] import org.apache.spark.sql.SQLContext > > [error] ^ > > [error] /Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:10: > object SparkSession is not a member of package org.apache.spark.sql > > [error] import org.apache.spark.sql.SparkSession > > [error] ^ > > [error] /Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:11: > object types is not a member of package org.apache.spark.sql > > [error] import org.apache.spark.sql.types.StructField > > [error] ^ > > [error] /Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:12: > object types is not a member of package org.apache.spark.sql > > [error] import org.apache.spark.sql.types.StringType > > [error] ^ > > [error] /Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:13: > object types is not a member of package org.apache.spark.sql > > [error] import org.apache.spark.sql.types.StructType > > [error] ^ > > [error] /Users/karanalang/Documents/Technology/Coursera_spark_ > scala/spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:14: > object Row is not a member of package org.apache.spark.sql > > [error] import org.apache.spark.sql.Row > > [error] ^ > > > > On Sat, Jun 17, 2017 at 12:35 AM, Jozef.koval <jozef.ko...@protonmail.ch> > wrote: > >> Hi Karan, >> >> spark-streaming-kafka is for old spark (version < 1.6.3) >> spark-streaming-kafka-0.8 is for current spark (version > 2.0) >> >> Jozef >> >> n.b. there is also version for kafka 0.10+ see [this]( >> https://spark.apache.org/docs/latest/streaming-kafka-integration.html) >> >> Sent from [ProtonMail](https://protonmail.ch), encrypted email based in >> Switzerland. >> >> >> -------- Original Message -------- >> Subject: Kafka-Spark Integration - build failing with sbt >> Local Time: June 17, 2017 1:50 AM >> UTC Time: June 16, 2017 11:50 PM >> From: karan.al...@gmail.com >> To: users@kafka.apache.org >> >> I"m trying to compile kafka & Spark Streaming integration code i.e. >> reading >> from Kafka using Spark Streaming, >> and the sbt build is failing with error - >> >> [error] (*:update) sbt.ResolveException: unresolved dependency: >> org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found >> >> Scala version -> 2.10.7 >> Spark Version -> 2.1.0 >> Kafka version -> 0.9 >> sbt version -> 0.13 >> >> Contents of sbt files is as shown below -> >> >> 1) >> vi spark_kafka_code/project/plugins.sbt >> >> addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2") >> >> 2) >> vi spark_kafka_code/sparkkafka.sbt >> >> import AssemblyKeys._ >> assemblySettings >> >> name := "SparkKafka Project" >> >> version := "1.0" >> scalaVersion := "2.11.7" >> >> val sparkVers = "2.1.0" >> >> // Base Spark-provided dependencies >> libraryDependencies ++= Seq( >> "org.apache.spark" %% "spark-core" % sparkVers % "provided", >> "org.apache.spark" %% "spark-streaming" % sparkVers % "provided", >> "org.apache.spark" %% "spark-streaming-kafka" % sparkVers) >> >> mergeStrategy in assembly := { >> case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard >> case m if m.toLowerCase.startsWith("META-INF") => MergeStrategy.discard >> case "reference.conf" => MergeStrategy.concat >> case m if m.endsWith("UnusedStubClass.class") => MergeStrategy.discard >> case _ => MergeStrategy.first >> } >> >> i launch sbt, and then try to create an eclipse project, complete error >> is as shown below - >> >> --------------------- >> >> sbt >> [info] Loading global plugins from /Users/karanalang/.sbt/0.13/plugins >> [info] Loading project definition from >> /Users/karanalang/Documents/Technology/Coursera_spark_scala/ >> spark_kafka_code/project >> [info] Set current project to SparkKafka Project (in build >> file:/Users/karanalang/Documents/Technology/Coursera_spark_ >> scala/spark_kafka_code/) >> > eclipse >> [info] About to create Eclipse project files for your project(s). >> [info] Updating >> {file:/Users/karanalang/Documents/Technology/Coursera_spark_ >> scala/spark_kafka_code/}spark_kafka_code... >> [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 ... >> [warn] module not found: >> org.apache.spark#spark-streaming-kafka_2.11;2.1.0 >> [warn] ==== local: tried >> [warn] >> /Users/karanalang/.ivy2/local/org.apache.spark/spark-streami >> ng-kafka_2.11/2.1.0/ivys/ivy.xml >> [warn] ==== activator-launcher-local: tried >> [warn] >> /Users/karanalang/.activator/repository/org.apache.spark/spa >> rk-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >> [warn] ==== activator-local: tried >> [warn] >> /Users/karanalang/Documents/Technology/SCALA/activator-dist- >> 1.3.10/repository/org.apache.spark/spark-streaming-kafka_2. >> 11/2.1.0/ivys/ivy.xml >> [warn] ==== public: tried >> [warn] >> https://repo1.maven.org/maven2/org/apache/spark/spark-stream >> ing-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom >> [warn] ==== typesafe-releases: tried >> [warn] >> http://repo.typesafe.com/typesafe/releases/org/apache/spark/ >> spark-streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom >> [warn] ==== typesafe-ivy-releasez: tried >> [warn] >> http://repo.typesafe.com/typesafe/ivy-releases/org.apache. >> spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >> [info] Resolving jline#jline;2.12.1 ... >> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >> [warn] :: UNRESOLVED DEPENDENCIES :: >> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >> [warn] :: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found >> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >> [warn] >> [warn] Note: Unresolved dependencies path: >> [warn] org.apache.spark:spark-streaming-kafka_2.11:2.1.0 >> (/Users/karanalang/Documents/Technology/Coursera_spark_scala >> /spark_kafka_code/sparkkafka.sbt#L12-16) >> [warn] +- sparkkafka-project:sparkkafka-project_2.11:1.0 >> [trace] Stack trace suppressed: run last *:update for the full output. >> [error] (*:update) sbt.ResolveException: unresolved dependency: >> org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found >> [info] Updating >> {file:/Users/karanalang/Documents/Technology/Coursera_spark_ >> scala/spark_kafka_code/}spark_kafka_code... >> [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 ... >> [warn] module not found: >> org.apache.spark#spark-streaming-kafka_2.11;2.1.0 >> [warn] ==== local: tried >> [warn] >> /Users/karanalang/.ivy2/local/org.apache.spark/spark-streami >> ng-kafka_2.11/2.1.0/ivys/ivy.xml >> [warn] ==== activator-launcher-local: tried >> [warn] >> /Users/karanalang/.activator/repository/org.apache.spark/spa >> rk-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >> [warn] ==== activator-local: tried >> [warn] >> /Users/karanalang/Documents/Technology/SCALA/activator-dist- >> 1.3.10/repository/org.apache.spark/spark-streaming-kafka_2. >> 11/2.1.0/ivys/ivy.xml >> [warn] ==== public: tried >> [warn] >> https://repo1.maven.org/maven2/org/apache/spark/spark-stream >> ing-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom >> [warn] ==== typesafe-releases: tried >> [warn] >> http://repo.typesafe.com/typesafe/releases/org/apache/spark/ >> spark-streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom >> [warn] ==== typesafe-ivy-releasez: tried >> [warn] >> http://repo.typesafe.com/typesafe/ivy-releases/org.apache. >> spark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >> [info] Resolving jline#jline;2.12.1 ... >> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >> [warn] :: UNRESOLVED DEPENDENCIES :: >> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >> [warn] :: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found >> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >> [warn] >> [warn] Note: Unresolved dependencies path: >> [warn] org.apache.spark:spark-streaming-kafka_2.11:2.1.0 >> (/Users/karanalang/Documents/Technology/Coursera_spark_scala >> /spark_kafka_code/sparkkafka.sbt#L12-16) >> [warn] +- sparkkafka-project:sparkkafka-project_2.11:1.0 >> [trace] Stack trace suppressed: run last *:update for the full output. >> [error] (*:update) sbt.ResolveException: unresolved dependency: >> org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found >> [error] Could not create Eclipse project files: >> [error] Error evaluating task "scalacOptions": error >> [error] Error evaluating task "externalDependencyClasspath": error >> > >> > >
