Hi Jozef - i do have a additional basic question .. When i tried to compile the code in Eclipse, i was not able to do that
eg. import org.apache.spark.streaming.kafka.KafkaUtils gave errors saying KafaUtils was not part of the package. However, when i used sbt to compile - the compilation went through fine So, I assume additional libraries are being downloaded when i provide the appropriate packages in LibraryDependencies ? which ones would have helped compile this ? On Sat, Jun 17, 2017 at 2:52 PM, karan alang <karan.al...@gmail.com> wrote: > Hey Jozef, > > Thanks for the quick response .. > yes, you are right .. spark-sql dependency was missing .. added that & it > worked fine. > > regds, > Karan Alang > > On Sat, Jun 17, 2017 at 2:24 PM, Jozef.koval <jozef.ko...@protonmail.ch> > wrote: > >> Hey Karan, >> I believe you are missing spark-sql dependency. >> >> Jozef >> >> Sent from ProtonMail <https://protonmail.ch>, encrypted email based in >> Switzerland. >> >> >> -------- Original Message -------- >> Subject: Re: Kafka-Spark Integration - build failing with sbt >> Local Time: June 17, 2017 10:52 PM >> UTC Time: June 17, 2017 8:52 PM >> From: karan.al...@gmail.com >> To: users@kafka.apache.org, Jozef.koval <jozef.ko...@protonmail.ch> >> >> >> Thanks, i was able to get this working. >> here is what i added in build.sbt file >> ------------------------------------------------------------ >> ---------------------------------- >> >> scalaVersion := "2.11.7" >> >> val sparkVers = "2.1.0" >> >> // Base Spark-provided dependencies >> >> libraryDependencies ++= Seq( >> >> "org.apache.spark" %% "spark-core" % sparkVers % "provided", >> >> "org.apache.spark" %% "spark-streaming" % sparkVers % "provided", >> >> "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % sparkVers) >> >> ------------------------------------------------------------ >> ------------------------------------ >> >> However, i'm running into addition issue when compiling the file using >> sbt .. it gives errors as shown below : >> >> *Pls note - I've added the jars in eclipse, and it works .. however when >> i use sbt to compile, it is failing.* >> >> *What needs to be done ? * >> >> *I've also tried added the jars in CLASSPATH, but still get the same >> error. * >> >> ------------------------------------------------------------ >> ------------------------------------- >> >> [info] Compiling 1 Scala source to /Users/karanalang/Documents/Te >> chnology/Coursera_spark_scala/spark_kafka_code/target/scala- >> 2.11/classes... >> >> [error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/ >> spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:6: object >> SQLContext is not a member of package org.apache.spark.sql >> >> [error] import org.apache.spark.sql.SQLContext >> >> [error] ^ >> >> [error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/ >> spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:10: >> object SparkSession is not a member of package org.apache.spark.sql >> >> [error] import org.apache.spark.sql.SparkSession >> >> [error] ^ >> >> [error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/ >> spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:11: >> object types is not a member of package org.apache.spark.sql >> >> [error] import org.apache.spark.sql.types.StructField >> >> [error] ^ >> >> [error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/ >> spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:12: >> object types is not a member of package org.apache.spark.sql >> >> [error] import org.apache.spark.sql.types.StringType >> >> [error] ^ >> >> [error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/ >> spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:13: >> object types is not a member of package org.apache.spark.sql >> >> [error] import org.apache.spark.sql.types.StructType >> >> [error] ^ >> >> [error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/ >> spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:14: >> object Row is not a member of package org.apache.spark.sql >> >> [error] import org.apache.spark.sql.Row >> >> [error] ^ >> >> >> >> On Sat, Jun 17, 2017 at 12:35 AM, Jozef.koval <jozef.ko...@protonmail.ch> >> wrote: >> >>> Hi Karan, >>> >>> spark-streaming-kafka is for old spark (version < 1.6.3) >>> spark-streaming-kafka-0.8 is for current spark (version > 2.0) >>> >>> Jozef >>> >>> n.b. there is also version for kafka 0.10+ see [this]( >>> https://spark.apache.org/docs/latest/streaming-kafka-integration.html) >>> >>> Sent from [ProtonMail](https://protonmail.ch), encrypted email based in >>> Switzerland. >>> >>> >>> -------- Original Message -------- >>> Subject: Kafka-Spark Integration - build failing with sbt >>> Local Time: June 17, 2017 1:50 AM >>> UTC Time: June 16, 2017 11:50 PM >>> From: karan.al...@gmail.com >>> To: users@kafka.apache.org >>> >>> I"m trying to compile kafka & Spark Streaming integration code i.e. >>> reading >>> from Kafka using Spark Streaming, >>> and the sbt build is failing with error - >>> >>> [error] (*:update) sbt.ResolveException: unresolved dependency: >>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found >>> >>> Scala version -> 2.10.7 >>> Spark Version -> 2.1.0 >>> Kafka version -> 0.9 >>> sbt version -> 0.13 >>> >>> Contents of sbt files is as shown below -> >>> >>> 1) >>> vi spark_kafka_code/project/plugins.sbt >>> >>> addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2") >>> >>> 2) >>> vi spark_kafka_code/sparkkafka.sbt >>> >>> import AssemblyKeys._ >>> assemblySettings >>> >>> name := "SparkKafka Project" >>> >>> version := "1.0" >>> scalaVersion := "2.11.7" >>> >>> val sparkVers = "2.1.0" >>> >>> // Base Spark-provided dependencies >>> libraryDependencies ++= Seq( >>> "org.apache.spark" %% "spark-core" % sparkVers % "provided", >>> "org.apache.spark" %% "spark-streaming" % sparkVers % "provided", >>> "org.apache.spark" %% "spark-streaming-kafka" % sparkVers) >>> >>> mergeStrategy in assembly := { >>> case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard >>> case m if m.toLowerCase.startsWith("META-INF") => MergeStrategy.discard >>> case "reference.conf" => MergeStrategy.concat >>> case m if m.endsWith("UnusedStubClass.class") => MergeStrategy.discard >>> case _ => MergeStrategy.first >>> } >>> >>> i launch sbt, and then try to create an eclipse project, complete error >>> is as shown below - >>> >>> --------------------- >>> >>> sbt >>> [info] Loading global plugins from /Users/karanalang/.sbt/0.13/plugins >>> [info] Loading project definition from >>> /Users/karanalang/Documents/Technology/Coursera_spark_scala/ >>> spark_kafka_code/project >>> [info] Set current project to SparkKafka Project (in build >>> file:/Users/karanalang/Documents/Technology/Coursera_spark_s >>> cala/spark_kafka_code/) >>> > eclipse >>> [info] About to create Eclipse project files for your project(s). >>> [info] Updating >>> {file:/Users/karanalang/Documents/Technology/Coursera_spark_ >>> scala/spark_kafka_code/}spark_kafka_code... >>> [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 ... >>> [warn] module not found: >>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0 >>> [warn] ==== local: tried >>> [warn] >>> /Users/karanalang/.ivy2/local/org.apache.spark/spark-streami >>> ng-kafka_2.11/2.1.0/ivys/ivy.xml >>> [warn] ==== activator-launcher-local: tried >>> [warn] >>> /Users/karanalang/.activator/repository/org.apache.spark/spa >>> rk-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >>> [warn] ==== activator-local: tried >>> [warn] >>> /Users/karanalang/Documents/Technology/SCALA/activator-dist- >>> 1.3.10/repository/org.apache.spark/spark-streaming-kafka_2.1 >>> 1/2.1.0/ivys/ivy.xml >>> [warn] ==== public: tried >>> [warn] >>> https://repo1.maven.org/maven2/org/apache/spark/spark-stream >>> ing-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom >>> [warn] ==== typesafe-releases: tried >>> [warn] >>> http://repo.typesafe.com/typesafe/releases/org/apache/spark/ >>> spark-streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom >>> [warn] ==== typesafe-ivy-releasez: tried >>> [warn] >>> http://repo.typesafe.com/typesafe/ivy-releases/org.apache.sp >>> ark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >>> [info] Resolving jline#jline;2.12.1 ... >>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >>> [warn] :: UNRESOLVED DEPENDENCIES :: >>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >>> [warn] :: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found >>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >>> [warn] >>> [warn] Note: Unresolved dependencies path: >>> [warn] org.apache.spark:spark-streaming-kafka_2.11:2.1.0 >>> (/Users/karanalang/Documents/Technology/Coursera_spark_scala >>> /spark_kafka_code/sparkkafka.sbt#L12-16) >>> [warn] +- sparkkafka-project:sparkkafka-project_2.11:1.0 >>> [trace] Stack trace suppressed: run last *:update for the full output. >>> [error] (*:update) sbt.ResolveException: unresolved dependency: >>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found >>> [info] Updating >>> {file:/Users/karanalang/Documents/Technology/Coursera_spark_ >>> scala/spark_kafka_code/}spark_kafka_code... >>> [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 ... >>> [warn] module not found: >>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0 >>> [warn] ==== local: tried >>> [warn] >>> /Users/karanalang/.ivy2/local/org.apache.spark/spark-streami >>> ng-kafka_2.11/2.1.0/ivys/ivy.xml >>> [warn] ==== activator-launcher-local: tried >>> [warn] >>> /Users/karanalang/.activator/repository/org.apache.spark/spa >>> rk-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >>> [warn] ==== activator-local: tried >>> [warn] >>> /Users/karanalang/Documents/Technology/SCALA/activator-dist- >>> 1.3.10/repository/org.apache.spark/spark-streaming-kafka_2.1 >>> 1/2.1.0/ivys/ivy.xml >>> [warn] ==== public: tried >>> [warn] >>> https://repo1.maven.org/maven2/org/apache/spark/spark-stream >>> ing-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom >>> [warn] ==== typesafe-releases: tried >>> [warn] >>> http://repo.typesafe.com/typesafe/releases/org/apache/spark/ >>> spark-streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom >>> [warn] ==== typesafe-ivy-releasez: tried >>> [warn] >>> http://repo.typesafe.com/typesafe/ivy-releases/org.apache.sp >>> ark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml >>> [info] Resolving jline#jline;2.12.1 ... >>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >>> [warn] :: UNRESOLVED DEPENDENCIES :: >>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >>> [warn] :: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found >>> [warn] :::::::::::::::::::::::::::::::::::::::::::::: >>> [warn] >>> [warn] Note: Unresolved dependencies path: >>> [warn] org.apache.spark:spark-streaming-kafka_2.11:2.1.0 >>> (/Users/karanalang/Documents/Technology/Coursera_spark_scala >>> /spark_kafka_code/sparkkafka.sbt#L12-16) >>> [warn] +- sparkkafka-project:sparkkafka-project_2.11:1.0 >>> [trace] Stack trace suppressed: run last *:update for the full output. >>> [error] (*:update) sbt.ResolveException: unresolved dependency: >>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found >>> [error] Could not create Eclipse project files: >>> [error] Error evaluating task "scalacOptions": error >>> [error] Error evaluating task "externalDependencyClasspath": error >>> > >>> >> >> >