Hi Jozef - i do have a additional basic question ..

When i tried to compile the code in Eclipse, i was not able to do that

eg.
import org.apache.spark.streaming.kafka.KafkaUtils

gave errors saying KafaUtils was not part of the package.
However, when i used sbt to compile - the compilation went through fine

So, I assume additional libraries are being downloaded when i provide the
appropriate packages in LibraryDependencies ?
which ones would have helped compile this ?


On Sat, Jun 17, 2017 at 2:52 PM, karan alang <karan.al...@gmail.com> wrote:

> Hey Jozef,
>
> Thanks for the quick response ..
> yes, you are right .. spark-sql dependency was missing .. added that & it
> worked fine.
>
> regds,
> Karan Alang
>
> On Sat, Jun 17, 2017 at 2:24 PM, Jozef.koval <jozef.ko...@protonmail.ch>
> wrote:
>
>> Hey Karan,
>> I believe you are missing spark-sql dependency.
>>
>> Jozef
>>
>> Sent from ProtonMail <https://protonmail.ch>, encrypted email based in
>> Switzerland.
>>
>>
>> -------- Original Message --------
>> Subject: Re: Kafka-Spark Integration - build failing with sbt
>> Local Time: June 17, 2017 10:52 PM
>> UTC Time: June 17, 2017 8:52 PM
>> From: karan.al...@gmail.com
>> To: users@kafka.apache.org, Jozef.koval <jozef.ko...@protonmail.ch>
>>
>>
>> Thanks, i was able to get this working.
>> here is what i added in build.sbt file
>> ------------------------------------------------------------
>> ----------------------------------
>>
>> scalaVersion := "2.11.7"
>>
>> val sparkVers = "2.1.0"
>>
>> // Base Spark-provided dependencies
>>
>> libraryDependencies ++= Seq(
>>
>>   "org.apache.spark" %% "spark-core" % sparkVers % "provided",
>>
>>   "org.apache.spark" %% "spark-streaming" % sparkVers % "provided",
>>
>>   "org.apache.spark" % "spark-streaming-kafka-0-8_2.11" % sparkVers)
>>
>> ------------------------------------------------------------
>> ------------------------------------
>>
>> However, i'm running into addition issue when compiling the file using
>> sbt .. it gives errors as shown below :
>>
>> *Pls note - I've added the jars in eclipse, and it works .. however when
>> i use sbt to compile, it is failing.*
>>
>> *What needs to be done ? *
>>
>> *I've also tried added the jars in CLASSPATH, but still get the same
>> error. *
>>
>> ------------------------------------------------------------
>> -------------------------------------
>>
>> [info] Compiling 1 Scala source to /Users/karanalang/Documents/Te
>> chnology/Coursera_spark_scala/spark_kafka_code/target/scala-
>> 2.11/classes...
>>
>> [error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/
>> spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:6: object
>> SQLContext is not a member of package org.apache.spark.sql
>>
>> [error] import org.apache.spark.sql.SQLContext
>>
>> [error]        ^
>>
>> [error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/
>> spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:10:
>> object SparkSession is not a member of package org.apache.spark.sql
>>
>> [error] import org.apache.spark.sql.SparkSession
>>
>> [error]        ^
>>
>> [error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/
>> spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:11:
>> object types is not a member of package org.apache.spark.sql
>>
>> [error] import org.apache.spark.sql.types.StructField
>>
>> [error]                             ^
>>
>> [error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/
>> spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:12:
>> object types is not a member of package org.apache.spark.sql
>>
>> [error] import org.apache.spark.sql.types.StringType
>>
>> [error]                             ^
>>
>> [error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/
>> spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:13:
>> object types is not a member of package org.apache.spark.sql
>>
>> [error] import org.apache.spark.sql.types.StructType
>>
>> [error]                             ^
>>
>> [error] /Users/karanalang/Documents/Technology/Coursera_spark_scala/
>> spark_kafka_code/src/main/scala/spark/kafka/SparkKafkaDS.scala:14:
>> object Row is not a member of package org.apache.spark.sql
>>
>> [error] import org.apache.spark.sql.Row
>>
>> [error]        ^
>>
>>
>>
>> On Sat, Jun 17, 2017 at 12:35 AM, Jozef.koval <jozef.ko...@protonmail.ch>
>> wrote:
>>
>>> Hi Karan,
>>>
>>> spark-streaming-kafka is for old spark (version < 1.6.3)
>>> spark-streaming-kafka-0.8 is for current spark (version > 2.0)
>>>
>>> Jozef
>>>
>>> n.b. there is also version for kafka 0.10+ see [this](
>>> https://spark.apache.org/docs/latest/streaming-kafka-integration.html)
>>>
>>> Sent from [ProtonMail](https://protonmail.ch), encrypted email based in
>>> Switzerland.
>>>
>>>
>>> -------- Original Message --------
>>> Subject: Kafka-Spark Integration - build failing with sbt
>>> Local Time: June 17, 2017 1:50 AM
>>> UTC Time: June 16, 2017 11:50 PM
>>> From: karan.al...@gmail.com
>>> To: users@kafka.apache.org
>>>
>>> I"m trying to compile kafka & Spark Streaming integration code i.e.
>>> reading
>>> from Kafka using Spark Streaming,
>>> and the sbt build is failing with error -
>>>
>>> [error] (*:update) sbt.ResolveException: unresolved dependency:
>>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found
>>>
>>> Scala version -> 2.10.7
>>> Spark Version -> 2.1.0
>>> Kafka version -> 0.9
>>> sbt version -> 0.13
>>>
>>> Contents of sbt files is as shown below ->
>>>
>>> 1)
>>> vi spark_kafka_code/project/plugins.sbt
>>>
>>> addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")
>>>
>>> 2)
>>> vi spark_kafka_code/sparkkafka.sbt
>>>
>>> import AssemblyKeys._
>>> assemblySettings
>>>
>>> name := "SparkKafka Project"
>>>
>>> version := "1.0"
>>> scalaVersion := "2.11.7"
>>>
>>> val sparkVers = "2.1.0"
>>>
>>> // Base Spark-provided dependencies
>>> libraryDependencies ++= Seq(
>>> "org.apache.spark" %% "spark-core" % sparkVers % "provided",
>>> "org.apache.spark" %% "spark-streaming" % sparkVers % "provided",
>>> "org.apache.spark" %% "spark-streaming-kafka" % sparkVers)
>>>
>>> mergeStrategy in assembly := {
>>> case m if m.toLowerCase.endsWith("manifest.mf") => MergeStrategy.discard
>>> case m if m.toLowerCase.startsWith("META-INF") => MergeStrategy.discard
>>> case "reference.conf" => MergeStrategy.concat
>>> case m if m.endsWith("UnusedStubClass.class") => MergeStrategy.discard
>>> case _ => MergeStrategy.first
>>> }
>>>
>>> i launch sbt, and then try to create an eclipse project, complete error
>>> is as shown below -
>>>
>>> ---------------------
>>>
>>> sbt
>>> [info] Loading global plugins from /Users/karanalang/.sbt/0.13/plugins
>>> [info] Loading project definition from
>>> /Users/karanalang/Documents/Technology/Coursera_spark_scala/
>>> spark_kafka_code/project
>>> [info] Set current project to SparkKafka Project (in build
>>> file:/Users/karanalang/Documents/Technology/Coursera_spark_s
>>> cala/spark_kafka_code/)
>>> > eclipse
>>> [info] About to create Eclipse project files for your project(s).
>>> [info] Updating
>>> {file:/Users/karanalang/Documents/Technology/Coursera_spark_
>>> scala/spark_kafka_code/}spark_kafka_code...
>>> [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 ...
>>> [warn] module not found:
>>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0
>>> [warn] ==== local: tried
>>> [warn]
>>> /Users/karanalang/.ivy2/local/org.apache.spark/spark-streami
>>> ng-kafka_2.11/2.1.0/ivys/ivy.xml
>>> [warn] ==== activator-launcher-local: tried
>>> [warn]
>>> /Users/karanalang/.activator/repository/org.apache.spark/spa
>>> rk-streaming-kafka_2.11/2.1.0/ivys/ivy.xml
>>> [warn] ==== activator-local: tried
>>> [warn]
>>> /Users/karanalang/Documents/Technology/SCALA/activator-dist-
>>> 1.3.10/repository/org.apache.spark/spark-streaming-kafka_2.1
>>> 1/2.1.0/ivys/ivy.xml
>>> [warn] ==== public: tried
>>> [warn]
>>> https://repo1.maven.org/maven2/org/apache/spark/spark-stream
>>> ing-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom
>>> [warn] ==== typesafe-releases: tried
>>> [warn]
>>> http://repo.typesafe.com/typesafe/releases/org/apache/spark/
>>> spark-streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom
>>> [warn] ==== typesafe-ivy-releasez: tried
>>> [warn]
>>> http://repo.typesafe.com/typesafe/ivy-releases/org.apache.sp
>>> ark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml
>>> [info] Resolving jline#jline;2.12.1 ...
>>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>> [warn] :: UNRESOLVED DEPENDENCIES ::
>>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>> [warn] :: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found
>>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>> [warn]
>>> [warn] Note: Unresolved dependencies path:
>>> [warn] org.apache.spark:spark-streaming-kafka_2.11:2.1.0
>>> (/Users/karanalang/Documents/Technology/Coursera_spark_scala
>>> /spark_kafka_code/sparkkafka.sbt#L12-16)
>>> [warn] +- sparkkafka-project:sparkkafka-project_2.11:1.0
>>> [trace] Stack trace suppressed: run last *:update for the full output.
>>> [error] (*:update) sbt.ResolveException: unresolved dependency:
>>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found
>>> [info] Updating
>>> {file:/Users/karanalang/Documents/Technology/Coursera_spark_
>>> scala/spark_kafka_code/}spark_kafka_code...
>>> [info] Resolving org.apache.spark#spark-streaming-kafka_2.11;2.1.0 ...
>>> [warn] module not found:
>>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0
>>> [warn] ==== local: tried
>>> [warn]
>>> /Users/karanalang/.ivy2/local/org.apache.spark/spark-streami
>>> ng-kafka_2.11/2.1.0/ivys/ivy.xml
>>> [warn] ==== activator-launcher-local: tried
>>> [warn]
>>> /Users/karanalang/.activator/repository/org.apache.spark/spa
>>> rk-streaming-kafka_2.11/2.1.0/ivys/ivy.xml
>>> [warn] ==== activator-local: tried
>>> [warn]
>>> /Users/karanalang/Documents/Technology/SCALA/activator-dist-
>>> 1.3.10/repository/org.apache.spark/spark-streaming-kafka_2.1
>>> 1/2.1.0/ivys/ivy.xml
>>> [warn] ==== public: tried
>>> [warn]
>>> https://repo1.maven.org/maven2/org/apache/spark/spark-stream
>>> ing-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom
>>> [warn] ==== typesafe-releases: tried
>>> [warn]
>>> http://repo.typesafe.com/typesafe/releases/org/apache/spark/
>>> spark-streaming-kafka_2.11/2.1.0/spark-streaming-kafka_2.11-2.1.0.pom
>>> [warn] ==== typesafe-ivy-releasez: tried
>>> [warn]
>>> http://repo.typesafe.com/typesafe/ivy-releases/org.apache.sp
>>> ark/spark-streaming-kafka_2.11/2.1.0/ivys/ivy.xml
>>> [info] Resolving jline#jline;2.12.1 ...
>>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>> [warn] :: UNRESOLVED DEPENDENCIES ::
>>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>> [warn] :: org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found
>>> [warn] ::::::::::::::::::::::::::::::::::::::::::::::
>>> [warn]
>>> [warn] Note: Unresolved dependencies path:
>>> [warn] org.apache.spark:spark-streaming-kafka_2.11:2.1.0
>>> (/Users/karanalang/Documents/Technology/Coursera_spark_scala
>>> /spark_kafka_code/sparkkafka.sbt#L12-16)
>>> [warn] +- sparkkafka-project:sparkkafka-project_2.11:1.0
>>> [trace] Stack trace suppressed: run last *:update for the full output.
>>> [error] (*:update) sbt.ResolveException: unresolved dependency:
>>> org.apache.spark#spark-streaming-kafka_2.11;2.1.0: not found
>>> [error] Could not create Eclipse project files:
>>> [error] Error evaluating task "scalacOptions": error
>>> [error] Error evaluating task "externalDependencyClasspath": error
>>> >
>>>
>>
>>
>

Reply via email to