Hi all, Good news! I was able to create a Kinesis consumer and assemble it into an "uber jar" following http://spark.apache.org/docs/latest/streaming-kinesis-integration.html and example https://github.com/apache/spark/blob/master/extras/kinesis-asl/src/main/scala/org/apache/spark/examples/streaming/KinesisWordCountASL.scala .
However when I try to spark-submit it I get the following exception: *Exception in thread "main" java.lang.NoClassDefFoundError: com/amazonaws/auth/AWSCredentialsProvider* Do I need to include KCL dependency in *build.sbt*, here's what it looks like currently: import AssemblyKeys._ name := "Kinesis Consumer" version := "1.0" organization := "com.myconsumer" scalaVersion := "2.11.5" libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.0" % "provided" libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.3.0" % "provided" libraryDependencies += "org.apache.spark" %% "spark-streaming-kinesis-asl" % "1.3.0" % "provided" assemblySettings jarName in assembly := "consumer-assembly.jar" assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala=false) Any help appreciated. Thanks, Vadim ᐧ On Thu, Apr 2, 2015 at 1:15 PM, Kelly, Jonathan <jonat...@amazon.com> wrote: > It looks like you're attempting to mix Scala versions, so that's going > to cause some problems. If you really want to use Scala 2.11.5, you must > also use Spark package versions built for Scala 2.11 rather than 2.10. > Anyway, that's not quite the correct way to specify Scala dependencies in > build.sbt. Instead of placing the Scala version after the artifactId (like > "spark-core_2.10"), what you actually want is to use just "spark-core" with > two percent signs before it. Using two percent signs will make it use the > version of Scala that matches your declared scalaVersion. For example: > > libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.0" % > "provided" > > libraryDependencies += "org.apache.spark" %% "spark-streaming" % > "1.3.0" % "provided" > > libraryDependencies += "org.apache.spark" %% > "spark-streaming-kinesis-asl" % "1.3.0" > > I think that may get you a little closer, though I think you're probably > going to run into the same problems I ran into in this thread: > https://www.mail-archive.com/user@spark.apache.org/msg23891.html I never > really got an answer for that, and I temporarily moved on to other things > for now. > > > ~ Jonathan Kelly > > From: 'Vadim Bichutskiy' <vadim.bichuts...@gmail.com> > Date: Thursday, April 2, 2015 at 9:53 AM > To: "user@spark.apache.org" <user@spark.apache.org> > Subject: Spark + Kinesis > > Hi all, > > I am trying to write an Amazon Kinesis consumer Scala app that processes > data in the > Kinesis stream. Is this the correct way to specify *build.sbt*: > > ------- > *import AssemblyKeys._* > *name := "Kinesis Consumer"* > > > > > > > *version := "1.0" organization := "com.myconsumer" scalaVersion := > "2.11.5" libraryDependencies ++= Seq("org.apache.spark" % "spark-core_2.10" > % "1.3.0" % "provided", "org.apache.spark" % "spark-streaming_2.10" % > "1.3.0" "org.apache.spark" % "spark-streaming-kinesis-asl_2.10" % "1.3.0")* > > > > * assemblySettings jarName in assembly := "consumer-assembly.jar" > assemblyOption in assembly := (assemblyOption in > assembly).value.copy(includeScala=false)* > -------- > > In *project/assembly.sbt* I have only the following line: > > *addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.13.0")* > > I am using sbt 0.13.7. I adapted Example 7.7 in the Learning Spark book. > > Thanks, > Vadim > >
