Thanks for the tips!
Yeah its as a working SBT project. I.e. if I do an SBT run it picks up
Test1 as a main class and runs it for me without error. Its only in
IntelliJ. I opened the project from the folder afresh by choosing the
build.sbt file. I re-tested by deleting .idea and just choosing the project
folder and I get the same result. Not using gen-idea in sbt.
On Wed, Jan 14, 2015 at 8:52 AM, Jay Vyas <jayunit100.apa...@gmail.com>
wrote:
> I find importing a working SBT project into IntelliJ is the way to go.....
>
> How did you load the project into intellij?
>
> On Jan 13, 2015, at 4:45 PM, Enno Shioji <eshi...@gmail.com> wrote:
>
> Had the same issue. I can't remember what the issue was but this works:
>
> libraryDependencies ++= {
> val sparkVersion = "1.2.0"
> Seq(
> "org.apache.spark" %% "spark-core" % sparkVersion % "provided",
> "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided",
> "org.apache.spark" %% "spark-streaming-twitter" % sparkVersion %
> "provided",
> "org.apache.spark" %% "spark-streaming-kafka" % sparkVersion %
> "provided",
> "javax.servlet" % "javax.servlet-api" % "3.0.1" % "provided"
> )
> }
>
> In order to run classes in "main" source in Intellij, you must invoke it
> from a source under "test" as Intellij won't provide the "provided" scope
> libraries when running code in "main" source (but it will for sources under
> "test").
>
> With this config you can "sbt assembly" in order to get the fat jar
> without Spark jars.
>
>
> ᐧ
>
> On Tue, Jan 13, 2015 at 12:16 PM, Night Wolf <nightwolf...@gmail.com>
> wrote:
>
>> Hi,
>>
>> I'm trying to load up an SBT project in IntelliJ 14 (windows) running 1.7
>> JDK, SBT 0.13.5 -I seem to be getting errors with the project.
>>
>> The build.sbt file is super simple;
>>
>> name := "scala-spark-test1"
>> version := "1.0"
>>
>> scalaVersion := "2.10.4"
>>
>> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.2.0"
>>
>>
>> Then I have a super simple test class;
>>
>> package test
>>
>> import org.apache.spark.{SparkContext, SparkConf}
>>
>> case class Blah(s: Int, d: String)
>>
>> object Test1 {
>> def main(args: Array[String]): Unit = {
>> val sparkconf = new
>> SparkConf().setMaster("local[4]").setAppName("test-spark")
>> val sc = new SparkContext(sparkconf)
>>
>> val rdd = sc.parallelize(Seq(
>> Blah(1,"dsdsd"),
>> Blah(2,"daaa"),
>> Blah(3,"dhghghgh")
>> ))
>>
>> rdd.collect().foreach(println)
>>
>> }
>> }
>>
>>
>> When I try to run the Test1 object in IntelliJ I get the following error;
>>
>> Exception in thread "main" java.lang.NoClassDefFoundError:
>> javax/servlet/http/HttpServletResponse
>> at org.apache.spark.HttpServer.org
>> $apache$spark$HttpServer$$doStart(HttpServer.scala:73)
>> at org.apache.spark.HttpServer$$anonfun$1.apply(HttpServer.scala:60)
>> at org.apache.spark.HttpServer$$anonfun$1.apply(HttpServer.scala:60)
>> at
>> org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1676)
>> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
>> at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1667)
>> at org.apache.spark.HttpServer.start(HttpServer.scala:60)
>> at org.apache.spark.HttpFileServer.initialize(HttpFileServer.scala:45)
>> at org.apache.spark.SparkEnv$.create(SparkEnv.scala:304)
>> at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:159)
>> at org.apache.spark.SparkContext.<init>(SparkContext.scala:232)
>> at test.Test1$.main(Test1.scala:10)
>> at test.Test1.main(Test1.scala)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
>> Caused by: java.lang.ClassNotFoundException:
>> javax.servlet.http.HttpServletResponse
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> ... 18 more
>>
>>
>> For whatever reason it seems that IntelliJ isnt pulling in these deps.
>> Doing an sbt run works fine. Looking at the project structure it seems that
>> 7 libs dont get marked as a dependency for my module... But they are on the
>> dep tree.... http://pastebin.com/REkQh5ux
>>
>> <image.png>
>>
>> Is this something to do with the libs and scoping or shading in Spark and
>> its associated libs? Has anyone else seen this issue?
>>
>> Cheers,
>> NW
>>
>
>
