Take a look at https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-Eclipse
On Wed, Jul 8, 2015 at 7:47 AM, Daniel Siegmann <daniel.siegm...@teamaol.com > wrote: > To set up Eclipse for Spark you should install the Scala IDE plugins: > http://scala-ide.org/download/current.html > > Define your project in Maven with Scala plugins configured (you should be > able to find documentation online) and import as an existing Maven project. > The source code should be in src/main/scala but otherwise the project > structure will be the same as you'd expect in Java. > > Nothing special is needed for Spark. Just define the desired Spark jars ( > spark-core and possibly others, such as spark-sql) in your Maven POM as > dependencies. You should scope these dependencies as provided, since they > will automatically be on the classpath when you deploy your project to a > Spark cluster. > > One thing to keep in mind is that Scala dependencies require separate jars > for different versions of Scala, and it is convention to append the Scala > version to the artifact ID. For example, if you are using Scala 2.11.x, > your dependency will be spark-core_2.11 (look on search.maven.org if > you're not sure). I think you can omit the Scala version if you're using > SBT (not sure why you would, but some people seem to prefer it). > > Unit testing Spark is briefly explained in the programming guide > <https://spark.apache.org/docs/latest/programming-guide.html#unit-testing> > . > > To deploy using spark-submit you can build the jar using mvn package if > and only if you don't have any non-Spark dependencies. Otherwise, the > simplest thing is to build a jar with dependencies (typically using the > assembly > <http://maven.apache.org/plugins/maven-assembly-plugin/single-mojo.html> > or shade <https://maven.apache.org/plugins/maven-shade-plugin/> plugins). > > > > > > On Wed, Jul 8, 2015 at 9:38 AM, Prateek . <prat...@aricent.com> wrote: > >> Hi >> >> >> >> I am beginner to scala and spark. I am trying to set up eclipse >> environment to develop spark program in scala, then take it’s jar for >> spark-submit. >> >> How shall I start? To start my task includes, setting up eclipse for >> scala and spark, getting dependencies resolved, building project using >> maven/sbt. >> >> Is there any good blog or documentation that is can follow. >> >> >> >> Thanks >> >> "DISCLAIMER: This message is proprietary to Aricent and is intended >> solely for the use of the individual to whom it is addressed. It may >> contain privileged or confidential information and should not be circulated >> or used for any purpose other than for what it is intended. If you have >> received this message in error, please notify the originator immediately. >> If you are not the intended recipient, you are notified that you are >> strictly prohibited from using, copying, altering, or disclosing the >> contents of this message. Aricent accepts no responsibility for loss or >> damage arising from the use of the information transmitted by this email >> including damage from virus." >> > >
