To add to Mich, I put the build.sb under the Myproject root folder : MyProject/buildt.sbt and the assembly.sbt is placed in the folder called "project" under the MyProject folder: MyProject/project/assembly.sbt
also the the first line in build.sbt is to import the assembly keys as below: import AssemblyKeys._ assemblySettings name := "tripdata" version := "1.0" scalaVersion := "2.10.4" resolvers += "Typesafe Repository" at " http://repo.typesafe.com/typesafe/releases/"; libraryDependencies += "com.typesafe.akka" % "akka-actor" % "2.0.2" libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.0" % "provided" libraryDependencies += "com.databricks" % "spark-csv_2.10" % "1.3.0" libraryDependencies += "org.apache.spark" % "spark-sql_2.10" % "1.3.0" % "provided" scalacOptions += "-deprecation" // The following is the class that will run when the jar is compiled! mainClass in assembly := Some("TripData") ---------------------------------------------------------------- Run the command from MyProject folder as below: MyProject $sbt assembly Regards, Haroon Syed. On 4 April 2016 at 11:42, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Thanks all > > Actually I have a generic shell script that for a given Scala program > creates jar file using Maven or SBT. > > I modified that one to create a uber jar file as well using SBT assembly. > The root directory structure in uber has one more sub-directory. > > In addition SBT relies on a single sbt file. Maven relies on pom.xml. In > contrast SBT assembly requires a build.sbt file plus assembly,sbt file > > The assembly sbt file asssenly.sbt simply has one liner > > addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0") > > The buid.sbt has the following in my case (a generic one) > > lazy val root = (project in file(".")). > settings( > name := "${APPLICATION}", > version := "1.0", > scalaVersion := "2.10.4", > mainClass in Compile := Some("myPackage.${APPLICATION}") > ) > libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" % > "provided" > libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1" % > "provided" > libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" % > "provided" > libraryDependencies += "junit" % "junit" % "4.12" > libraryDependencies += "org.scala-sbt" % "test-interface" % "1.0" > libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.6.1" % > "provided" > libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka" % > "1.6.1" > // META-INF discarding > mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) => > { > case PathList("META-INF", xs @ _*) => MergeStrategy.discard > case x => MergeStrategy.first > } > } > > So if I want to create an sbt assembly jar file and assuming that the > Scala file is called TestStream_assembly then I just run the shell script > passing the Application 'A' and type 'T' to the shell > > ./generic.ksh -A TestStream_assembly -T assembly > > For simple sbt that will be > > ./generic.ksh -A TestStream -T sbt > > and for maven it will be > > ./generic.ksh -A TestStream -T mvn > > > Cheers, > > > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > > On 4 April 2016 at 16:30, vetal king <greenve...@gmail.com> wrote: > >> >> ---------- Forwarded message ---------- >> From: vetal king <greenve...@gmail.com> >> Date: Mon, Apr 4, 2016 at 8:59 PM >> Subject: Re: All inclusive uber-jar >> To: Mich Talebzadeh <mich.talebza...@gmail.com> >> >> >> Not sure how to create uber jar using sbt, but this is how you can do it >> using maven. >> >> <plugins> >> .... >> <plugin> >> <groupId>org.apache.maven.plugins</groupId> >> <artifactId>maven-shade-plugin</artifactId> >> <version>2.4.3</version> >> <executions> >> <execution> >> <phase>package</phase> >> <goals> >> <goal>shade</goal> >> </goals> >> <configuration> >> <artifactSet> >> <includes> >> * <include>*:*</include>* >> </includes> >> </artifactSet> >> </configuration> >> </execution> >> </executions> >> </plugin> >> >> </plugins> >> >> But instead of creating a uber jar, consider providing maven coordinates >> with the help of sark-submit's --packages and --repositories options >> >> >> >> On Mon, Apr 4, 2016 at 7:06 PM, Mich Talebzadeh < >> mich.talebza...@gmail.com> wrote: >> >>> Hi, >>> >>> >>> When one builds a project for Spark in this case Spark streaming with >>> SBT, as usual I add dependencies as follows: >>> >>> libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.6.1" >>> libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka" % >>> "1.6.1" >>> >>> However when I submit it through spark-submit I need to put the package >>> containing KafkaUtils the same way I do it in spark-shell >>> >>> ${SPARK_HOME}/bin/spark-submit \ >>> --jars >>> /home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar \ >>> ..... >>> >>> Now if I want to distribute this as all-in-one package so that it can be >>> run from any node, I have been told that I need to create an uber-jar. I >>> have not done this before so I assume an uber-jar will be totally self >>> contained with all the classes etc. >>> >>> Can someone elaborate on this please? >>> >>> Thanks >>> >>> Dr Mich Talebzadeh >>> >>> >>> >>> LinkedIn * >>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* >>> >>> >>> >>> http://talebzadehmich.wordpress.com >>> >>> >>> >> >> >> >
