Thanks all
Actually I have a generic shell script that for a given Scala program
creates jar file using Maven or SBT.
I modified that one to create a uber jar file as well using SBT assembly.
The root directory structure in uber has one more sub-directory.
In addition SBT relies on a single sbt file. Maven relies on pom.xml. In
contrast SBT assembly requires a build.sbt file plus assembly,sbt file
The assembly sbt file asssenly.sbt simply has one liner
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.12.0")
The buid.sbt has the following in my case (a generic one)
lazy val root = (project in file(".")).
settings(
name := "${APPLICATION}",
version := "1.0",
scalaVersion := "2.10.4",
mainClass in Compile := Some("myPackage.${APPLICATION}")
)
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" %
"provided"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1" %
"provided"
libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" %
"provided"
libraryDependencies += "junit" % "junit" % "4.12"
libraryDependencies += "org.scala-sbt" % "test-interface" % "1.0"
libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.6.1" %
"provided"
libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka" %
"1.6.1"
// META-INF discarding
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("META-INF", xs @ _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
}
So if I want to create an sbt assembly jar file and assuming that the Scala
file is called TestStream_assembly then I just run the shell script passing
the Application 'A' and type 'T' to the shell
./generic.ksh -A TestStream_assembly -T assembly
For simple sbt that will be
./generic.ksh -A TestStream -T sbt
and for maven it will be
./generic.ksh -A TestStream -T mvn
Cheers,
Dr Mich Talebzadeh
LinkedIn *
https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
http://talebzadehmich.wordpress.com
On 4 April 2016 at 16:30, vetal king <[email protected]> wrote:
>
> ---------- Forwarded message ----------
> From: vetal king <[email protected]>
> Date: Mon, Apr 4, 2016 at 8:59 PM
> Subject: Re: All inclusive uber-jar
> To: Mich Talebzadeh <[email protected]>
>
>
> Not sure how to create uber jar using sbt, but this is how you can do it
> using maven.
>
> <plugins>
> ....
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-shade-plugin</artifactId>
> <version>2.4.3</version>
> <executions>
> <execution>
> <phase>package</phase>
> <goals>
> <goal>shade</goal>
> </goals>
> <configuration>
> <artifactSet>
> <includes>
> * <include>*:*</include>*
> </includes>
> </artifactSet>
> </configuration>
> </execution>
> </executions>
> </plugin>
>
> </plugins>
>
> But instead of creating a uber jar, consider providing maven coordinates
> with the help of sark-submit's --packages and --repositories options
>
>
>
> On Mon, Apr 4, 2016 at 7:06 PM, Mich Talebzadeh <[email protected]
> > wrote:
>
>> Hi,
>>
>>
>> When one builds a project for Spark in this case Spark streaming with
>> SBT, as usual I add dependencies as follows:
>>
>> libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.6.1"
>> libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka" %
>> "1.6.1"
>>
>> However when I submit it through spark-submit I need to put the package
>> containing KafkaUtils the same way I do it in spark-shell
>>
>> ${SPARK_HOME}/bin/spark-submit \
>> --jars
>> /home/hduser/jars/spark-streaming-kafka-assembly_2.10-1.6.1.jar \
>> .....
>>
>> Now if I want to distribute this as all-in-one package so that it can be
>> run from any node, I have been told that I need to create an uber-jar. I
>> have not done this before so I assume an uber-jar will be totally self
>> contained with all the classes etc.
>>
>> Can someone elaborate on this please?
>>
>> Thanks
>>
>> Dr Mich Talebzadeh
>>
>>
>>
>> LinkedIn *
>> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
>> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>>
>>
>>
>> http://talebzadehmich.wordpress.com
>>
>>
>>
>
>
>
