An observation Once compiled with MVN the job submit works as follows:
+ /usr/lib/spark-1.5.2-bin-hadoop2.6/bin/spark-submit --packages com.databricks:spark-csv_2.11:1.3.0 --class ImportCSV --master spark:// 50.140.197.217:7077 --executor-memory=12G --executor-cores=12 --num-executors=2 *target/scala-1.0.jar* With sbt it takes this form + /usr/lib/spark-1.5.2-bin-hadoop2.6/bin/spark-submit --packages com.databricks:spark-csv_2.11:1.3.0 --class ImportCSV --master spark:// 50.140.197.217:7077 --executor-memory=12G --executor-cores=12 --num-executors=2 *target/scala-2.10/importcsv_2.10-1.0.jar* They both return the same results. However, why mvnjar file name is different (may be a naive question!)? thanks Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 15 March 2016 at 22:43, Mich Talebzadeh <mich.talebza...@gmail.com> wrote: > Many thanks Ted and thanks for heads up Jakob > > Just these two changes to dependencies > > <dependency> > <groupId>org.apache.spark</groupId> > <artifactId>spark-core*_2.10*</artifactId> > <version>1.5.1</version> > </dependency> > <dependency> > <groupId>org.apache.spark</groupId> > <artifactId>spark-sql*_2.10*</artifactId> > <version>1.5.1</version> > </dependency> > > > [DEBUG] endProcessChildren: artifact=spark:scala:jar:1.0 > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD SUCCESS > [INFO] > ------------------------------------------------------------------------ > [INFO] Total time: 01:04 min > [INFO] Finished at: 2016-03-15T22:55:08+00:00 > [INFO] Final Memory: 32M/1089M > [INFO] > ------------------------------------------------------------------------ > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > > > On 15 March 2016 at 22:18, Jakob Odersky <ja...@odersky.com> wrote: > >> Hi Mich, >> probably unrelated to the current error you're seeing, however the >> following dependencies will bite you later: >> <artifactId>spark-hive_2.10</artifactId> >> <artifactId>spark-csv_2.11</artifactId> >> the problem here is that you're using libraries built for different >> Scala binary versions (the numbers after the underscore). The simple >> fix here is to specify the Scala binary version you're project builds >> for (2.10 in your case, however note that version is EOL, you should >> upgrade to scala 2.11.8 if possible). >> >> On a side note, sbt takes care of handling correct scala versions for >> you (the double %% actually is a shorthand for appending >> "_scalaBinaryVersion" to your dependency). It also enables you to >> build and publish your project seamlessly against multiple versions. I >> would strongly recommend to use it in Scala projects. >> >> cheers, >> --Jakob >> >> >> >> On Tue, Mar 15, 2016 at 3:08 PM, Mich Talebzadeh >> <mich.talebza...@gmail.com> wrote: >> > Hi, >> > >> > I normally use sbt and using this sbt file works fine for me >> > >> > cat ImportCSV.sbt >> > name := "ImportCSV" >> > version := "1.0" >> > scalaVersion := "2.10.4" >> > libraryDependencies += "org.apache.spark" %% "spark-core" % "1.5.1" >> > libraryDependencies += "org.apache.spark" %% "spark-sql" % "1.5.1" >> > libraryDependencies += "org.apache.spark" %% "spark-hive" % "1.5.1" >> > libraryDependencies += "com.databricks" % "spark-csv_2.11" % "1.3.0" >> > >> > This is my first trial using Mavan and pom >> > >> > >> > my pom.xml file looks like this but throws error at build >> > >> > >> > [DEBUG] com.univocity:univocity-parsers:jar:1.5.1:compile >> > [INFO] >> > ------------------------------------------------------------------------ >> > [INFO] BUILD FAILURE >> > [INFO] >> > ------------------------------------------------------------------------ >> > [INFO] Total time: 1.326 s >> > [INFO] Finished at: 2016-03-15T22:17:29+00:00 >> > [INFO] Final Memory: 14M/455M >> > [INFO] >> > ------------------------------------------------------------------------ >> > [ERROR] Failed to execute goal on project scala: Could not resolve >> > dependencies for project spark:scala:jar:1.0: The following artifacts >> could >> > not be resolved: org.apache.spark:spark-core:jar:1.5.1, >> > org.apache.spark:spark-sql:jar:1.5.1: Failure to find >> > org.apache.spark:spark-core:jar:1.5.1 in >> > https://repo.maven.apache.org/maven2 was cached in the local >> repository, >> > resolution will not be reattempted until the update interval of central >> has >> > elapsed or updates are forced -> [Help 1] >> > >> > >> > My pom file is >> > >> > >> > cat pom.xml >> > <project xmlns="http://maven.apache.org/POM/4.0.0"; >> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >> > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 >> > http://maven.apache.org/maven-v4_0_0.xsd";> >> > <modelVersion>4.0.0</modelVersion> >> > <groupId>spark</groupId> >> > <version>1.0</version> >> > <name>${project.artifactId}</name> >> > >> > <properties> >> > <maven.compiler.source>1.7</maven.compiler.source> >> > <maven.compiler.target>1.7</maven.compiler.target> >> > <encoding>UTF-8</encoding> >> > <scala.version>2.10.4</scala.version> >> > <maven-scala-plugin.version>2.15.2</maven-scala-plugin.version> >> > </properties> >> > >> > <dependencies> >> > <dependency> >> > <groupId>org.scala-lang</groupId> >> > <artifactId>scala-library</artifactId> >> > <version>2.10.2</version> >> > </dependency> >> > <dependency> >> > <groupId>org.apache.spark</groupId> >> > <artifactId>spark-core</artifactId> >> > <version>1.5.1</version> >> > </dependency> >> > <dependency> >> > <groupId>org.apache.spark</groupId> >> > <artifactId>spark-sql</artifactId> >> > <version>1.5.1</version> >> > </dependency> >> > <dependency> >> > <groupId>org.apache.spark</groupId> >> > <artifactId>spark-hive_2.10</artifactId> >> > <version>1.5.0</version> >> > </dependency> >> > <dependency> >> > <groupId>com.databricks</groupId> >> > <artifactId>spark-csv_2.11</artifactId> >> > <version>1.3.0</version> >> > </dependency> >> > </dependencies> >> > >> > <build> >> > <sourceDirectory>src/main/scala</sourceDirectory> >> > <plugins> >> > <plugin> >> > <groupId>org.scala-tools</groupId> >> > <artifactId>maven-scala-plugin</artifactId> >> > <version>${maven-scala-plugin.version}</version> >> > <executions> >> > <execution> >> > <goals> >> > <goal>compile</goal> >> > </goals> >> > </execution> >> > </executions> >> > <configuration> >> > <jvmArgs> >> > <jvmArg>-Xms64m</jvmArg> >> > <jvmArg>-Xmx1024m</jvmArg> >> > </jvmArgs> >> > </configuration> >> > </plugin> >> > <plugin> >> > <groupId>org.apache.maven.plugins</groupId> >> > <artifactId>maven-shade-plugin</artifactId> >> > <version>1.6</version> >> > <executions> >> > <execution> >> > <phase>package</phase> >> > <goals> >> > <goal>shade</goal> >> > </goals> >> > <configuration> >> > <filters> >> > <filter> >> > <artifact>*:*</artifact> >> > <excludes> >> > <exclude>META-INF/*.SF</exclude> >> > <exclude>META-INF/*.DSA</exclude> >> > <exclude>META-INF/*.RSA</exclude> >> > </excludes> >> > </filter> >> > </filters> >> > <transformers> >> > <transformer >> > >> implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> >> > <mainClass>com.group.id.Launcher1</mainClass> >> > </transformer> >> > </transformers> >> > </configuration> >> > </execution> >> > </executions> >> > </plugin> >> > </plugins> >> > </build> >> > >> > <artifactId>scala</artifactId> >> > </project> >> > >> > >> > I am sure I have omitted something? >> > >> > >> > Thanks >> > >> > >> > Dr Mich Talebzadeh >> > >> > >> > >> > LinkedIn >> > >> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw >> > >> > >> > >> > http://talebzadehmich.wordpress.com >> > >> > >> > >
