Hi RJ, I think I remember noticing in the past that some Guava metadata ends up overwriting maven-generated metadata in the assembly's manifest. That's probably something we should fix if that still affects the build.
That being said, this is probably happening because you're using "install-file" instead of "install". If you want a workaround that doesn't require unshading things, you can change assembly.pom to (i) not skip the install plugin and (ii) have "jar" as the packaging, instead of pom. On Wed, Jan 14, 2015 at 1:08 PM, RJ Nowling <rnowl...@gmail.com> wrote: > Hi Sean, > > I confirmed that if I take the Spark 1.2.0 release (a428c446), undo the > guava PR [1], and use -Dmaven.install.skip=false with the workflow above, > the problem is fixed. > > RJ > > > [1] > https://github.com/apache/spark/commit/c9f743957fa963bc1dbed7a44a346ffce1a45cf2#diff-6382f8428b13fa6082fa688178f3dbcc > > On Wed, Jan 14, 2015 at 2:59 PM, RJ Nowling <rnowl...@gmail.com> wrote: > >> Thanks, Sean. >> >> Yes, Spark is incorrectly copying the spark assembly jar to >> com/google/guava in the maven repository. This is for the 1.2.0 release, >> just to clarify. >> >> I reverted the patches that shade Guava and removed the parts disabling >> the install plugin and it seemed to fix the issue. >> >> It seems that Spark poms are inheriting something from Guava. >> >> RJ >> >> On Wed, Jan 14, 2015 at 2:33 PM, Sean Owen <so...@cloudera.com> wrote: >> >>> Guava is shaded, although one class is left in its original package. >>> This shouldn't have anything to do with Spark's package or namespace >>> though. What are you saying is in com/google/guava? >>> >>> You can un-skip the install plugin with -Dmaven.install.skip=false >>> >>> On Wed, Jan 14, 2015 at 7:26 PM, RJ Nowling <rnowl...@gmail.com> wrote: >>> > Hi all, >>> > >>> > I'm trying to upgrade some Spark RPMs from 1.1.0 to 1.2.0. As part of >>> the >>> > RPM process, we build Spark with Maven. With Spark 1.2.0, though, the >>> > artifacts are placed in com/google/guava and there is no >>> org/apache/spark. >>> > >>> > I saw that the pom.xml files had been modified to prevent the install >>> > command and that the guava dependency was modified. Could someone who >>> is >>> > more familiar with the Spark maven files comment on what might be >>> causing >>> > this oddity? >>> > >>> > Thanks, >>> > RJ >>> > >>> > We build Spark like so: >>> > $ mvn -Phadoop-2.4 -Dmesos.version=0.20.0 -DskipTests clean package >>> > >>> > Then build a local Maven repo: >>> > >>> > mvn -Phadoop-2.4 \ >>> > -Dmesos.version=0.20.0 \ >>> > -DskipTests install:install-file \ >>> > >>> -Dfile=assembly/target/scala-2.10/spark-assembly-1.2.0-hadoop2.4.0.jar \ >>> > -DcreateChecksum=true \ >>> > -DgeneratePom=true \ >>> > -DartifactId=spark-assembly_2.1.0 \ >>> > -DlocalRepositoryPath=../maven2 >>> >> >> -- Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
