I issued the same basic command and it worked fine. RADTech-MBP:spark $ ./make-distribution.sh --name hadoop-2.6 --tgz -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -Phive -Phive-thriftserver -DskipTests
Which created: spark-1.6.0-SNAPSHOT-bin-hadoop-2.6.tgz in the root directory of the project. FWIW, the environment was an MBP with OS X 10.10.5 and Java: java version "1.8.0_51" Java(TM) SE Runtime Environment (build 1.8.0_51-b16) Java HotSpot(TM) 64-Bit Server VM (build 25.51-b03, mixed mode) -Todd On Tue, Oct 27, 2015 at 12:17 PM, Ted Yu <yuzhih...@gmail.com> wrote: > I used the following command: > make-distribution.sh --name custom-spark --tgz -Phadoop-2.4 -Phive > -Phive-thriftserver -Pyarn > > spark-1.6.0-SNAPSHOT-bin-custom-spark.tgz was generated (with patch from > SPARK-11348) > > Can you try above command ? > > Thanks > > On Tue, Oct 27, 2015 at 7:03 AM, Kayode Odeyemi <drey...@gmail.com> wrote: > >> Ted, I switched to this: >> >> ./make-distribution.sh --name spark-latest --tgz -Dhadoop.version=2.6.0 >> -Phadoop-2.6 -Phive -Phive-thriftserver -Pyarn -DskipTests clean package -U >> >> Same error. No .gz file. Here's the bottom output log: >> >> + rm -rf /home/emperor/javaprojects/spark/dist >> + mkdir -p /home/emperor/javaprojects/spark/dist/lib >> + echo 'Spark [WARNING] See >> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin (git revision >> 3689beb) built for Hadoop [WARNING] See >> http://docs.codehaus.org/display/MAVENUSER/Shade+Pl >> + echo 'Build flags: -Dhadoop.version=2.6.0' -Phadoop-2.6 -Phive >> -Phive-thriftserver -Pyarn -DskipTests clean package -U >> + cp >> /home/emperor/javaprojects/spark/assembly/target/scala-2.10/spark-assembly-1.6.0-SNAPSHOT-hadoop2.6.0.jar >> /home/emperor/javaprojects/spark/dist/lib/ >> + cp >> /home/emperor/javaprojects/spark/examples/target/scala-2.10/spark-examples-1.6.0-SNAPSHOT-hadoop2.6.0.jar >> /home/emperor/javaprojects/spark/dist/lib/ >> + cp >> /home/emperor/javaprojects/spark/network/yarn/target/scala-2.10/spark-1.6.0-SNAPSHOT-yarn-shuffle.jar >> /home/emperor/javaprojects/spark/dist/lib/ >> + mkdir -p /home/emperor/javaprojects/spark/dist/examples/src/main >> + cp -r /home/emperor/javaprojects/spark/examples/src/main >> /home/emperor/javaprojects/spark/dist/examples/src/ >> + '[' 1 == 1 ']' >> + cp >> /home/emperor/javaprojects/spark/lib_managed/jars/datanucleus-api-jdo-3.2.6.jar >> /home/emperor/javaprojects/spark/lib_managed/jars/datanucleus-core-3.2.10.jar >> /home/emperor/javaprojects >> ed/jars/datanucleus-rdbms-3.2.9.jar >> /home/emperor/javaprojects/spark/dist/lib/ >> + cp /home/emperor/javaprojects/spark/LICENSE >> /home/emperor/javaprojects/spark/dist >> + cp -r /home/emperor/javaprojects/spark/licenses >> /home/emperor/javaprojects/spark/dist >> + cp /home/emperor/javaprojects/spark/NOTICE >> /home/emperor/javaprojects/spark/dist >> + '[' -e /home/emperor/javaprojects/spark/CHANGES.txt ']' >> + cp -r /home/emperor/javaprojects/spark/data >> /home/emperor/javaprojects/spark/dist >> + mkdir /home/emperor/javaprojects/spark/dist/conf >> + cp /home/emperor/javaprojects/spark/conf/docker.properties.template >> /home/emperor/javaprojects/spark/conf/fairscheduler.xml.template >> /home/emperor/javaprojects/spark/conf/log4j.properties >> emperor/javaprojects/spark/conf/metrics.properties.template >> /home/emperor/javaprojects/spark/conf/slaves.template >> /home/emperor/javaprojects/spark/conf/spark-defaults.conf.template /home/em >> ts/spark/conf/spark-env.sh.template >> /home/emperor/javaprojects/spark/dist/conf >> + cp /home/emperor/javaprojects/spark/README.md >> /home/emperor/javaprojects/spark/dist >> + cp -r /home/emperor/javaprojects/spark/bin >> /home/emperor/javaprojects/spark/dist >> + cp -r /home/emperor/javaprojects/spark/python >> /home/emperor/javaprojects/spark/dist >> + cp -r /home/emperor/javaprojects/spark/sbin >> /home/emperor/javaprojects/spark/dist >> + cp -r /home/emperor/javaprojects/spark/ec2 >> /home/emperor/javaprojects/spark/dist >> + '[' -d /home/emperor/javaprojects/spark/R/lib/SparkR ']' >> + '[' false == true ']' >> + '[' true == true ']' >> + TARDIR_NAME='spark-[WARNING] See >> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest' >> + TARDIR='/home/emperor/javaprojects/spark/spark-[WARNING] See >> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest' >> + rm -rf '/home/emperor/javaprojects/spark/spark-[WARNING] See >> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest' >> + cp -r /home/emperor/javaprojects/spark/dist >> '/home/emperor/javaprojects/spark/spark-[WARNING] See >> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest' >> cp: cannot create directory >> `/home/emperor/javaprojects/spark/spark-[WARNING] See >> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest': >> No such file or directory >> >> >> On Tue, Oct 27, 2015 at 2:14 PM, Ted Yu <yuzhih...@gmail.com> wrote: >> >>> Can you try the same command shown in the pull request ? >>> >>> Thanks >>> >>> On Oct 27, 2015, at 12:40 AM, Kayode Odeyemi <drey...@gmail.com> wrote: >>> >>> Thank you. >>> >>> But I'm getting same warnings and it's still preventing the archive from >>> being generated. >>> >>> I've ran this on both OSX Lion and Ubuntu 12. Same error. No .gz file >>> >>> On Mon, Oct 26, 2015 at 9:10 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>> >>>> Looks like '-Pyarn' was missing in your command. >>>> >>>> On Mon, Oct 26, 2015 at 12:06 PM, Kayode Odeyemi <drey...@gmail.com> >>>> wrote: >>>> >>>>> I used this command which is synonymous to what you have: >>>>> >>>>> ./make-distribution.sh --name spark-latest --tgz --mvn mvn >>>>> -Dhadoop.version=2.6.0 -Phadoop-2.6 -Phive -Phive-thriftserver -DskipTests >>>>> clean package -U >>>>> >>>>> But I still see WARNINGS like this in the output and no .gz file >>>>> created: >>>>> >>>>> cp: /usr/local/spark-latest/spark-[WARNING] See >>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/.part-r-00005.gz.parquet.crc: >>>>> No such file or directory >>>>> cp: /usr/local/spark-latest/spark-[WARNING] See >>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/part-r-00005.gz.parquet: >>>>> No such file or directory >>>>> cp: /usr/local/spark-latest/spark-[WARNING] See >>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9: >>>>> No such file or directory >>>>> cp: >>>>> /usr/local/spark-latest/dist/python/test_support/sql/parquet_partitioned/year=2015/month=9: >>>>> unable to copy extended attributes to >>>>> /usr/local/spark-latest/spark-[WARNING] See >>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9: >>>>> No such file or directory >>>>> cp: /usr/local/spark-latest/spark-[WARNING] See >>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1: >>>>> No such file or directory >>>>> cp: >>>>> /usr/local/spark-latest/dist/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1: >>>>> unable to copy extended attributes to >>>>> /usr/local/spark-latest/spark-[WARNING] See >>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1: >>>>> No such file or directory >>>>> cp: /usr/local/spark-latest/spark-[WARNING] See >>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/.part-r-00007.gz.parquet.crc: >>>>> No such file or directory >>>>> >>>>> On Mon, Oct 26, 2015 at 8:58 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>>>> >>>>>> If you use the command shown in: >>>>>> https://github.com/apache/spark/pull/9281 >>>>>> >>>>>> You should have got the following: >>>>>> >>>>>> >>>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/part-r-00008.gz.parquet >>>>>> >>>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=9/day=1/part-r-00007.gz.parquet >>>>>> >>>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00004.gz.parquet >>>>>> >>>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=25/part-r-00002.gz.parquet >>>>>> >>>>>> ./dist/python/test_support/sql/parquet_partitioned/year=2015/month=10/day=26/part-r-00005.gz.parquet >>>>>> >>>>>> On Mon, Oct 26, 2015 at 11:47 AM, Kayode Odeyemi <drey...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> I see a lot of stuffs like this after the a successful maven build: >>>>>>> >>>>>>> cp: /usr/local/spark-latest/spark-[WARNING] See >>>>>>> http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin-bin-spark-latest/python/test_support/sql/parquet_partitioned/year=2014/month=9/day=1/ >>>>>>> part-r-00008.gz.parquet: No such file or directory >>>>>>> >>>>>>> Seems it fails when it tries to package the build as an archive. >>>>>>> >>>>>>> I'm using the latest code on github master. >>>>>>> >>>>>>> Any ideas please? >>>>>>> >>>>>>> On Mon, Oct 26, 2015 at 6:20 PM, Yana Kadiyska < >>>>>>> yana.kadiy...@gmail.com> wrote: >>>>>>> >>>>>>>> In 1.4 ./make_distribution produces a .tgz file in the root >>>>>>>> directory (same directory that make_distribution is in) >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Oct 26, 2015 at 8:46 AM, Kayode Odeyemi <drey...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> The ./make_distribution task completed. However, I can't seem to >>>>>>>>> locate the >>>>>>>>> .tar.gz file. >>>>>>>>> >>>>>>>>> Where does Spark save this? or should I just work with the dist >>>>>>>>> directory? >>>>>>>>> >>>>>>>>> On Fri, Oct 23, 2015 at 4:23 PM, Kayode Odeyemi <drey...@gmail.com >>>>>>>>> > wrote: >>>>>>>>> >>>>>>>>>> I saw this when I tested manually (without ./make-distribution) >>>>>>>>>> >>>>>>>>>> Detected Maven Version: 3.2.2 is not in the allowed range 3.3.3. >>>>>>>>>> >>>>>>>>>> So I simply upgraded maven to 3.3.3. >>>>>>>>>> >>>>>>>>>> Resolved. Thanks >>>>>>>>>> >>>>>>>>>> On Fri, Oct 23, 2015 at 3:17 PM, Sean Owen <so...@cloudera.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> This doesn't show the actual error output from Maven. I have a >>>>>>>>>>> strong >>>>>>>>>>> guess that you haven't set MAVEN_OPTS to increase the memory >>>>>>>>>>> Maven can >>>>>>>>>>> use. >>>>>>>>>>> >>>>>>>>>>> On Fri, Oct 23, 2015 at 6:14 AM, Kayode Odeyemi < >>>>>>>>>>> drey...@gmail.com> wrote: >>>>>>>>>>> > Hi, >>>>>>>>>>> > >>>>>>>>>>> > I can't seem to get a successful maven build. Please see >>>>>>>>>>> command output >>>>>>>>>>> > below: >>>>>>>>>>> > >>>>>>>>>>> > bash-3.2$ ./make-distribution.sh --name spark-latest --tgz >>>>>>>>>>> --mvn mvn >>>>>>>>>>> > -Dhadoop.version=2.7.0 -Phadoop-2.7 -Phive -Phive-thriftserver >>>>>>>>>>> -DskipTests >>>>>>>>>>> > clean package >>>>>>>>>>> > +++ dirname ./make-distribution.sh >>>>>>>>>>> > ++ cd . >>>>>>>>>>> > ++ pwd >>>>>>>>>>> > + SPARK_HOME=/usr/local/spark-latest >>>>>>>>>>> > + DISTDIR=/usr/local/spark-latest/dist >>>>>>>>>>> > + SPARK_TACHYON=false >>>>>>>>>>> > + TACHYON_VERSION=0.7.1 >>>>>>>>>>> > + TACHYON_TGZ=tachyon-0.7.1-bin.tar.gz >>>>>>>>>>> > + >>>>>>>>>>> > TACHYON_URL= >>>>>>>>>>> https://github.com/amplab/tachyon/releases/download/v0.7.1/tachyon-0.7.1-bin.tar.gz >>>>>>>>>>> > + MAKE_TGZ=false >>>>>>>>>>> > + NAME=none >>>>>>>>>>> > + MVN=/usr/local/spark-latest/build/mvn >>>>>>>>>>> > + (( 12 )) >>>>>>>>>>> > + case $1 in >>>>>>>>>>> > + NAME=spark-latest >>>>>>>>>>> > + shift >>>>>>>>>>> > + shift >>>>>>>>>>> > + (( 10 )) >>>>>>>>>>> > + case $1 in >>>>>>>>>>> > + MAKE_TGZ=true >>>>>>>>>>> > + shift >>>>>>>>>>> > + (( 9 )) >>>>>>>>>>> > + case $1 in >>>>>>>>>>> > + MVN=mvn >>>>>>>>>>> > + shift >>>>>>>>>>> > + shift >>>>>>>>>>> > + (( 7 )) >>>>>>>>>>> > + case $1 in >>>>>>>>>>> > + break >>>>>>>>>>> > + '[' -z >>>>>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Home ']' >>>>>>>>>>> > + '[' -z >>>>>>>>>>> /Library/Java/JavaVirtualMachines/jdk1.8.0_20.jdk/Contents/Home ']' >>>>>>>>>>> > ++ command -v git >>>>>>>>>>> > + '[' /usr/bin/git ']' >>>>>>>>>>> > ++ git rev-parse --short HEAD >>>>>>>>>>> > + GITREV=487d409 >>>>>>>>>>> > + '[' '!' -z 487d409 ']' >>>>>>>>>>> > + GITREVSTRING=' (git revision 487d409)' >>>>>>>>>>> > + unset GITREV >>>>>>>>>>> > ++ command -v mvn >>>>>>>>>>> > + '[' '!' /usr/bin/mvn ']' >>>>>>>>>>> > ++ mvn help:evaluate -Dexpression=project.version >>>>>>>>>>> -Dhadoop.version=2.7.0 >>>>>>>>>>> > -Phadoop-2.7 -Phive -Phive-thriftserver -DskipTests clean >>>>>>>>>>> package >>>>>>>>>>> > ++ grep -v INFO >>>>>>>>>>> > ++ tail -n 1 >>>>>>>>>>> > + VERSION='[ERROR] [Help 1] >>>>>>>>>>> > >>>>>>>>>>> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException >>>>>>>>>>> ' >>>>>>>>>>> > >>>>>>>>>>> > Same output error with JDK 7 >>>>>>>>>>> > >>>>>>>>>>> > Appreciate your help. >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
