Hi All,
I tried the make distribution script and it worked well. I was able to compile the spark binary on our CDH5 cluster. Once I compiled Spark, I copied over the binaries in the dist folder to all the other machines in the cluster. However, I run into an issue while submit a job in yarn-client mode. I get an error message that says the following Resource file:/opt/spark/spark-1.0.0-bin-hadoop2/lib/spark-assembly-1.1.0-SNAPSHOT-hadoop2.3.0.jar" changed on src filesystem (expected 1404845211000, was 1404845404000) My end goal is to submit a job (that uses MLLib) in our Yarn cluster. Any thoughts anyone? Regards, Krishna On Tue, Jul 8, 2014 at 9:49 AM, Sandy Ryza <sandy.r...@cloudera.com> wrote: > > Hi Srikrishna, > > The binaries are built with something like > mvn package -Pyarn -Dhadoop.version=2.3.0-cdh5.0.1 > -Dyarn.version=2.3.0-cdh5.0.1 > > -Sandy > > > On Tue, Jul 8, 2014 at 3:14 AM, 田毅 <tia...@asiainfo.com> wrote: >> >> try this command: >> >> make-distribution.sh --hadoop 2.3.0-cdh5.0.0 --with-yarn --with-hive >> >> >> >> >> 田毅 >> =========================================== >> 橘云平台产品线 >> 大数据产品部 >> 亚信联创科技（中国）有限公司 >> 手机：13910177261 >> 电话：010－82166322 >> 传真：010－82166617 >> Q Q：20057509 >> MSN：yi.t...@hotmail.com >> 地址：北京市海淀区东北旺西路10号院东区 亚信联创大厦 >> >> >> =========================================== >> >> 在 2014年7月8日，上午11:53，Krishna Sankar <ksanka...@gmail.com> 写道： >> >> Couldn't find any reference of CDH in pom.xml - profiles or the >> hadoop.version.Am also wondering how the cdh compatible artifact was >> compiled. >> Cheers >> <k/> >> >> >> On Mon, Jul 7, 2014 at 8:07 PM, Srikrishna S <srikrishna...@gmail.com> wrote: >>> >>> Hi All, >>> >>> Does anyone know what the command line arguments to mvn are to generate the >>> pre-built binary for spark on Hadoop 2-CHD5. >>> >>> I would like to pull in a recent bug fix in spark-master and rebuild the >>> binaries in the exact same way that was used for that provided on the >>> website. >>> >>> I have tried the following: >>> >>> mvn install -Pyarn -Dhadoop.version=2.3.0-cdh5.0.1 >>> >>> And it doesn't quite work. >>> >>> Any thoughts anyone? >>> >> >> >
