Re: Spark 1.1.0 with Hadoop 2.5.0

2014-10-07 Thread Li HM
Here is the hive-site.xml hive.metastore.local false controls whether to connect to remove metastore server or open a new metastore server in Hive Client JVM hive.metastore.uris thrift://*:2513 Remote location of the metastore server hive.metastore.wareh

Re: Spark 1.1.0 with Hadoop 2.5.0

2014-10-07 Thread Cheng Lian
So it seems that the classpath issue has been resolved. The instantiation failure should be related to your hive-site.xml. Would you mind to create a public gist for your hive-site.xml? On 10/8/14 4:34 AM, Li HM wrote: Thanks Cheng. Here is the error message after a fresh build. $ mvn -Pyarn

Re: Spark 1.1.0 with Hadoop 2.5.0

2014-10-07 Thread Li HM
Thanks Cheng. Here is the error message after a fresh build. $ mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.5.0 -Phive -DskipTests clean package [INFO] [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM ...

Re: Spark 1.1.0 with Hadoop 2.5.0

2014-10-07 Thread Cheng Lian
The build command should be correct. What exact error did you encounter when trying Spark 1.1 + Hive 0.12 + Hadoop 2.5.0? On 10/7/14 2:21 PM, Li HM wrote: Thanks for the replied. Please refer to my another post entitled "How to make ./bin/spark-sql work with hive". It has all the error/excepti

Re: Spark 1.1.0 with Hadoop 2.5.0

2014-10-06 Thread Sean Owen
That's a Hive version issue, not Hadoop version issue. On Tue, Oct 7, 2014 at 7:21 AM, Li HM wrote: > Thanks for the replied. > > Please refer to my another post entitled "How to make ./bin/spark-sql > work with hive". It has all the error/exceptions I am getting. > > If I understand you correctl

Re: Spark 1.1.0 with Hadoop 2.5.0

2014-10-06 Thread Li HM
Thanks for the replied. Please refer to my another post entitled "How to make ./bin/spark-sql work with hive". It has all the error/exceptions I am getting. If I understand you correctly, I can build the package with mvn -Phive,hadoop-2.4 -Dhadoop.version=2.5.0 clean package This is what I actua

Re: Spark 1.1.0 with Hadoop 2.5.0

2014-10-06 Thread Sean Owen
The hadoop-2.4 profile is really intended to be "Hadoop 2.4+". It should compile and run fine with Hadoop 2.5 as far as I know. CDH 5.2 is Hadoop 2.5 + Spark 1.1, so there is evidence it works. You didn't say what doesn't work. On Tue, Oct 7, 2014 at 6:07 AM, hmxxyy wrote: > Does Spark 1.1.0 work

Spark 1.1.0 with Hadoop 2.5.0

2014-10-06 Thread hmxxyy
Does Spark 1.1.0 work with Hadoop 2.5.0? The maven build instruction only has command options up to hadoop 2.4. Anybody ever made it work? I am trying to run spark-sql with hive 0.12 on top of hadoop 2.5.0 but can't make it work. -- View this message in context: http://apache-spark-user-lis