Hi Sophia,
There is no Hadoop-2.6. I believe you should use Hadoop-2.4 as shown below mvn -Phadoop-2.4 -Dhadoop.version=2.6.0 -DskipTests clean package Also if you are building it for Hive on Spark engine, you should not include Hadoop.jar files in your build. For example I tried to build spark 1.3 from source code (I read that this version works OK with Hive, having tried unsuccessfully spark 1.5.2). The following command created the tar file ./make-distribution.sh --name "hadoop2-without-hive" --tgz "-Pyarn,hadoop-provided,hadoop-2.4,parquet-provided" spark-1.3.0-bin-hadoop2-without-hive.tar.gz Now I have other issues making Hive to use Spark execution engine (requires Hive 1.1 or above ) In hive I do set spark.home=/usr/lib/spark; set hive.execution.engine=spark; set spark.master=spark://127.0.0.1:7077; set spark.eventLog.enabled=true; set spark.eventLog.dir=/usr/lib/spark/logs; set spark.executor.memory=512m; set spark.serializer=org.apache.spark.serializer.KryoSerializer; use asehadoop; select count(1) from t; I get the following OK Time taken: 0.753 seconds Query ID = hduser_20151127003523_e9863e84-9a81-4351-939c-36b3bef36478 Total jobs = 1 Launching Job 1 out of 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)' FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.spark.SparkTask HTH, Mich NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Technology Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility. From: Sofia [mailto:[email protected]] Sent: 18 November 2015 16:50 To: [email protected] Subject: Hive version with Spark Hello After various failed tries to use my Hive (1.2.1) with my Spark (Spark 1.4.1 built for Hadoop 2.2.0) I decided to try to build again Spark with Hive. I would like to know what is the latest Hive version that can be used to build Spark at this point. When downloading Spark 1.5 source and trying: mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -Phive -Phive-1.2.1 -Phive-thriftserver -DskipTests clean package I get : The requested profile "hive-1.2.1" could not be activated because it does not exist. Thank you Sofia
