Agree with Andrew, we shouldn't block the release for this.
This issue won't be there in Spark distribution from Hortonworks since we
set the HDP version.
If you want to use the Apache Spark with HDP you can modify mapred-site.xml
to replace the hdp.version property with the right value for your cluster.
You can find the right value by invoking the hdp-select script on a node
that has HDP installed. On my system running it returns the following:
hdp-select status hadoop-client
hadoop-client - 2.2.5.0-2644
Here is a one line script to get the version:
export HDP_VER=`hdp-select status hadoop-client | sed 's/hadoop-client -
\(.*\)/\1/'`
CAUTION - if you modify mapred-site.xml on a node on the cluster, this will
break rolling upgrades in certain scenarios where a program like oozie
submitting a job from that node will use the hardcoded version instead of
the version specified by the client.
So what does the Hortonworks distribution do under the covers to support
hdp.version?
create a file called java-opts with the following config value in it
-Dhdp.version=2.2.5.0-2644. You can also specify the same value using
SPARK_JAVA_OPTS, i.e. export SPARK_JAVA_OPTS="-Dhdp.version=2.2.5.0-2644"
add the following options to spark-defaults.conf:
spark.driver.extraJavaOptions -Dhdp.version=2.2.5.0-2644
spark.yarn.am.extraJavaOptions -Dhdp.version=2.2.5.0-2644
--
View this message in context:
http://apache-spark-developers-list.1001551.n3.nabble.com/VOTE-Release-Apache-Spark-1-6-0-RC3-tp15660p15699.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
