Re: Turning on Tez for Hive

[Tim Harsch]

Hi Lefty,
I came across those documents as well.  They gave me some good hints, but
were in some places too specific to Horton Works.  For the record, I did
get Hive 0.13.1 and Tez 0.4.1 working together.  (and I immediately saw a
200% speed up on my corpus of queries).  For future travelers here are my
(somewhat raw) notes:

Get the source distro at:
http://www.apache.org/dyn/closer.cgi/incubator/tez/tez-0.4.1-incubating/

explode it, and run 'mvm package -DskipTests -Dtar'

explode the tar from tez-dist/target/tez-0.4.1-incubating.tar.gz to a
location which will become TEZ_INSTALL_DIR.

Tez needs a user dir for to match unix user, or you will get
java.io.FileNotFoundException: File does not exist: hdfs:/user/tharsch:
% hdfs dfs -mkdir /user/tharsch; hdfs dfs -chmod g+w /user/tharsch

# NOTE: set HIVE_AUX_JARS_PATH using if -z test or get
java.lang.IllegalArgumentException: Can not create a Path from an empty
string
# NOTE: TEZ_JARS cannot include "*" or you will get error
"java.io.FileNotFoundException: File
file:/home/users/tharsch/apps/tez/tez-0.4.1-incubating/* does not exist"

export TEZ_INSTALL_DIR=/home/users/tharsch/apps/tez/tez-0.4.1-incubating
export TEZ_CONF_DIR=$TEZ_INSTALL_DIR/conf
export TEZ_JARS=$(echo "$TEZ_INSTALL_DIR"/*.jar | tr ' ' ':'):$(echo
"$TEZ_INSTALL_DIR"/lib/*.jar | tr ' ' ':')
if [ -z "$HIVE_AUX_JARS_PATH" ]; then
     export HIVE_AUX_JARS_PATH="$TEZ_JARS"
else
     export HIVE_AUX_JARS_PATH="$HIVE_AUX_JARS_PATH:$TEZ_JARS"
fi

NOTE:  Be sure to copy TEZ jars to HDFS and set HADOOP_CLASSPATH or you
will get:
org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez
jars, tez.lib.uris is not defined in the configurartion

export HADOOP_CLASSPATH="${TEZ_CONF_DIR}:${TEZ_JARS}/*:${TEZ_JARS}/lib/*"


% hdfs dfs -mkdir -p /apps/tez-0.4.1; hdfs dfs -chmod g+w /apps/tez-0.4.1
% hdfs dfs -copyFromLocal $TEZ_INSTALL_DIR/* /apps/tez-0.4.1
% hdfs dfs -copyFromLocal $TEZ_INSTALL_DIR/lib/* /apps/tez-0.4.1

NOTE: Set mapred-site.xml
        <property>
                <name>mapreduce.framework.name
<http://mapreduce.framework.name/></name>
                <value>yarn-tez</value>
        </property>


NOTE: create tez-site.xml in $TEZ_CONF_DIR
<configuration>
     <property>
          <name>tez.lib.uris</name>
          <value>${fs.default.name}/apps/tez-0.4.1/</value>
     </property>
</configuration>


NOTE: HIVE Settings:
set hive.execution.engine=tez;
set hive.use.tez.natively=true;
set hive.enable.mrr=true;