Aggarwal-Raghav commented on PR #6343: URL: https://github.com/apache/hive/pull/6343#issuecomment-4017416145
Hi @abstractdog , i tested this patch on my local setup (hadoop, hive, TEZ-4682 branch and Zookeeper - installed via brew) and **its working**🚀. Ran a basic insert command Attaching the HS2 logs and tez_am docker logs [tez-am-working.log](https://github.com/user-attachments/files/25818516/tez-am-working.log) [working_hs2.log](https://github.com/user-attachments/files/25818518/working_hs2.log) **Few things to note:** 1. Add the following in `/etc/hosts/` file. this is necesary for docker to communicate with hdfs running on localhost ```127.0.0.1 host.docker.internal``` 2. tez-am docker image needs hive-exec jar i.e resorce localization. For now I added it using the plugin directory way i.e. ```-v "/Users/raghav/Desktop/plugin:/opt/tez/plugins" \``` and plugin directory contains only hive-exec jar otherwise . ``` Vertex vertex_1769280834537_0000_1_01 [Reducer 2] killed/failed due to: INIT_FAILURE] .... vertex=vertex_1769280834537_0000_1_00 [Map 1], org.apache.tez.dag.api .TezUncheckedException: java. lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.ha doop.hive.ql.io. CombineHiveInputFormat not found ``` 5. `core-site.xml` ``` <property> <name>fs.defaultFS</name> <value>hdfs://host.docker.internal:9000</value> </property> ``` 6. `hdfs-site.xml` Connecting docker AM to datanode was really problematic, took lot of time 😅 ``` <property> <name>dfs.datanode.use.datanode.hostname</name> <value>true</value> </property> <property> <name>dfs.datanode.address</name> <value>host.docker.internal:9866</value> </property> <property> <name>dfs.datanode.hostname</name> <value>host.docker.internal</value> </property> ``` 7. In `docker tez-am tez-site.xml ` (Will update the PR for this as well in TEZ-4682) ``` <property> <name>dfs.client.use.datanode.hostname</name> <value>true</value> </property> ``` 8. Some `--add-opens` in tez-entrypoint.sh needs to be updated. Basically hive project --add-opens are also required. Will update the PR for TEZ-4682. **NOTE/IMPORTANT**: There was a flaky behaviour observed with `tez-conf.pb` in tez-staging directory in hdfs. It was throwing error for `No file found`. Not sure why it was occuring but it was fixed after some time automatically and not facing it again. Will post here if I face it again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
