[
https://issues.apache.org/jira/browse/HIVE-25401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480361#comment-17480361
]
zhangbutao commented on HIVE-25401:
-----------------------------------
I think you can try this parameter and its value is mutiple cluster deFaultFS :
<property>
<name>mapreduce.job.hdfs-servers</name>
<value>hdfs://cluster1,hdfs://cluster2</value>
</property>
> Insert overwrite a table which location is on other cluster fail in
> kerberos cluster
> --------------------------------------------------------------------------------------
>
> Key: HIVE-25401
> URL: https://issues.apache.org/jira/browse/HIVE-25401
> Project: Hive
> Issue Type: Bug
> Components: Hive
> Affects Versions: 2.3.0, 3.1.2
> Environment: hive 2.3
> hadoop3 cluster with kerberos
> Reporter: Max Xie
> Assignee: Max Xie
> Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-25401.patch, image-2021-07-29-14-25-23-418.png
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> we have tow hdfs clusters with kerberos security, it means that mapreduce
> task need delegation tokens to authenticate namenode when hive on mapreduce
> run.
> Insert overwrite a table which location is on other cluster fail in kerberos
> cluster. For example,
> # yarn cluster's default fs is hdfs://cluster1
> # tb1's location is hdfs://cluster1/tb1
> # tb2's location is hdfs://cluster2/tb2
> # sql `INSERT OVERWRITE TABLE tb2 SELECT * from tb1` run on yarn cluster
> will fail
>
> reduce task error log:
> !image-2021-07-29-14-25-23-418.png!
> How to fix:
> After dig it, web found mapreduce job just obtain delegation tokens for input
> files in FileInputFormat. But Hive context get extendal scratchDir base on
> table's location, If the table 's location is on other cluster, the
> delegation token will not be obtained.
> So we need to obtaine delegation tokens for hive scratchDirs before hive
> submit mapreduce job.
>
> How to test:
> no test
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
