[
https://issues.apache.org/jira/browse/HIVE-25452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17487426#comment-17487426
]
Nikhil Gupta commented on HIVE-25452:
-------------------------------------
The issue is ABFS storage specific.
ABFS doesn't support the same lease mechanism that HDFS supports. If one tries
to have the staging directory as ABFS then due to the background thread
clearing the scratch directory causes the rename operation to fail with
DestParentDirectoryNotFound Exception.
To avoid this issue set config `hive.server2.clear.dangling.scratchdir=false`
at server level
> Hive job fails while closing reducer output - Unable to rename
> --------------------------------------------------------------
>
> Key: HIVE-25452
> URL: https://issues.apache.org/jira/browse/HIVE-25452
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
> Affects Versions: 3.1.0
> Reporter: Nikhil Gupta
> Assignee: Nikhil Gupta
> Priority: Major
>
> I have observed similar issue for Hive on LLAP:
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename
> output from:
> abfs://<redacted>/tmp/hive/root/277a283f-9d32-4525-b35f-03551c6a364e/hive_2021-08-15_20-22-24_848_2416200416962826887-27/-mr-10001/.hive-staging_hive_2021-08-15_20-22-24_848_2416200416962826887-27/_task_tmp.-ext-10002/_tmp.000000_3
> to:
> abfs://<redacted>/tmp/hive/root/277a283f-9d32-4525-b35f-03551c6a364e/hive_2021-08-15_20-22-24_848_2416200416962826887-27/-mr-10001/.hive-staging_hive_2021-08-15_20-22-24_848_2416200416962826887-27/_tmp.-ext-10002/000000_3
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commitOneOutPath(FileSinkOperator.java:272)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:236)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$400(FileSinkOperator.java:168)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1331)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:732)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:756)
> at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:756)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:383)
> ... 16 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
