Not a solution, but looking at the source code of S3AFileSystem.java (Hadoop 2.8.5), I think the Exception raised inside S3AFileSystem.rename() is swallowed and only a new HiveException is reported. So, in order to find out the root cause, I guess you might need to set Log level to DEBUG and see what Exception is raised inside S3AFileSystem.innerRename() (which is called from rename()). S3AFileSystem.innerRename() is implemented as copy-and-delete, so nasty things could happen inside the method.
Related Hive JIRA: https://issues.apache.org/jira/browse/HIVE-20517 --- gla On Tue, Feb 4, 2020 at 5:05 PM Souvikk Roy <souvikpl...@gmail.com> wrote: > Hello, > > We are using some external tables backed by aws S3. And we are > intermittently getting this error, most likely at the last stage of the > reduce, I see some similar posts in net but could not find any solution, Is > there any way yo solve it: > > org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename output > from: > s3a://xxx/pn/.hive-staging_hive_2020-01-27_08-05-54_195_618381349208890136-176126/_task_tmp.-ext-10000/year=2019/product=xxx/abc=2019-12-16/xxx=BCD/yz=x/pattern=xxx/_tmp.000001_0 > to: > s3a://xxx/tables/xxx/xxxx/xxx/xx/xxxx/xx/.hive-staging_hive_2020-01-27_08-05-54_195_6 > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.commit(FileSinkOperator.java:236) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator$FSPaths.access$200(FileSinkOperator.java:133) > at > org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1014) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:598) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610) > at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:610) > at > org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:287) > at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:453) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) > > thanks > Souvik >
