[
https://issues.apache.org/jira/browse/HIVE-27688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17764575#comment-17764575
]
liang yu edited comment on HIVE-27688 at 9/13/23 10:19 AM:
-----------------------------------------------------------
I traced the code of moveTask, and find that when we execute sql "insert
overwrite into table partition (XX) select ....", it will throw HiveException
whose exception message is overwritten by method getHiveException(e, msg). Here
is the chain of how it gets overwritten:
{code:java}
execute(handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
273)> handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 392)>
loadPartition(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472)
->replaceFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 1817) ->
moveFile(org/apache/hadoop/hive/ql/metadata/Hive.java: line 4136) ->
needToCopy(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3681) ->
throw HiveException(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3856)
{code}
but this HiveException's message was overwritten by method
moveFile(org/apache/hadoop/hive/ql/metadata/Hive.java: line 4136), it catches
the exception and replaced the message with("Unable to move source
/path/to/source to destination /path/to/dest, which is a very regular error
message")
But when I execute sql "insert into table partition (XX) select ...", it will
throw HiveException which is not overwritten. Here is the chain of how it
throws the correct Exception:
{code:java}
execute(handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
273)> handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 392)>
loadPartition(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472) ->
copyFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 1821) ->
copyFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3937) ->
needToCopy(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3389) ->
throw HiveException(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3856)
{code}
this HiveException was thrown and caught by method loadPartition
(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472) whose error message is
not overwritten.
Solution:
I changed the code in org/apache/hadoop/hive/ql/metadata/Hive.java: line 3771
{code:java}
throw getHiveException(e, msg); {code}
to
{code:java}
throw getHiveException(e, e.getMessage(), msg){code}
and it returns the correct error message
was (Author: JIRAUSER299608):
I traced the code of moveTask, and find that when we execute sql "insert
overwrite into table partition (XX) select ....", it will throw HiveException
whose exception message is overwritten by method getHiveException(e, msg). Here
is the chain of how it gets overwritten:
{code:java}
execute(handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
273)> handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 392)>
loadPartition(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472)
->replaceFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 1817) ->
moveFile(org/apache/hadoop/hive/ql/metadata/Hive.java: line 4136) ->
needToCopy(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3681) ->
throw HiveException(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3856)
{code}
but this HiveException's message was overwritten by method
moveFile(org/apache/hadoop/hive/ql/metadata/Hive.java: line 4136), it catches
the exception and replaced the message with("Unable to move source
/path/to/source to destination /path/to/dest, which is a very regular error
message")
But when I execute sql "insert into table partition (XX) select ...", it will
throw HiveException which is not overwritten. Here is the chain of how it
throws the correct Exception:
{code:java}
execute(handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line
273)> handleStaticParts(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 392)>
loadPartition(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472) ->
copyFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 1821) ->
copyFiles(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3937) ->
needToCopy(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3389) ->
throw HiveException(org/apache/hadoop/hive/ql/metadata/Hive.java: line 3856)
{code}
this HiveException was thrown and caught by method loadPartition
(org/apache/hadoop/hive/ql/exec/MoveTask.java:line 472) whose error message is
not overwritten.
> hive MoveTask cannot show the correct exception message
> -------------------------------------------------------
>
> Key: HIVE-27688
> URL: https://issues.apache.org/jira/browse/HIVE-27688
> Project: Hive
> Issue Type: Bug
> Reporter: liang yu
> Assignee: liang yu
> Priority: Major
> Attachments: image-2023-09-13-17-39-27-864.png,
> image-2023-09-13-17-40-02-981.png
>
>
> I am using hive.version=3.1.3; hadoop.version=3.3.4.
> Setting hive.load.data.owner to hive, and I used user ubd_by to execute sql.
> When I try to insert *overwrite* to an {*}existing table partition{*}, I get
> the exception that: Unable to move source /path/to/source to destination
> /path/to/dest, which is a very regular error message, gives me no helpful
> information.
>
> {code:java}
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.Movelask.Unable to move source
> hdfs://xl/user/ubd master/ubd_b_dwa.
> db/dwa_m_user/month_id=xxx/prov_id=xx/.staging to destination
> hdfs://xl/user/ubd master/ubd_b_dwa.
> db/dwa_m_user/month_id=xxx/prov_id=xx/.staging {code}
>
> But when I try to insert *into* an {*}existing table partition{*}, I get the
> exception that: Unable to move source /path/to/source to destination
> /path/to/dest as the file is not owned by hive and load data is also not ran
> as hive. which gives me a very helpful error message that I should change the
> hive.load.data.owner to hive.
>
> {code:java}
> FAILED: Execution Error, return code l from org.
> apache.hadoop.hive.ql.metadata.HiveException: org.
> apache.hadoop.hive.gl.exec.Movelask: Load Data failed for hdfs://xl/user/ubd
> master/ubd_b_dwa. db/dwa_m_user/month_id=xxx/prov_id=xx/
> hive-staging hive 2023-09-13 17-34-31 302 5892190500368248766-1/-ext-10000 as
> the file is not owned by hive and load data is also not ran as hive{code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
