[
https://issues.apache.org/jira/browse/HIVE-28620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-28620:
--------------------------------
Description:
HIVE-25907 fixed a case when an insert overwrite directory didn't write the
data to the final path when the query results cache was enabled. The solution
was implemented by checking the QB.parseInfo at a certain point in the semantic
analysis and writing/reading a flag to an instance. The problem is that
depending on the query (which produces the data to be inserted) can be of
different complexity, and only took care of queries like:
{code}
INSERT OVERWRITE DIRECTORY "<destination directory>" SELECT * FROM iowd;
{code}
but for queries like this:
{code}
EXPLAIN EXTENDED INSERT OVERWRITE DIRECTORY '/tmp' select a13.CATEGORY_ID
CATEGORY_ID, max(a14.CATEGORY_DESC) CATEGORY_DESC,a12.SUBCAT_ID
SUBCAT_ID,max(a13.SUBCAT_LONG_DESC) SUBCAT_DESC,avg((a11.QTY_SOLD *
(a11.UNIT_PRICE - a11.DISCOUNT))) WJXBFS1,sum((a11.QTY_SOLD * a11.UNIT_COST))
WJXBFS2 from ORDER_DETAIL a11 join LU_ITEM a12 on (a11.ITEM_ID = a12.ITEM_ID)
join LU_SUBCATEG a13 on (a12.SUBCAT_ID = a13.SUBCAT_ID) join LU_CATEGORY a14
on (a13.CATEGORY_ID = a14.CATEGORY_ID) group by a13.CATEGORY_ID, a12.SUBCAT_ID;
{code}
it doesn't work. The root cause is that the flag is set in [one QB
instance|https://github.com/apache/hive/blob/883d5dfe25929ba5dcac635e752adf5561d28402/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1836],
but during the plan generation, a genPlan is recursively called, moreover,
this.qb doesn't refers to the main/root QB, as
[setQB|https://github.com/apache/hive/blob/883d5dfe25929ba5dcac635e752adf5561d28402/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L12435]
is called at every step, so when iterating and reaching the file sink
generation plan, the QB instance passed is not the same as the one on which the
isInsertOverwriteDirectory flag was set.
> Query result is cached in case of IOWD if the subquery is not trivial
> ---------------------------------------------------------------------
>
> Key: HIVE-28620
> URL: https://issues.apache.org/jira/browse/HIVE-28620
> Project: Hive
> Issue Type: Bug
> Security Level: Public(Viewable by anyone)
> Reporter: László Bodor
> Assignee: László Bodor
> Priority: Major
>
> HIVE-25907 fixed a case when an insert overwrite directory didn't write the
> data to the final path when the query results cache was enabled. The solution
> was implemented by checking the QB.parseInfo at a certain point in the
> semantic analysis and writing/reading a flag to an instance. The problem is
> that depending on the query (which produces the data to be inserted) can be
> of different complexity, and only took care of queries like:
> {code}
> INSERT OVERWRITE DIRECTORY "<destination directory>" SELECT * FROM iowd;
> {code}
> but for queries like this:
> {code}
> EXPLAIN EXTENDED INSERT OVERWRITE DIRECTORY '/tmp' select a13.CATEGORY_ID
> CATEGORY_ID, max(a14.CATEGORY_DESC) CATEGORY_DESC,a12.SUBCAT_ID
> SUBCAT_ID,max(a13.SUBCAT_LONG_DESC) SUBCAT_DESC,avg((a11.QTY_SOLD *
> (a11.UNIT_PRICE - a11.DISCOUNT))) WJXBFS1,sum((a11.QTY_SOLD *
> a11.UNIT_COST)) WJXBFS2 from ORDER_DETAIL a11 join LU_ITEM a12 on
> (a11.ITEM_ID = a12.ITEM_ID) join LU_SUBCATEG a13 on (a12.SUBCAT_ID =
> a13.SUBCAT_ID) join LU_CATEGORY a14 on (a13.CATEGORY_ID = a14.CATEGORY_ID)
> group by a13.CATEGORY_ID, a12.SUBCAT_ID;
> {code}
> it doesn't work. The root cause is that the flag is set in [one QB
> instance|https://github.com/apache/hive/blob/883d5dfe25929ba5dcac635e752adf5561d28402/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1836],
> but during the plan generation, a genPlan is recursively called, moreover,
> this.qb doesn't refers to the main/root QB, as
> [setQB|https://github.com/apache/hive/blob/883d5dfe25929ba5dcac635e752adf5561d28402/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L12435]
> is called at every step, so when iterating and reaching the file sink
> generation plan, the QB instance passed is not the same as the one on which
> the isInsertOverwriteDirectory flag was set.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
