Hello Aman Sinha, Quanlong Huang, Fang-Yu Rao, Joe McDonnell, Michael Smith,
Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24079
to look at the new patch set (#4).
Change subject: IMPALA-14820: Calcite Planner: Memory allocations are too high
......................................................................
IMPALA-14820: Calcite Planner: Memory allocations are too high
The following queries are showing memory allocations that are too
high:
In bloom_filters.test:
with l as (select * from tpch.lineitem UNION ALL select * from tpch.lineitem)
select STRAIGHT_JOIN count(*) from (select * from tpch.lineitem a LIMIT 1) a
join (select * from l LIMIT 125000) b on a.l_orderkey = -b.l_orderkey
In admission-max-min-mem-limits.test:
select * from functional_parquet.alltypes limit 1;
The issue in the first query is that the stats need to be recomputed after
the limit is set. Setting the limit propagates into its children nodes, so
the stats also need to be recomputed when the limit is set.
The issue in the second query is that the top level needs to know when there
is only one row output, and the case with a "limit 1" clause was missing from
the check. Note: Calcite uses the Sort RelNode for "order by", "limit" and
"offset".
The bloom_filters.test is still incorrect, but the memory allocation is much
lower after this commit. IMPALA-14811 has been created for this issue.
Change-Id: I9e84061b65f115ea65a9d2c2d40173843040ca0c
---
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/rel/node/ImpalaSortRel.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteSingleNodePlanner.java
2 files changed, 17 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/79/24079/4
--
To view, visit http://gerrit.cloudera.org:8080/24079
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9e84061b65f115ea65a9d2c2d40173843040ca0c
Gerrit-Change-Number: 24079
Gerrit-PatchSet: 4
Gerrit-Owner: Steve Carlin <[email protected]>
Gerrit-Reviewer: Aman Sinha <[email protected]>
Gerrit-Reviewer: Fang-Yu Rao <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Steve Carlin <[email protected]>
