Hello Aman Sinha, Riza Suminto, Joe McDonnell, Michael Smith, Steve Carlin,
Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22883
to look at the new patch set (#10).
Change subject: IMPALA-13982: Support regular views for Calcite planner in
Impala
......................................................................
IMPALA-13982: Support regular views for Calcite planner in Impala
Before this patch, the Calcite planner in Impala only supported inline
views like 'temp' in the following query.
select id from (
select * from functional.alltypes
) as temp;
Regular views, on the other hand, were not supported. For instance, the
Calcite planner in Impala did not support regular views like
'functional.alltypes_view' created via the following statement and
hence queries against such regular views like
"select id from functional.alltypes_view" were not supported.
CREATE VIEW functional.alltypes_view
AS SELECT * FROM functional.alltypes;
This patch adds the support for regular views to the Calcite planner
via adding a ViewTableMacro for each regular view in the given query
when populating the Calcite schema. This is also how regular views are
supported in PlannerTest#testView() at
https://github.com/apache/calcite/blob/main/core/src/test/java/org/apache/calcite/tools/PlannerTest.java
in Apache Calcite. Specifically, each regular view is added in
CalciteAssert#addSchema() that instantiates a ViewTableMacro so that
the regular view could be expanded and validated later on.
To be able to expand the view in ViewTableMacro, we need a newer
version of protobuf-java, e.g., 3.21.12. Currently 2.5.0 is
transitively pulled in by dependencies like
org.apache.hadoop:hadoop-hdfs. To address this issue, based on the
suggestion from Stamatis Zampetakis, we exclude avatica-core from the
artifacts that are pulled in by calcite-core, and explicitly add
avatica as a dependency, which already shades the protobuf-java it
requires to expand a regular view.
The patch supports the following cases, where
'functional.alltypes_view' is a regular view defined as
"SELECT * FROM functional.alltypes".
1. select id from functional.alltypes_view.
2. select alltypes_view.id from functional.alltypes_view.
3. select functional.alltypes_view.id from functional.alltypes_view.
Joining a regular view with an HDFS table like the following is also
supported.
select alltypestiny.id
from functional.alltypes_view, functional.alltypestiny
Note that after this patch, queries against regular views are supported
only in the legacy catalog mode but not the local catalog mode. In
fact, queries against HDFS tables in the local catalog mode are not
supported yet by the Calcite planner either. We will deal with this in
IMPALA-14080.
Testing:
- Added test cases mentioned above to calcite.test. This makes sure
the test cases are supported when we start the Impala server with
the flag of '--use_calcite_planner=true'.
- Manually verified the test cases above are supported if we start
the Impala server with the environment variable USE_CALCITE_PLANNER
set to true and the query option use_calcite_planner set to 1.
Change-Id: I600aae816727ae942fb221fae84c2aac63ae1893
---
M java/calcite-planner/pom.xml
A
java/calcite-planner/src/main/java/org/apache/impala/calcite/schema/ImpalaViewTableMacro.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteAnalysisDriver.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteMetadataHandler.java
M
java/calcite-planner/src/main/java/org/apache/impala/calcite/service/CalciteRelNodeConverter.java
M testdata/workloads/functional-query/queries/QueryTest/calcite.test
6 files changed, 278 insertions(+), 17 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/22883/10
--
To view, visit http://gerrit.cloudera.org:8080/22883
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I600aae816727ae942fb221fae84c2aac63ae1893
Gerrit-Change-Number: 22883
Gerrit-PatchSet: 10
Gerrit-Owner: Fang-Yu Rao <fangyu....@cloudera.com>
Gerrit-Reviewer: Aman Sinha <amsi...@cloudera.com>
Gerrit-Reviewer: Fang-Yu Rao <fangyu....@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com>
Gerrit-Reviewer: Michael Smith <michael.sm...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com>
Gerrit-Reviewer: Steve Carlin <scar...@cloudera.com>
