ji chen has uploaded a new patch set (#15). (
http://gerrit.cloudera.org:8080/22914 )
Change subject: IMPALA-14081: Support create/drop paimon table for impala
......................................................................
IMPALA-14081: Support create/drop paimon table for impala
This patch mainly implement the creation/drop of paimon table
through impala.
Supported impala data types:
- BOOLEAN
- TINYINT
- SMALLINT
- INTEGER
- BIGINT
- FLOAT
- DOUBLE
- STRING
- DECIMAL(P,S)
- TIMESTAMP
- CHAR(N)
- VARCHAR(N)
- BINARY
- DATE
Syntax for creating paimon table:
CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
(
[col_name data_type ,...]
[PRIMARY KEY (col1,col2)]
)
[PARTITIONED BY (col_name data_type [COMMENT 'col_comment'], ...)]
STORED AS PAIMON
[LOCATION 'hdfs_path']
[TBLPROPERTIES (
'primary-key'='col1,col2',
'file.format' = 'orc/parquet',
'bucket' = '2',
'bucket-key' = 'col3',
];
Two types of paimon catalogs are supported.
(1) Create table with hive catalog:
CREATE TABLE paimon_hive_cat(userid INT,movieId INT)
STORED AS PAIMON;
(2) Create table with hadoop catalog:
CREATE [EXTERNAL] TABLE paimon_hadoop_cat
STORED AS PAIMON
TBLPROPERTIES('paimon.catalog'='hadoop',
'paimon.catalog_location'='/path/to/paimon_hadoop_catalog',
'paimon.table_identifier'='paimondb.paimontable');
SHOW TABLE STAT/SHOW COLUMN STAT/SHOW PARTITIONS/SHOW FILES
statements are also supported.
TODO:
- Patches pending submission:
- Query support for paimon data files.
- Partition pruning and predicate push down.
- Query support with time travel.
- Query support for paimon meta tables.
- WIP:
- Complex type query support.
- Virtual Column query support for querying
paimon data table.
- Native paimon table scanner, instead of
jni based.
Testing:
- Add unit test for paimon impala type conversion.
- Update default_file_format TestEnumCase in
be/src/service/query-options-test.cc.
- Update test case in
testdata/workloads/functional-query/queries/QueryTest/set.test.
- Add test cases in metadata/test_show_create_table.py.
- Add custom test test_paimon.py.
Change-Id: I57e77f28151e4a91353ef77050f9f0cd7d9d05ef
---
M be/src/service/query-options-test.cc
M bin/impala-config.sh
M bin/rat_exclude_files.txt
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M docs/impala.ditamap
M docs/impala_keydefs.ditamap
A docs/topics/impala_paimon.xml
M docs/topics/impala_reserved_words.xml
M fe/pom.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/AnalysisUtils.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java
M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java
M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
A fe/src/main/java/org/apache/impala/analysis/paimon/PaimonAnalyzer.java
M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java
M fe/src/main/java/org/apache/impala/catalog/HdfsStorageDescriptor.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/VirtualColumn.java
A fe/src/main/java/org/apache/impala/catalog/local/LocalPaimonTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
A fe/src/main/java/org/apache/impala/catalog/paimon/FePaimonTable.java
A fe/src/main/java/org/apache/impala/catalog/paimon/FeShowFileStmtSupport.java
A fe/src/main/java/org/apache/impala/catalog/paimon/ImpalaTypeUtils.java
A fe/src/main/java/org/apache/impala/catalog/paimon/PaimonCatalogOpExecutor.java
A fe/src/main/java/org/apache/impala/catalog/paimon/PaimonTable.java
A
fe/src/main/java/org/apache/impala/catalog/paimon/PaimonTableLoadingException.java
A fe/src/main/java/org/apache/impala/catalog/paimon/PaimonUtil.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/util/DebugUtils.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
A fe/src/test/java/org/apache/impala/catalog/paimon/ImpalaTypeUtilsTest.java
M java/pom.xml
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/bucket-0/data-0ca8873e-4b8c-4a80-ac31-3cd5bd95ae01-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/bucket-0/data-488765b8-d0d7-4091-ad1a-2753f9fb0b63-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/bucket-0/data-5ab52dc4-cd44-4de4-9b81-fcd30baeee0d-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/bucket-0/data-6684e5ef-26fa-4f13-a1bd-9c2a5bc9557c-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/bucket-0/data-ad75a7e8-4ae0-4803-8f49-92d58fd4a222-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/index-manifest-1eecc41b-3dcd-4388-afdb-54527b51be2a-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/manifest-3505f2c0-e77f-42ff-9a54-368038f7eb3e-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/manifest-671ffa6b-2665-4d30-b039-d21a5565d698-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/manifest-9e6bcbde-1da9-4b06-a17c-c71579de18a5-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/manifest-b4b02924-241f-4abf-8ad8-d7a2ac5536e2-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/manifest-c32aa2fd-3d98-4f09-9067-8a269d8115eb-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/manifest-list-025dfcce-8e34-42c0-a8f7-def795d489df-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/manifest-list-025dfcce-8e34-42c0-a8f7-def795d489df-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/manifest-list-5cd761f7-4c95-4181-841e-aa8b405e71be-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/manifest-list-5cd761f7-4c95-4181-841e-aa8b405e71be-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/manifest-list-6e11688a-4b7c-4bfd-9118-5e49eac61016-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/manifest-list-6e11688a-4b7c-4bfd-9118-5e49eac61016-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/manifest-list-d86f1254-05df-4560-8885-693e94b46cbf-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/manifest/manifest-list-d86f1254-05df-4560-8885-693e94b46cbf-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/schema/schema-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/schema/schema-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/snapshot/EARLIEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/snapshot/LATEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/snapshot/snapshot-10
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/snapshot/snapshot-11
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/snapshot/snapshot-12
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/snapshot/snapshot-9
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_non_partitioned/statistics/stat-eaea66dc-9657-4d89-823a-98315aa27329-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/manifest/manifest-1db1d7b8-9b21-4241-a0c6-8189c368a0bc-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/manifest/manifest-5a2ea60f-e1b6-4126-aee4-088ff0b82f28-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/manifest/manifest-9216adfd-451c-486d-a6db-b66884314880-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/manifest/manifest-list-22716ac0-0c0e-4f03-a48c-31c807cbf8d9-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/manifest/manifest-list-22716ac0-0c0e-4f03-a48c-31c807cbf8d9-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/manifest/manifest-list-fececc90-17e5-4cde-bc69-84a98da91b79-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/manifest/manifest-list-fececc90-17e5-4cde-bc69-84a98da91b79-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/schema/schema-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/schema/schema-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/schema/schema-2
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/schema/schema-3
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/schema/schema-4
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/schema/schema-5
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/snapshot/EARLIEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/snapshot/LATEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/snapshot/snapshot-5
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/snapshot/snapshot-6
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/statistics/stat-7061d17d-5a4d-4532-b98f-714d66103721-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/userId=1/bucket-0/data-d7e0b3b7-a5e7-407f-b591-033cdf2da158-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/userId=2/bucket-0/data-010c93fa-86ab-4389-aa03-4d76b7ece890-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/userId=3/bucket-0/data-4decfe67-c7e1-43f9-96ee-ae5188779a39-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/userId=4/bucket-0/data-ee1eb48c-19ba-4275-9a73-ca788d55c57d-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_partitioned/userId=5/bucket-0/data-86902019-80d0-4395-8bd4-666e1ffe2422-0.parquet
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
A testdata/workloads/functional-query/queries/QueryTest/paimon-ddl-negative.test
A testdata/workloads/functional-query/queries/QueryTest/paimon-ddl.test
A testdata/workloads/functional-query/queries/QueryTest/paimon-show-stats.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
A
testdata/workloads/functional-query/queries/QueryTest/show-create-table-paimon.test
M tests/metadata/test_show_create_table.py
A tests/query_test/test_paimon.py
98 files changed, 3,986 insertions(+), 36 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/22914/15
--
To view, visit http://gerrit.cloudera.org:8080/22914
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I57e77f28151e4a91353ef77050f9f0cd7d9d05ef
Gerrit-Change-Number: 22914
Gerrit-PatchSet: 15
Gerrit-Owner: ji chen <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Pranav Lodha <[email protected]>
Gerrit-Reviewer: Riza Suminto <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: ji chen <[email protected]>
