[
https://issues.apache.org/jira/browse/IMPALA-14081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18019672#comment-18019672
]
ASF subversion and git services commented on IMPALA-14081:
----------------------------------------------------------
Commit 826c8cf9b083ed951fbfe649dc9551436efceb7e in impala's branch
refs/heads/master from jichen0919
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=826c8cf9b ]
IMPALA-14081: Support create/drop paimon table for impala
This patch mainly implement the creation/drop of paimon table
through impala.
Supported impala data types:
- BOOLEAN
- TINYINT
- SMALLINT
- INTEGER
- BIGINT
- FLOAT
- DOUBLE
- STRING
- DECIMAL(P,S)
- TIMESTAMP
- CHAR(N)
- VARCHAR(N)
- BINARY
- DATE
Syntax for creating paimon table:
CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
(
[col_name data_type ,...]
[PRIMARY KEY (col1,col2)]
)
[PARTITIONED BY (col_name data_type [COMMENT 'col_comment'], ...)]
STORED AS PAIMON
[LOCATION 'hdfs_path']
[TBLPROPERTIES (
'primary-key'='col1,col2',
'file.format' = 'orc/parquet',
'bucket' = '2',
'bucket-key' = 'col3',
];
Two types of paimon catalogs are supported.
(1) Create table with hive catalog:
CREATE TABLE paimon_hive_cat(userid INT,movieId INT)
STORED AS PAIMON;
(2) Create table with hadoop catalog:
CREATE [EXTERNAL] TABLE paimon_hadoop_cat
STORED AS PAIMON
TBLPROPERTIES('paimon.catalog'='hadoop',
'paimon.catalog_location'='/path/to/paimon_hadoop_catalog',
'paimon.table_identifier'='paimondb.paimontable');
SHOW TABLE STAT/SHOW COLUMN STAT/SHOW PARTITIONS/SHOW FILES
statements are also supported.
TODO:
- Patches pending submission:
- Query support for paimon data files.
- Partition pruning and predicate push down.
- Query support with time travel.
- Query support for paimon meta tables.
- WIP:
- Complex type query support.
- Virtual Column query support for querying
paimon data table.
- Native paimon table scanner, instead of
jni based.
Testing:
- Add unit test for paimon impala type conversion.
- Add unit test for ToSqlTest.java.
- Add unit test for AnalyzeDDLTest.java.
- Update default_file_format TestEnumCase in
be/src/service/query-options-test.cc.
- Update test case in
testdata/workloads/functional-query/queries/QueryTest/set.test.
- Add test cases in metadata/test_show_create_table.py.
- Add custom test test_paimon.py.
Change-Id: I57e77f28151e4a91353ef77050f9f0cd7d9d05ef
Reviewed-on: http://gerrit.cloudera.org:8080/22914
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Riza Suminto <[email protected]>
> Support create/drop paimon table for impala
> -------------------------------------------
>
> Key: IMPALA-14081
> URL: https://issues.apache.org/jira/browse/IMPALA-14081
> Project: IMPALA
> Issue Type: Sub-task
> Reporter: jichen
> Assignee: jichen
> Priority: Minor
> Labels: paimon
>
> this patch mainly implement the creation/drop of paimon table through impala.
> syntax for creating paimon table:
> CREATE [EXTERNAL] TABLE [IF NOT EXISTS] [db_name.]table_name
> (
> [col_name data_type ,...]
> [PRIMARY KEY (col1,col2)]
> )
> [PARTITIONED BY (col_name data_type [COMMENT 'col_comment'], ...)]
> STORED AS PAIMON
> [LOCATION 'hdfs_path']
> [TBLPROPERTIES (
> 'primary-key'='col1,col2',
> 'file.format' = 'orc/parquet',
> 'bucket' = '2',
> 'bucket-key' = 'col3',
> ];
> two types of paimon catalogs are supported.
> (1) create table with hive catalog:
> CREATE TABLE paimon_hive_cat(userid INT,movieId INT)
> STORED AS PAIMON;
> (2) create table with hadoop catalog:
> CREATE EXTERNAL TABLE paimon_hadoop_cat
> STORED AS PAIMON
> TBLPROPERTIES('paimon.catalog'='hadoop',
> 'paimon.catalog_location'='/path/to/paimon_hadoop_catalog',
> 'paimon.table_identifier'='paimondb.paimontable');
> SHOW TABLE STAT/SHOW COLUMN STAT/SHOW PARTITIONS/SHOW FILES/SHOW CREATE TABLE
> statements are also supported.
> Testing:
> - Add test cases in metadata/test_show_create_table.py.
> - Add custom cluster test test_paimon.py.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
