ji chen has uploaded a new patch set (#11). (
http://gerrit.cloudera.org:8080/23613 )
Change subject: IMPALA-14092 Part2: Support querying of paimon data table via
JNI
......................................................................
IMPALA-14092 Part2: Support querying of paimon data table via JNI
This patch mainly implement the querying of paimon data table
through JNI based scanner.
Features implemented:
- support column pruning.
The partition pruning and predicate push down will be submitted
as the third part of the patch.
We implemented this by treating the paimon table as normal
unpartitioned table. When querying paimon table:
- PaimonScanNode will decide paimon splits need to be scanned,
and then transfer splits to BE do the jni-based scan operation.
- We also collect the required columns that need to be scanned,
and pass the columns to Scanner for column pruning.This is
implemented by passing the field ids of the columns to BE,
instead of column position to support schema evolution.
- In the original implementation, PaimonJniScanner will directly
pass paimon row object to BE, and call corresponding paimon row
field accessor, which is a java method to convert row fields to
impala row batch tuples. We find it is slow due to overhead of
JVM method calling.
To minimize the overhead, we refashioned the implementation,
the PaimonJniScanner will convert the paimon row batches to
arrow recordbatch, which stores data in offheap region of
impala JVM. And PaimonJniScanner will pass the arrow offheap
record batch memory pointer to the BE backend.
BE PaimonJniScanNode will directly read data from JVM offheap
region, and convert the arrow record batch to impala row batch.
The benchmark shows the later implementation is 2.x better
than the original implementation.
Current supported impala data types for query includes:
- BOOLEAN
- TINYINT
- SMALLINT
- INTEGER
- BIGINT
- FLOAT
- DOUBLE
- STRING
- DECIMAL(P,S)
- TIMESTAMP
- CHAR(N)
- VARCHAR(N)
- BINARY
- DATE
TODO:
- Patches pending submission:
- Support tpcds/tpch data-loading
for paimon data table.
- Virtual Column query support for querying
paimon data table.
- Query support with time travel.
- Query support for paimon meta tables.
- WIP:
- Snapshot incremental read.
- Complex type query support.
- Native paimon table scanner, instead of
jni based.
Testing:
- Create tests table in functional_schema_template.sql
- Add TestPaimonScannerWithLimit in test_scanners.py
- Add test_paimon_query in test_paimon.py.
- Already passed the tpcds/tpch test for paimon table, due to the
testing table data is currently generated by spark, and it is
not supported by impala now, we have to do this since hive
doesn't support generating paimon table for dynamic-partitioned
tables. we plan to submit a separate patch for tpcds/tpch data
loading and associated tpcds/tpch query tests.
- JVM Offheap memory leak tests, have run looped tpch tests for
1 day, no obvious offheap memory increase is observed,
offheap memory usage is within 10M.
Change-Id: Ie679a89a8cc21d52b583422336b9f747bdf37384
---
M CMakeLists.txt
M be/CMakeLists.txt
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/paimon/CMakeLists.txt
A be/src/exec/paimon/paimon-jni-row-reader.cc
A be/src/exec/paimon/paimon-jni-row-reader.h
A be/src/exec/paimon/paimon-jni-scan-node.cc
A be/src/exec/paimon/paimon-jni-scan-node.h
A be/src/exec/paimon/paimon-jni-scanner.cc
A be/src/exec/paimon/paimon-jni-scanner.h
A be/src/exec/paimon/paimon-scan-plan-node.cc
A be/src/exec/paimon/paimon-scan-plan-node.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/scheduler.cc
M be/src/service/impalad-main.cc
M bin/bootstrap_toolchain.py
M bin/impala-config.sh
A cmake_modules/FindArrow.cmake
M common/thrift/CatalogObjects.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/Types.thrift
M fe/pom.xml
M fe/src/main/java/org/apache/impala/catalog/Column.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/Type.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalPaimonTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
A fe/src/main/java/org/apache/impala/catalog/paimon/PaimonColumn.java
M fe/src/main/java/org/apache/impala/catalog/paimon/PaimonHiveTypeUtils.java
M fe/src/main/java/org/apache/impala/catalog/paimon/PaimonImpalaTypeUtils.java
A fe/src/main/java/org/apache/impala/catalog/paimon/PaimonStructField.java
M fe/src/main/java/org/apache/impala/catalog/paimon/PaimonTable.java
M fe/src/main/java/org/apache/impala/catalog/paimon/PaimonUtil.java
A fe/src/main/java/org/apache/impala/planner/PaimonScanNode.java
A fe/src/main/java/org/apache/impala/planner/PaimonScanPlanner.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
A fe/src/main/java/org/apache/impala/planner/paimon/PaimonSplit.java
A fe/src/main/java/org/apache/impala/util/paimon/ArrowRootAllocation.java
A
fe/src/main/java/org/apache/impala/util/paimon/PaimonArrowFieldTypeFactory.java
A
fe/src/main/java/org/apache/impala/util/paimon/PaimonArrowFieldWriterFactory.java
A
fe/src/main/java/org/apache/impala/util/paimon/PaimonArrowFormatNativeWriter.java
A fe/src/main/java/org/apache/impala/util/paimon/PaimonArrowFormatWriter.java
A fe/src/main/java/org/apache/impala/util/paimon/PaimonArrowUtils.java
A fe/src/main/java/org/apache/impala/util/paimon/PaimonJniScanner.java
M fe/src/test/java/org/apache/impala/catalog/paimon/ImpalaTypeUtilsTest.java
M java/pom.xml
A testdata/data/paimon_test/README.md
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-275038ac-cf80-4caa-b9c3-43854884f257-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-4724d1a4-637d-4275-8439-d4edd1a4583e-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-7201ded8-6883-42ef-a39f-a99fc8a83938-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-7e80aa3a-5863-4cb5-bdb4-84d0d9002dd0-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-81c20e92-de0c-4ea1-bf5e-989ecb844645-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-9a6dbc25-fd98-471d-adca-02aa6c8f2751-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-9b24cb51-8c3a-448b-a0f3-605ed71e7caa-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-c079a770-9eb6-4b91-b3c8-f6c84fbaf7ce-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-d943b655-eed4-4ee0-9a2e-024ff4596832-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-dffe00a3-de35-4a74-96df-cd0fb7d08070-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-e6940374-4033-48c7-b6d9-ebca0e7d0a4d-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/bucket-0/data-eef44c7a-9856-4817-8f1f-a2a7dbc331c5-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/manifest/manifest-fab63fbc-acd6-424a-ac4e-2a848ea51f52-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/manifest/manifest-list-fa3d4f50-7f45-4d37-b5e7-22e6cf3f2cb0-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/manifest/manifest-list-fa3d4f50-7f45-4d37-b5e7-22e6cf3f2cb0-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/schema/schema-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/snapshot/EARLIEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/snapshot/LATEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_paimon/snapshot/snapshot-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-04a2d395-e55d-4720-bdf9-d0aea3177a01-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-0ff3cbcd-bd56-43ef-8b00-543a83cc5e6f-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-15fe0349-ce6f-4228-9031-f1d556dfc808-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-43259d5c-33d4-4fa2-bed0-40e8966ed9e0-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-4e2d864e-0aa9-4b5e-8b6d-4261540072f3-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-602c116d-3e51-40b6-914d-950170ece098-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-6a5bf48f-e00f-45dc-8d44-62582cf6d881-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-72d3fa1b-ff44-4088-a2d6-22902d35e795-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-7f917551-249f-4a13-9ec9-43be271f7b21-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-8b7a002b-cf8d-4d17-ba96-0389d2dfb456-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-b5785d5b-c764-429f-82c5-aa76bbf15a63-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/bucket-0/data-f71174f8-719a-4ed5-84d0-244a6c26604b-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/manifest/manifest-85ea4264-6480-4517-9d44-2a5ba022a922-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/manifest/manifest-list-67f31438-ac2d-4d03-9bd9-4e1d55120c5c-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/manifest/manifest-list-67f31438-ac2d-4d03-9bd9-4e1d55120c5c-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/schema/schema-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/snapshot/EARLIEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/snapshot/LATEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/alltypes_structs_paimon/snapshot/snapshot-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-58e01102-7ce1-4410-8022-50afc44aa215-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-6d2bcd70-7bfc-4e04-8a37-52bb441bccc1-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-7f39c604-2f10-4bea-bd61-32787ba71065-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-8def1f02-46fd-4114-8c89-cbae117a1b61-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-a100aae7-ed6e-4f04-8648-be059c9e2942-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-bddf22a5-302c-4e14-bf86-dd897d868b42-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-cf7233f7-0e36-49e0-bd10-7b1f94ca1408-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/bucket-0/data-ddc2526d-a35f-4c0f-91e8-def352b6432e-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/manifest/manifest-1e64bd23-89a3-40ab-8f11-b11467e340fe-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/manifest/manifest-8c8eef70-e1cc-4019-bd2b-1df0c773321a-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/manifest/manifest-list-b15e96d1-cea5-44c2-a5b4-dc8c7041463f-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/manifest/manifest-list-b15e96d1-cea5-44c2-a5b4-dc8c7041463f-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/manifest/manifest-list-b9930407-daa0-43cc-b4fa-107aef9a5ffc-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/manifest/manifest-list-b9930407-daa0-43cc-b4fa-107aef9a5ffc-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/schema/schema-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/snapshot/EARLIEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/snapshot/LATEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/snapshot/snapshot-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_decimal_tbl/snapshot/snapshot-2
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-3728bff4-abe9-4ab9-8837-081b21a80e70-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-493438a3-8481-4c98-8125-856ff5b42a12-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-59e8fabf-ccba-47d6-a88a-0571d4e0e7a0-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-6db45077-3cc7-46cf-b0c8-f7b5f42c19f7-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-a73a0979-5cb2-4876-b7eb-1a8d8dcd90f7-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-af00a36c-e45c-48c3-8004-9055399d7147-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/bucket-0/data-b9ea2c17-a695-4a31-98da-ac1da8ca242f-0.parquet
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/manifest/manifest-32d94d25-0a6f-499e-9953-e4d2dd8de146-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/manifest/manifest-65f28661-1546-4a1a-930e-f31118e55bd9-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/manifest/manifest-list-5db2c8c6-e253-4fba-b0c2-71dc19d66678-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/manifest/manifest-list-5db2c8c6-e253-4fba-b0c2-71dc19d66678-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/manifest/manifest-list-c6343add-3b4b-4a00-a470-de159bd83b3d-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/manifest/manifest-list-c6343add-3b4b-4a00-a470-de159bd83b3d-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/schema/schema-0
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/snapshot/EARLIEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/snapshot/LATEST
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/snapshot/snapshot-1
A
testdata/data/paimon_test/paimon_catalog/warehouse/functional.db/paimon_primitive_alltypes/snapshot/snapshot-2
M testdata/datasets/functional/functional_schema_template.sql
M testdata/datasets/functional/schema_constraints.csv
M testdata/workloads/functional-query/queries/QueryTest/paimon-negative.test
A testdata/workloads/functional-query/queries/QueryTest/paimon-query.test
M tests/query_test/test_paimon.py
M tests/query_test/test_scanners.py
130 files changed, 3,667 insertions(+), 52 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/23613/11
--
To view, visit http://gerrit.cloudera.org:8080/23613
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie679a89a8cc21d52b583422336b9f747bdf37384
Gerrit-Change-Number: 23613
Gerrit-PatchSet: 11
Gerrit-Owner: ji chen <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Riza Suminto <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: ji chen <[email protected]>
