Zoltan Borok-Nagy has uploaded this change for review. (
http://gerrit.cloudera.org:8080/21950
Change subject: IMPALA-13463: Impala should ignore case of Iceberg schema
elements
......................................................................
IMPALA-13463: Impala should ignore case of Iceberg schema elements
Schema is case insensitive in Impala. Via Spark it's possible to create
schema elements with upper/lower case letters and store them in the
metadata JSON files of Iceberg, e.g.:
"schemas" : [ {
"type" : "struct",
"schema-id" : 0,
"fields" : [ {
"id" : 1,
"name" : "ID",
"required" : false,
"type" : "string"
}, {
"id" : 2,
"name" : "OWNERID",
"required" : false,
"type" : "string"
} ]
} ],
This can cause problems in Impala during predicate pushdown, as we can
get a ValidationException from the Iceberg library (as Impala pushes
down predicates with lower case column names, while Iceberg sees upper
case names).
With this patch Impala invokes Scan.caseSensitive(boolean caseSensitive)
on the TableScan object to set case insensitivity.
Testing:
* added e2e test
Change-Id: Iedaf152d8a0c02a124c3dcf8acb59b4ba4e81cf4
---
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
A
testdata/workloads/functional-query/queries/QueryTest/iceberg-column-case-sensitivity-issue.test
M tests/query_test/test_iceberg.py
3 files changed, 33 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/21950/1
--
To view, visit http://gerrit.cloudera.org:8080/21950
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Iedaf152d8a0c02a124c3dcf8acb59b4ba4e81cf4
Gerrit-Change-Number: 21950
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
