[
https://issues.apache.org/jira/browse/IMPALA-14569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18041928#comment-18041928
]
ASF subversion and git services commented on IMPALA-14569:
----------------------------------------------------------
Commit 6cf21464b4bc80bd7a0219e4e4e2f42da694476b in impala's branch
refs/heads/master from Peter Rozsa
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=6cf21464b ]
IMPALA-14569: Fix IllegalStateException in partition pruning on type mismatch
This fixes an IllegalStateException in HdfsPartitionPruner when
evaluating 'IN' predicates whose consist of two compatible types, for
example DATE and STRING: date_col in (<date as string>).
Previously, 'canEvalUsingPartitionMd' did not check if the slot type
matched the literal type. This caused the frontend to attempt invalid
comparisons via 'LiteralExpr.compareTo', leading to
IllegalStateException or incorrect pruning.
The fix ensures 'canEvalUsingPartitionMd' returns false on type
mismatches, deferring evaluation to the backend where proper casting
occurs.
Testing:
- Added regression test in hdfs-partition-pruning.test.
Change-Id: Idc226a628c8df559329a060cb963b81e27e21eda
Reviewed-on: http://gerrit.cloudera.org:8080/23706
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Partition pruning fails with IllegalStateException on mismatching types
> -----------------------------------------------------------------------
>
> Key: IMPALA-14569
> URL: https://issues.apache.org/jira/browse/IMPALA-14569
> Project: IMPALA
> Issue Type: Bug
> Components: Frontend
> Affects Versions: Impala 4.5.0
> Reporter: Peter Rozsa
> Assignee: Peter Rozsa
> Priority: Major
>
> HDFS Partition pruning fails with
> {code:java}
> java.lang.IllegalStateException
> at
> com.google.common.base.Preconditions.checkState(Preconditions.java:496)
> at
> org.apache.impala.analysis.LiteralExpr.compareTo(LiteralExpr.java:304)
> at
> org.apache.impala.analysis.DateLiteral.compareTo(DateLiteral.java:120)
> at
> org.apache.impala.analysis.DateLiteral.compareTo(DateLiteral.java:34)
> at java.util.TreeMap.getEntry(TreeMap.java:352)
> at java.util.TreeMap.get(TreeMap.java:278)
> at
> org.apache.impala.planner.HdfsPartitionPruner.evalInPredicate(HdfsPartitionPruner.java:449)
> at
> org.apache.impala.planner.HdfsPartitionPruner.evalSlotBindingFilter(HdfsPartitionPruner.java:510)
> at
> org.apache.impala.planner.HdfsPartitionPruner.prunePartitions(HdfsPartitionPruner.java:155)
> at
> org.apache.impala.planner.SingleNodePlanner.createHdfsScanPlan(SingleNodePlanner.java:1614)
> at
> org.apache.impala.planner.SingleNodePlanner.createScanNode(SingleNodePlanner.java:1907)
> at
> org.apache.impala.planner.SingleNodePlanner.createTableRefNode(SingleNodePlanner.java:2239)
> at
> org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(SingleNodePlanner.java:938)
> at
> org.apache.impala.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:753)
> at
> org.apache.impala.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:285)
> at
> org.apache.impala.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:177)
> at
> org.apache.impala.planner.Planner.createPlanFragments(Planner.java:137)
> at org.apache.impala.planner.Planner.createPlans(Planner.java:314)
> at
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:2041)
> at
> org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:3363)
> at
> org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:3137)
> at
> org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:2545)
> at
> org.apache.impala.service.Frontend.getTExecRequestWithFallback(Frontend.java:2414)
> at
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:2103)
> at
> org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:179){code}
> or older versions just skip matching partitions, as the comparison between
> different types in 'HdfsPartitionPruner.java:evalInPredicate' returns a false
> result. The problem comes from 'canEvalUsingPartitionMd' as it fails to
> detect the case when the bound slot and the literals in the IN clause have
> different types.
> Steps to reproduce:
> {code:java}
> create table a(a int) partitioned by (date_stored_as_string string);create
> table b(a int) partitioned by (date_stored_as_date date);insert into a(a,
> date_stored_as_string) values(1, '2025-12-12');insert into b(a,
> date_stored_as_date) values(1, '2025-12-12');select * from b left outer join
> a on date_stored_as_date = date_stored_as_string where date_stored_as_date in
> ( '2025-12-12'); {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
