[
https://issues.apache.org/jira/browse/HIVE-28564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis resolved HIVE-28564.
----------------------------------------
Fix Version/s: 4.1.0
Resolution: Fixed
Fixed in
[https://github.com/apache/hive/commit/b2a5933c0f396ade25273ac75cbf35e205969b21]
Thanks for the PR [~soumyakanti.das] !
> AssertionError for jdbc queries when JDBCExpandExpressionsRule produces non
> flat Filter conditions
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-28564
> URL: https://issues.apache.org/jira/browse/HIVE-28564
> Project: Hive
> Issue Type: Bug
> Security Level: Public(Viewable by anyone)
> Components: CBO, JDBC
> Affects Versions: 4.1.0
> Reporter: Soumyakanti Das
> Assignee: Soumyakanti Das
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
> Attachments: jdbc_non_flat_filter_condition.q,
> q_test_tpcds_tables_schema.postgres.sql
>
>
> To repro this, create four tpcds table in postgres by copying
> [^q_test_tpcds_tables_schema.postgres.sql] to {{hive/data/scripts/}}
> directory.
> Add [^jdbc_non_flat_filter_condition.q] to
> {{ql/src/test/queries/clientpositive/}} and run
> {noformat}
> mvn test -pl itests/qtest -Pitests -Dtest=TestMiniLlapLocalCliDriver
> -Dtest.output.overwrite=true
> -Dqfile=jdbc_non_flat_filter_condition.q{noformat}
> Error stack:
> {noformat}
> [INFO] Running org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 15.238 s <<< FAILURE! - in
> org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver
> [ERROR]
> org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[jdbc_non_flat_filter_condition]
> Time elapsed: 8.409 s <<< FAILURE!
> java.lang.AssertionError: AND(IN($1, 2000, 1999, 2001), OR(=($1, 2000),
> OR(AND(=($1, 1999), =($2, 12)), AND(=($1, 2001), =($2, 1)))), IS NOT NULL($0))
> at org.apache.calcite.rel.core.Filter.<init>(Filter.java:76)
> at
> org.apache.calcite.adapter.jdbc.JdbcRules$JdbcFilter.<init>(JdbcRules.java:629)
> at
> org.apache.calcite.adapter.jdbc.JdbcRules$JdbcFilter.copy(JdbcRules.java:635)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.jdbc.JDBCExpandExpressionsRule$FilterCondition.onMatch(JDBCExpandExpressionsRule.java:85)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542)
> at org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:271)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleCollection.execute(HepInstruction.java:74)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202)
> at org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2469)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.executeProgram(CalcitePlanner.java:2428)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPostJoinOrderingTransform(CalcitePlanner.java:2347)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1739)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1573)
> at
> org.apache.calcite.tools.Frameworks.lambda$withPlanner$0(Frameworks.java:131)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:914)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:180)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:126)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1325)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:573)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:13164)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:466)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:332)
> at
> org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:180)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:332)
> at org.apache.hadoop.hive.ql.Compiler.analyze(Compiler.java:224)
> at org.apache.hadoop.hive.ql.Compiler.compile(Compiler.java:109)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:499)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:451)
> at
> org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:415){noformat}
>
> In non jdbc Hive query
> ([cbo_query47.q.out|https://github.com/apache/hive/blob/6f7c55ab9bc4fd7c3d0c2a6ba3095275b17b3d2d/ql/src/test/results/clientpositive/perf/tpcds30tb/tez/cbo_query47.q.out#L59]),
> the condition is:
> {noformat}
> AND(IN($6, 2000, 1999, 2001), OR(=($6, 2000), IN(ROW($6, $8), ROW(1999, 12),
> ROW(2001, 1)))){noformat}
> [JDBCExpandExpressionsRule|https://github.com/apache/hive/blob/6f7c55ab9bc4fd7c3d0c2a6ba3095275b17b3d2d/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/JDBCExpandExpressionsRule.java#L209]
> converts it to:
> {noformat}
> AND(IN($1, 2000, 1999, 2001), OR(=($1, 2000), OR(AND(=($1, 1999), =($2, 12)),
> AND(=($1, 2001), =($2, 1)))), IS NOT NULL($0)){noformat}
> which is not flat as there is an OR within an OR which can be simplified
> using {{RexSimplify}} to:
> {noformat}
> AND(OR(=($1, 2000), AND(=($1, 1999), =($2, 12)), AND(=($1, 2001), =($2, 1))),
> IN($1, 2000, 1999, 2001), IS NOT NULL($0)){noformat}
>
> This can also be reproduced with:
> {noformat}
> --! qt:database:postgres:q_test_book_table.sql
> CREATE EXTERNAL TABLE book (id int, title varchar(100), author int)
> STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
> "hive.sql.database.type" = "POSTGRES",
> "hive.sql.jdbc.driver" = "org.postgresql.Driver",
> "hive.sql.jdbc.url" = "jdbc:postgresql://localhost:5432/qtestDB",
> "hive.sql.dbcp.username" = "qtestuser",
> "hive.sql.dbcp.password" = "qtestpassword",
> "hive.sql.table" = "book");
> explain cbo
> select * from book
> where id = 0 or (id = 1 and author = 11) or (id = 2 and author =
> 22);{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
