[
https://issues.apache.org/jira/browse/IMPALA-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17909055#comment-17909055
]
ASF subversion and git services commented on IMPALA-13622:
----------------------------------------------------------
Commit b1628d8644d21c3b45acd8b2715a284cf5e8379b in impala's branch
refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=b1628d864 ]
IMPALA-13622: Fix negative cardinality bug in AggregationNode.java
An incomplete COMPUTE STATS during data loading reveal a bug in
AggregationNode.java where estimateNumGroups() can return value less
than -1.
This patch fix the bug by implementing
PlanNode.smallestValidCardinality() and
MathUtil.saturatingMultiplyCardinalities(). Both function validates that
the function arguments are valid cardinality number.
smallestValidCardinality() correctly compares two cardinality numbers
and return the smallest and valid one. It generalizes and replaces
static function PlanNode.capCardinalityAtLimit().
saturatingMultiplyCardinalities() adds validation and normalization over
MathUtil.saturatingMultiply().
Reorder logic of tuple-based estimation from IMPALA-13405 such that
negative estimate is handled properly.
Testing:
- Added more preconditions in AgggregationNode.java.
- Added CardinalityTest.testSmallestValidCardinality and
MathUtilTest.testSaturatingMultiplyCardinality.
- Added test in resource-requirements.test that will consistently fail
without this fix.
- Pass testResourceRequirement.
Change-Id: Ib862a010b2094daa2cbdd5d555e46443009672ad
Reviewed-on: http://gerrit.cloudera.org:8080/22235
Reviewed-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Jason Fehr <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Precondition hit when tpch.customer stats are missing
> -----------------------------------------------------
>
> Key: IMPALA-13622
> URL: https://issues.apache.org/jira/browse/IMPALA-13622
> Project: IMPALA
> Issue Type: Bug
> Reporter: Pranav Yogi Lodha
> Assignee: Riza Suminto
> Priority: Major
> Labels: broken-build
>
> h2. Error Message:
> {noformat}
> Query: select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice,
> sum(l_quantity) from tpch.customer, tpch.orders, tpch.lineitem where
> o_orderkey in ( select l_orderkey from tpch.lineitem group by l_orderkey
> having sum(l_quantity) > 300 ) and c_custkey = o_custkey and o_orderkey =
> l_orderkey group by c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice
> order by o_totalprice desc, o_orderdate limit 100 Error Stack:
> java.lang.IllegalStateException: -600122 at
> com.google.common.base.Preconditions.checkState(Preconditions.java:512) at
> org.apache.impala.planner.AggregationNode.computeStats(AggregationNode.java:294)
> at org.apache.impala.planner.AggregationNode.init(AggregationNode.java:235)
> at
> org.apache.impala.planner.SingleNodePlanner.createAggregationPlan(SingleNodePlanner.java:1115)
> at
> org.apache.impala.planner.SingleNodePlanner.createAggregationPlan(SingleNodePlanner.java:1084)
> at
> org.apache.impala.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:760)
> at
> org.apache.impala.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:281)
> at
> org.apache.impala.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:173)
> at org.apache.impala.planner.Planner.createPlanFragments(Planner.java:132)
> at org.apache.impala.planner.Planner.createPlans(Planner.java:310) at
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1969) at
> org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:2984)
> at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:2742)
> at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:2269) at
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:2030) at
> org.apache.impala.planner.PlannerTestBase.testPlan(PlannerTestBase.java:565)
> at
> org.apache.impala.planner.PlannerTestBase.runTestCase(PlannerTestBase.java:466)
> at
> org.apache.impala.planner.PlannerTestBase.runPlannerTestFile(PlannerTestBase.java:997)
> at
> org.apache.impala.planner.PlannerTestBase.runPlannerTestFile(PlannerTestBase.java:939)
> at
> org.apache.impala.planner.PlannerTest.testResourceRequirements(PlannerTest.java:938)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:316)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:240)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:214)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
> at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495){noformat}
> h2. Stacktrace:
>
> {noformat}
> java.lang.AssertionError: Query: select c_name, c_custkey, o_orderkey,
> o_orderdate, o_totalprice, sum(l_quantity) from tpch.customer, tpch.orders,
> tpch.lineitem where o_orderkey in ( select l_orderkey from tpch.lineitem
> group by l_orderkey having sum(l_quantity) > 300 ) and c_custkey = o_custkey
> and o_orderkey = l_orderkey group by c_name, c_custkey, o_orderkey,
> o_orderdate, o_totalprice order by o_totalprice desc, o_orderdate limit 100
> Error Stack: java.lang.IllegalStateException: -600122 at
> com.google.common.base.Preconditions.checkState(Preconditions.java:512) at
> org.apache.impala.planner.AggregationNode.computeStats(AggregationNode.java:294)
> at org.apache.impala.planner.AggregationNode.init(AggregationNode.java:235)
> at
> org.apache.impala.planner.SingleNodePlanner.createAggregationPlan(SingleNodePlanner.java:1115)
> at
> org.apache.impala.planner.SingleNodePlanner.createAggregationPlan(SingleNodePlanner.java:1084)
> at
> org.apache.impala.planner.SingleNodePlanner.createSelectPlan(SingleNodePlanner.java:760)
> at
> org.apache.impala.planner.SingleNodePlanner.createQueryPlan(SingleNodePlanner.java:281)
> at
> org.apache.impala.planner.SingleNodePlanner.createSingleNodePlan(SingleNodePlanner.java:173)
> at org.apache.impala.planner.Planner.createPlanFragments(Planner.java:132)
> at org.apache.impala.planner.Planner.createPlans(Planner.java:310) at
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1969) at
> org.apache.impala.service.Frontend.getPlannedExecRequest(Frontend.java:2984)
> at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:2742)
> at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:2269) at
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:2030) at
> org.apache.impala.planner.PlannerTestBase.testPlan(PlannerTestBase.java:565)
> at
> org.apache.impala.planner.PlannerTestBase.runTestCase(PlannerTestBase.java:466)
> at
> org.apache.impala.planner.PlannerTestBase.runPlannerTestFile(PlannerTestBase.java:997)
> at
> org.apache.impala.planner.PlannerTestBase.runPlannerTestFile(PlannerTestBase.java:939)
> at
> org.apache.impala.planner.PlannerTest.testResourceRequirements(PlannerTest.java:938)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:316)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:240)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:214)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:385)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
> at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:507)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:495)
>
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
