[
https://issues.apache.org/jira/browse/CALCITE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16946209#comment-16946209
]
Julian Hyde commented on CALCITE-2792:
--------------------------------------
The balanced tree is a clever idea but it seems to be masking the problem.
The real solution is to not create huge expressions. So, a fix to CALCITE-2696
would be good.
Also, I think it would be useful to create a "$HARD_IN" function that has the
same arguments as IN but is not expanded to OR. It can pass through the
planning process unchanged. Users wouldn't reference it in their SQL, but
SqlToRelConverter could introduce it based on a parameter setting. Because it
has a large argument list it would be "flat" and not susceptible to
StackOverflowError. People could add some conservative optimizations.
> Stackoverflow while evaluating filter with large number of OR conditions
> ------------------------------------------------------------------------
>
> Key: CALCITE-2792
> URL: https://issues.apache.org/jira/browse/CALCITE-2792
> Project: Calcite
> Issue Type: Bug
> Components: core
> Affects Versions: 1.18.0
> Reporter: Dirk Mahler
> Priority: Major
> Labels: pull-request-available
> Attachments: calcite-stackoverflow.zip
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> As a workaround for CALCITE-2696 we're currently using OR conditions for
> filtering values, e.g. instead of
> {noformat}
> ... WHERE value2 IN (1,2,3)
> {noformat}
> {noformat}
> ... WHERE value2=1 OR value2=2 OR value2=3
> {noformat}
> We're now hitting a StackOverflowError because the number of values in the
> filter grows quite large (i.e. 1000-3000) and obviously the evaluation
> recursive:
> {noformat}
> java.lang.StackOverflowError
> at java.util.AbstractCollection.toArray(AbstractCollection.java:176)
> at
> org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.<init>(SqlShuttle.java:111)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visitScoped(SqlValidatorImpl.java:5699)
> at
> org.apache.calcite.sql.validate.SqlScopedShuttle.visit(SqlScopedShuttle.java:50)
> at
> org.apache.calcite.sql.validate.SqlScopedShuttle.visit(SqlScopedShuttle.java:33)
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:138)
> at
> org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.visitChild(SqlShuttle.java:134)
> at
> org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.visitChild(SqlShuttle.java:101)
> at org.apache.calcite.sql.SqlOperator.acceptCall(SqlOperator.java:865)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visitScoped(SqlValidatorImpl.java:5701)
> at
> org.apache.calcite.sql.validate.SqlScopedShuttle.visit(SqlScopedShuttle.java:50)
> at
> org.apache.calcite.sql.validate.SqlScopedShuttle.visit(SqlScopedShuttle.java:33)
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:138)
> at
> org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.visitChild(SqlShuttle.java:134)
> at
> org.apache.calcite.sql.util.SqlShuttle$CallCopyingArgHandler.visitChild(SqlShuttle.java:101)
> at org.apache.calcite.sql.SqlOperator.acceptCall(SqlOperator.java:865)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$Expander.visitScoped(SqlValidatorImpl.java:5701)
> at
> org.apache.calcite.sql.validate.SqlScopedShuttle.visit(SqlScopedShuttle.java:50)
> at
> org.apache.calcite.sql.validate.SqlScopedShuttle.visit(SqlScopedShuttle.java:33)
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:138)
> ...
> {noformat}
> We tried to increase the stack size of the virtual machine (-Xss8000k) but
> this leads to a problem with Janino:
> {noformat}
> java.sql.SQLException: exception while executing query: Compiling "Buzz":
> Code of method
> "execute(Lorg/apache/calcite/interpreter/Context;[Ljava/lang/Object;)V" of
> class "Buzz" grows beyond 64 KB
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:577)
> at
> org.apache.calcite.avatica.AvaticaPreparedStatement.executeQuery(AvaticaPreparedStatement.java:137)
> at
> org.apache.calcite.issue.StackOverflowTest.stackOverflow(StackOverflowTest.java:45)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
> com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
> at
> com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
> Caused by: org.codehaus.janino.InternalCompilerException: Compiling "Buzz":
> Code of method
> "execute(Lorg/apache/calcite/interpreter/Context;[Ljava/lang/Object;)V" of
> class "Buzz" grows beyond 64 KB
> at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:382)
> at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:237)
> at
> org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:465)
> at
> org.codehaus.janino.ClassBodyEvaluator.compileToClass(ClassBodyEvaluator.java:313)
> at
> org.codehaus.janino.ClassBodyEvaluator.cook(ClassBodyEvaluator.java:235)
> at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:207)
> at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:50)
> at
> org.codehaus.janino.ClassBodyEvaluator.createInstance(ClassBodyEvaluator.java:347)
> at
> org.apache.calcite.interpreter.JaninoRexCompiler.getScalar(JaninoRexCompiler.java:176)
> at
> org.apache.calcite.interpreter.JaninoRexCompiler.baz(JaninoRexCompiler.java:153)
> at
> org.apache.calcite.interpreter.JaninoRexCompiler.compile(JaninoRexCompiler.java:111)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.compile(Interpreter.java:487)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.compile(Nodes.java:42)
> at
> org.apache.calcite.interpreter.TableScanNode.createEnumerable(TableScanNode.java:266)
> at
> org.apache.calcite.interpreter.TableScanNode.createProjectableFilterable(TableScanNode.java:233)
> at
> org.apache.calcite.interpreter.TableScanNode.create(TableScanNode.java:81)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:69)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.calcite.util.ReflectUtil.invokeVisitorInternal(ReflectUtil.java:257)
> at
> org.apache.calcite.util.ReflectUtil.invokeVisitor(ReflectUtil.java:214)
> at
> org.apache.calcite.util.ReflectUtil$1.invokeVisitor(ReflectUtil.java:464)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visit(Interpreter.java:451)
> at
> org.apache.calcite.interpreter.Nodes$CoreCompiler.visit(Nodes.java:42)
> at
> org.apache.calcite.interpreter.Interpreter$CompilerImpl.visitRoot(Interpreter.java:405)
> at
> org.apache.calcite.interpreter.Interpreter.<init>(Interpreter.java:88)
> at Baz.bind(Unknown Source)
> at
> org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:355)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:309)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:506)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:497)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:182)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:64)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:43)
> at
> org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:573)
> ... 26 more
> Caused by: org.codehaus.janino.InternalCompilerException: Code of method
> "execute(Lorg/apache/calcite/interpreter/Context;[Ljava/lang/Object;)V" of
> class "Buzz" grows beyond 64 KB
> at org.codehaus.janino.CodeContext.makeSpace(CodeContext.java:1048)
> at org.codehaus.janino.CodeContext.write(CodeContext.java:925)
> at org.codehaus.janino.UnitCompiler.writeByte(UnitCompiler.java:12275)
> at org.codehaus.janino.UnitCompiler.load(UnitCompiler.java:11936)
> at org.codehaus.janino.UnitCompiler.load(UnitCompiler.java:11926)
> at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:4465)
> at org.codehaus.janino.UnitCompiler.access$8000(UnitCompiler.java:215)
> at
> org.codehaus.janino.UnitCompiler$16$1.visitLocalVariableAccess(UnitCompiler.java:4408)
> at
> org.codehaus.janino.UnitCompiler$16$1.visitLocalVariableAccess(UnitCompiler.java:4400)
> at org.codehaus.janino.Java$LocalVariableAccess.accept(Java.java:4274)
> at
> org.codehaus.janino.UnitCompiler$16.visitLvalue(UnitCompiler.java:4400)
> at
> org.codehaus.janino.UnitCompiler$16.visitLvalue(UnitCompiler.java:4396)
> at org.codehaus.janino.Java$Lvalue.accept(Java.java:4148)
> at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:4396)
> at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:4461)
> at org.codehaus.janino.UnitCompiler.access$7500(UnitCompiler.java:215)
> at
> org.codehaus.janino.UnitCompiler$16$1.visitAmbiguousName(UnitCompiler.java:4403)
> at
> org.codehaus.janino.UnitCompiler$16$1.visitAmbiguousName(UnitCompiler.java:4400)
> at org.codehaus.janino.Java$AmbiguousName.accept(Java.java:4224)
> at
> org.codehaus.janino.UnitCompiler$16.visitLvalue(UnitCompiler.java:4400)
> at
> org.codehaus.janino.UnitCompiler$16.visitLvalue(UnitCompiler.java:4396)
> at org.codehaus.janino.Java$Lvalue.accept(Java.java:4148)
> at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:4396)
> at
> org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:5662)
> at
> org.codehaus.janino.UnitCompiler.compileBoolean2(UnitCompiler.java:4120)
> at org.codehaus.janino.UnitCompiler.access$6600(UnitCompiler.java:215)
> at
> org.codehaus.janino.UnitCompiler$14.visitBinaryOperation(UnitCompiler.java:3957)
> at
> org.codehaus.janino.UnitCompiler$14.visitBinaryOperation(UnitCompiler.java:3935)
> at org.codehaus.janino.Java$BinaryOperation.accept(Java.java:4864)
> at
> org.codehaus.janino.UnitCompiler.compileBoolean(UnitCompiler.java:3935)
> at
> org.codehaus.janino.UnitCompiler.compileBoolean2(UnitCompiler.java:4078)
> at org.codehaus.janino.UnitCompiler.access$6600(UnitCompiler.java:215)
> at
> org.codehaus.janino.UnitCompiler$14.visitBinaryOperation(UnitCompiler.java:3957)
> at
> org.codehaus.janino.UnitCompiler$14.visitBinaryOperation(UnitCompiler.java:3935)
> at org.codehaus.janino.Java$BinaryOperation.accept(Java.java:4864)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
