[
https://issues.apache.org/jira/browse/HIVE-12864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15106500#comment-15106500
]
Jesus Camacho Rodriguez commented on HIVE-12864:
------------------------------------------------
[~pxiong], thanks for checking on this.
The original five methods are tree traversals implemented recursively.
The new ones in the patch are similar to the original ones, but I rewrote each
of them to be iterative -using stacks-. This avoid the StackOverflowError.
Concretely:
* setUnknownTokenBoundaries(): post-order
* dump(StringBuilder sb): pre- and post-order
* toStringTree(ASTNode rootNode): pre- and post-order
* processPositionAlias(ASTNode ast): pre-order
* findSubQueries(ASTNode node, List<ASTNode> subQueries): pre-order
These algorithms are part of the parsing logic, so you can pick any query in
the q files (e.g. lineage2.q, lineage3.q, subquery_in.q) to walk through each
of the algorithms.
> StackOverflowError parsing queries with very large predicates
> -------------------------------------------------------------
>
> Key: HIVE-12864
> URL: https://issues.apache.org/jira/browse/HIVE-12864
> Project: Hive
> Issue Type: Bug
> Components: Parser
> Affects Versions: 2.0.0, 2.1.0
> Reporter: Jesus Camacho Rodriguez
> Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-12864.01.patch, HIVE-12864.patch
>
>
> We have seen that queries with very large predicates might fail with the
> following stacktrace:
> {noformat}
> 016-01-12 05:47:36,516|beaver.machine|INFO|552|5072|Thread-22|Exception in
> thread "main" java.lang.StackOverflowError
> 2016-01-12 05:47:36,517|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:145)
> 2016-01-12 05:47:36,517|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,517|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,517|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,517|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,519|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,520|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,520|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,520|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,520|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,520|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,520|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,520|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,522|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,522|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,522|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,522|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,522|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,522|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,522|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,522|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,523|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,523|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,523|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,523|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,523|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,523|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,525|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,525|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,525|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,525|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,525|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,525|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,526|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,526|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,526|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,526|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,526|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,526|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,526|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,526|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,526|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,526|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,526|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,526|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,526|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,526|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,526|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:36,634|beaver.machine|INFO|552|5072|Thread-22|at
> org.antlr.runtime.tree.CommonTree.setUnknownTokenBoundaries(CommonTree.java:146)
> 2016-01-12 05:47:37,582|main|INFO|552|4568|MainThread|TEST "test_WideQuery"
> FAILED in 10.95 seconds
> {noformat}
> The problem could be solved by reimplementing some of the parsing methods so
> they are iterative instead of recursive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
