[
https://issues.apache.org/jira/browse/CALCITE-7422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenzhuang Zhu updated CALCITE-7422:
-----------------------------------
Description:
This subtask includes the following items (controlled by isLargePlanMode):
1.HepPlanner supports graph reuse across different HepPrograms.
Reason: Avoid calling setRoot multiple times when the user has multiple
optimization phases. (HepSubProgram cannot be used because some global state
must be changed between different HepPrograms.)
2.Fast graph GC for the subtree of a discarded vertex (discardedVertex).
Reason: Avoid performing a full-graph traversal during garbage collection. Full
graph GC is On. In the worst case, every transform in TOP_DOWN/BOTTOM_UP mode
can trigger a full graph GC.
3.Use a large-plan-friendly graph iterator.
Reason: DepthFirstIterator builds a list containing all nodes in the graph,
which costs On memory and additional time to build the list. This is not
suitable for large plans with 10K+ nodes. BreadthFirstIterator does not build
such a list, and after firedRulesCache is enabled, BreadthFirstIterator and
DepthFirstIterator fire the same number of rules per node. Therefore, for
large-plan mode, using BreadthFirstIterator is generally better than
DepthFirstIterator.
4.Recursivelly contract vertices.
Reason: Large plans provide more opportunities for recursive contraction.
Notes: A very deep plan may cause a stackoverflow when build the graph
(recursive addVertex), but this is not the bottomneck. Compiler or other
modules may need more stacks than optimizer, so we assume the -Xss is large
enough.
was:
This subtask includes the following items (controlled by isLargePlanMode):
1.HepPlanner supports graph reuse across different HepPrograms.
Reason: Avoid calling setRoot multiple times when the user has multiple
optimization phases. (HepSubProgram cannot be used because some global state
must be changed between different HepPrograms.)
2.Fast graph GC for the subtree of a discarded vertex (discardedVertex).
Reason: Avoid performing a full-graph traversal during garbage collection. Full
graph GC is O(n). In the worst case, every transform in TOP_DOWN/BOTTOM_UP mode
can trigger a full graph GC.
3.Use a large-plan-friendly graph iterator.
Reason: DepthFirstIterator builds a list containing all nodes in the graph,
which costs O(n) memory and additional time to build the list. This is not
suitable for large plans with 10K+ nodes. BreadthFirstIterator does not build
such a list, and after firedRulesCache is enabled, BreadthFirstIterator and
DepthFirstIterator fire the same number of rules per node. Therefore, for
large-plan mode, using BreadthFirstIterator is generally better than
DepthFirstIterator.
4.Recursivelly contract vertices.
Reason: Large plans provide more opportunities for recursive contraction.
Notes: A very deep plan may cause a stackoverflow when build the graph
(recursive addVertex), but this is not the bottomneck. Compiler or other
modules may need more stacks than optimizer, so we assume the -Xss is large
enough.
> Support large plan optimizaion mode for HepPlanner
> --------------------------------------------------
>
> Key: CALCITE-7422
> URL: https://issues.apache.org/jira/browse/CALCITE-7422
> Project: Calcite
> Issue Type: Sub-task
> Reporter: Wenzhuang Zhu
> Priority: Major
>
> This subtask includes the following items (controlled by isLargePlanMode):
> 1.HepPlanner supports graph reuse across different HepPrograms.
> Reason: Avoid calling setRoot multiple times when the user has multiple
> optimization phases. (HepSubProgram cannot be used because some global state
> must be changed between different HepPrograms.)
> 2.Fast graph GC for the subtree of a discarded vertex (discardedVertex).
> Reason: Avoid performing a full-graph traversal during garbage collection.
> Full graph GC is On. In the worst case, every transform in TOP_DOWN/BOTTOM_UP
> mode can trigger a full graph GC.
> 3.Use a large-plan-friendly graph iterator.
> Reason: DepthFirstIterator builds a list containing all nodes in the graph,
> which costs On memory and additional time to build the list. This is not
> suitable for large plans with 10K+ nodes. BreadthFirstIterator does not build
> such a list, and after firedRulesCache is enabled, BreadthFirstIterator and
> DepthFirstIterator fire the same number of rules per node. Therefore, for
> large-plan mode, using BreadthFirstIterator is generally better than
> DepthFirstIterator.
> 4.Recursivelly contract vertices.
> Reason: Large plans provide more opportunities for recursive contraction.
> Notes: A very deep plan may cause a stackoverflow when build the graph
> (recursive addVertex), but this is not the bottomneck. Compiler or other
> modules may need more stacks than optimizer, so we assume the -Xss is large
> enough.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
