askalt commented on PR #19462:
URL: https://github.com/apache/datafusion/pull/19462#issuecomment-3696773275
Added a benchmark to show the difference in performance. It compares two
approaches that allow to re-use a plan (we do not consider plan re-building as
it spends too much on optimizations).
Used plan is relatively small:
```
ProjectionExec
FilterExec
AggregateExec: mode=Final
CoalescePartitionsExec
AggregateExec: mode=Partial
RepartitionExec
FilterExec
TestMemoryExec
```
The plan is generated by default planner for the query:
```sql
SELECT aggr1(col1) as aggr1, aggr2(col2) as aggr2 FROM t
WHERE p1
HAVING p2
```
1) Re-set plan state on each use:
```rust
#[cfg(not(feature = "stateless_plan"))]
fn reset_plan_states(plan: Arc<dyn ExecutionPlan>) -> Arc<dyn ExecutionPlan>
{
plan.transform_up(|plan| {
let new_plan = Arc::clone(&plan).reset_state()?;
Ok(Transformed::yes(new_plan))
})
.unwrap()
.data
}
```
This requires to re-compute plans properties, as for many plans the method
`with_new_children` use `Self::try_new` to build a new plan. For example,
`FilterExec`, `AggregateExec`, `ProjectionExec`. So, on each iteration we
re-build the whole tree.
On my machine:
```sh
$ cargo bench --bench plan_reuse
...
time: [4.4009 ms 4.4068 ms 4.4130 ms]
```
2) Use `stateless_plan` feature (introduced by the patch) and instead of
re-build the whole plan on each iteration, we build
a state tree.
It gives:
```sh
$ cargo bench --features stateless_plan --bench plan_reuse
time: [72.424 µs 72.607 µs 72.795 µs]
```
Which does not depend on `with_new_children` logic and `Self::try_new`
analysis because they are not called at all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
