andygrove opened a new issue, #2842:
URL: https://github.com/apache/datafusion-ballista/issues/2842
Follow-up from #2721. Let's agree on the design here before we propose it
upstream.
When a stage is split between tasks, Ballista restricts each task's plan to
that task's partitions. `ExecutionPlan` has no method for "restrict yourself to
these partitions", so Ballista keeps a hand-maintained list
(`select_output_partitions`, `as_partition_sliceable`), plus the
`PartitionSliceable` trait and, since #2721, a `PartitionSliceableResolver` on
`SchedulerConfig` for nodes defined outside Ballista. It also can't tell
whether an interior node it doesn't know holds data per partition, so only
unknown leaves are guarded.
A defaulted method on `ExecutionPlan` would let each node answer for itself
and retire the list, the trait and the resolver. Something like:
```rust
/// Rebuild over `children` (already restricted), keeping only `partitions`,
in that order.
/// `None` means the node cannot be restricted.
fn restrict_to_partitions(
self: Arc<Self>,
children: Vec<Arc<dyn ExecutionPlan>>,
partitions: &[usize],
) -> Option<Result<Arc<dyn ExecutionPlan>>> {
None
}
```
Questions to settle before opening a DataFusion issue:
- the default: `None` (unknown, as today) or something stateless operators
can opt into cheaply, so an unknown interior node can be told apart from a
stateless one;
- whether DataFusion has a use for it on its own (e.g. partition pruning)
that makes the case upstream;
- how `DataSource` would take part, so `DataSourceExec` can delegate to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
