pepijnve commented on code in PR #16196:
URL: https://github.com/apache/datafusion/pull/16196#discussion_r2131883855
##########
datafusion/physical-optimizer/src/wrap_leaves_cancellation.rs:
##########
@@ -76,7 +77,8 @@ impl WrapLeaves {
plan: Arc<dyn ExecutionPlan>,
yield_frequency: usize,
) -> Result<Transformed<Arc<dyn ExecutionPlan>>> {
- let is_pipeline_breaker = plan.properties().emission_type ==
EmissionType::Final;
+ // todo this is a bit of a hack, we should probably have a more
explicit way to handle
+ let is_pipeline_breaker = plan.properties().emission_type ==
EmissionType::Final|| plan.as_any().is::<FilterExec>();
Review Comment:
I was curious to see how this case could be handled. The consequence is that
in many situations the yield logic will be injected unnecessarily causing the
root stream to emit pending even though ready's are arriving at a steady pace.
This is a bit at odds with the desire to keep performance overhead minimal.
I don't think there's a way to solve this external to `FilterExecStream`
since you're trying to count the number of times it consecutively discarded
full batches, not the number of times it polled its input.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
