jayzhan211 commented on PR #24353: URL: https://github.com/apache/datafusion/pull/24353#issuecomment-5380741644
> > Thank you @goutamadwant -- i think this makes sense, but we should get alignment on what the intention of `Unbounded` input is and make sure @2010YOUY01 agrees / understands the rationale before we merge this > > Yes, I want to know more about application level motivation, and why the existing behavior won't satisfy the requirement, and next see what should we do. > > Another concern is if we start to incorporate such no buffering behavior, extra implementation complexity will be added to all existing operators, for this `FilterExec` it's perfectly fine since it's simpler, but it would be very hard to do for other operators. > > Is it possible to split it to a `StreamingFilterStream`, or even maintain that variant in downstream. If we agree on supporting more "streaming" execution in DataFusion, I think we should extend that support across all existing operators. +1 from me on the overall direction. How to design this while keeping maintenance complexity low could be a separate discussion — maybe in [#24265](https://github.com/apache/datafusion/issues/24265) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
