crepererum commented on issue #8777: URL: https://github.com/apache/datafusion/issues/8777#issuecomment-3023920909
> Implementation wise, it could be done with a separate blocking deque (VecDeque?) for each consumer. > Incoming data is added to all queues (shared Arcs, no data copying). > Consumers pulls from the queue. When it's empty, it waits for new data or end of processing signal. I think this problem is actually very similar to `RepatitionExec` and the distributor channels in there, except that `RepartitionExec` sends 1 row to 1 output, the construct required here would just send them to ALL outputs. And we'll likely want a similar buffer limiter, so that the CTE output isn't buffered unlimited when no consumer polls the data (at the same time, multiple consumer may poll at the same time). So I would propose to use the same underlying "distributor channel" primitive (or whatever we replace that with). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
