Kontinuation commented on PR #15654: URL: https://github.com/apache/datafusion/pull/15654#issuecomment-2790105797
If I understand this PR correctly, the `SpillReaderStream` in this PR will read the next batch only when the stream is polled, so the latency of polling a batch is the time spent reading a batch plus some scheduling overhead. The original approach buffers at most 2 batches in each stream. If the batch is already buffered, the latency of polling a batch is the time spent consuming a batch from the mpsc channel, thus hiding the latency of reading files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
