alamb commented on PR #20481: URL: https://github.com/apache/datafusion/pull/20481#issuecomment-3958979921
> I was reading the paper more in depth in recent days and concluded that the essence boils down to those things: I agree > thread per core (in our case already true when setting target_partitions to cpu cores) Yes -- exactly -- and Tokio handles the scheudling > morsel = a large enough unit of work, large enough for communication overhead to be low, small enough to introduce enough parallelism In my mind, most of the plans use `RecordBatch` as the "Morsels" so a Stream<RecordBatch>` is very much a Morsel as described in the paper > work stealing / a queue with spreading "morsels" on the threads Indeed -- and in fact tokio has a [work stealing scheduler ](https://tokio.rs/blog/2019-10-scheduler) In my mind, what this PR does is to break up Parquet files into smaller chunks that are read (but the granularity is bigger than a RecordBatch / target batch size, it is a RowGroup) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
