robtandy commented on issue #46: URL: https://github.com/apache/datafusion-ray/issues/46#issuecomment-2563983395
Would it be possible to use a ray generator for the shuffle writer and reader execution plans? Rather than serializing all of the data to disk in the writer, and reading it in the reader, we yield record batches in the Task. Then ray sends those in the in memory object store (spilling if required). Then we can pipeline the execution of the writer and readers so that stages need not wait for completion of dependent stages to start processing. Would this work? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
