mbutrovich commented on issue #1906: URL: https://github.com/apache/datafusion-comet/issues/1906#issuecomment-2985599126
So the big challenge here seems to be mapping Comet to Spark's execution. Each generated Comet plan samples from its input stream, which itself is only a single partition of the data. We'll have to figure out a way to orchestrate sampling across all partitions (i.e., across workers) and then communicate that to the workers so they have consistent partition bounds. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
