andygrove commented on PR #15654: URL: https://github.com/apache/datafusion/pull/15654#issuecomment-2794652944
> @andygrove any chance you could check Comet's performance with this alternative implementation: https://github.com/ashdnazg/datafusion/tree/pull-batch-2 ? It attempts to remove the spawn overhead and to make buffering more efficient. I don't think Comet testing is going to help with this. Here are timings for q4 with this PR and the alternate for 5 runs of q4. In both cases there are tasks failing and restarting due to lack of memory. ## This PR ``` 14.834558725357056, 11.173914194107056, 11.313692808151245, 10.791407823562622, 11.371635913848877 ``` ## Alternate ``` 13.932721853256226, 12.08954644203186, 11.981270551681519, 12.231445550918579, 10.979195594787598 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
