parthchandra commented on PR #1175: URL: https://github.com/apache/datafusion-comet/pull/1175#issuecomment-2549700632
> > Just for clarification - what's the relation between shuffle write time, encoding and compression total time, and native shuffle total time? I would think `shuffle write time + encoding and compression total time = native shuffle total time`, but that does not seem to be the case? > > There is also evaluating the partition expressions (typically very fast if they are just column references) and then the time to actually split the batches into partitions. From the above screenshot, `shuffle write time + encoding and compression total time = 18.4s` and `native shuffle total time=28.8s`, so there is a difference of `10.4s` which is substantial. Wondering if we are missing something. Nonetheless, the PR certainly improves on the current. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
