robtandy opened a new pull request, #86: URL: https://github.com/apache/datafusion-ray/pull/86
This PR is long but it does not affect the core functionality of DataFusion for Ray, and does not differ from `0.1.0rc1` which has been extensively used by me in benchmarking from `test.pypi`. This PR, when merged should be tagged `0.1.0rc2` and I can push the artifacts created by CI to `test.pypi`. Upon successful tests, these same artifacts can be used for the actual `0.1.0` release. This PR squashed merged the work that represented building out a repeatable k8s benchmarking set up. It is far from perfect but it has enabled significant benchmarking that has resulted in intuition around execution, latency, and price trade offs that I think will help us communicate about the project. The scripts and instructions are written to start from a fresh ubuntu 24.04 ami on amazon, and will work on `arm` or `x86_64` architectures. Summary of changes - k8s benchmarking scripts and instructions - the bulk of the PR - format python code - remove the last mentions of `worker` and replace with `processor` where appropriate to indicate `DFRayProcessors` - adding benchmark results in `doc/benchmarks` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
