Shekharrajak commented on issue #3537: URL: https://github.com/apache/datafusion-comet/issues/3537#issuecomment-3924922355
1. Are we planning to have some kind of threshold for the benchmark result ? 2. comet should be performing better thatn spark JVM execution at least 1.1x improvement or something ? 3. We can github CI workflow defined to run the benchmark - I have worked on similar workflow in kubeflow SparkClient SDK https://github.com/kubeflow/sdk/pull/225/changes#diff-7d4dcd25ff96ca99340aa114debcd19250e2c0238c7c5597361bdccaa6fdb474R81 We can install kind , create create k8s cluster `brew install kind kubectl helm` and have spark operator CRD available. benchmark docker image build to run the spark job in different env. This could be little tricky and need to check resources usage when we run different unit tests, but let me try. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
