adriangb commented on issue #25148: URL: https://github.com/apache/datafusion/issues/25148#issuecomment-5635652061
> Each is a separate job compiling a heavily overlapping dependency graph from its own cache. They are all `Required`, so they also each add merge-queue start latency, which is the symptom in [#25088](https://github.com/apache/datafusion/issues/25088). Folding them into one job that loops over the crates, or into a matrix sharing a cache, looks like the largest single saving available inside this repo that changes no coverage. Actions now support parallel steps - this seems like a good opportunity to build the dependency graph once and then run the checks on each leaf crate in parallel? > right, so we can remove ` vars.USE_RUNS_ON == 'true'` completely, and then prs opened from forks will go through runs-on. but the main problem for that is our AWS credits @blaginin _who's_ AWS credits? AFAIK runs-on uses an ASF AWS account right now. Is this a much larger pool? I don't know what the spend or credits situation is on AWS. If that's a much larger pool of compute minutes, it could be a good solution to just move everything there. If we're just moving the problem around it's not a good solution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
