Hi Chris, I recall needing to run the S3A tests a while back when I was working on a change related to DistCp. At the time, I considered using GitHub Actions to run those tests using AWS credentials provided by the ASF. As far as I know, such credentials are available [1], but I never pursued it further since I didn’t revisit that area afterward.
You might want to check with the INFRA team to see if they also provide credentials for GCS—that could be a viable option to explore. One nice aspect of GitHub Actions is that they can also be run on individual forks. Contributors can configure their own AWS credentials as secrets in their forks of the Hadoop repo and run the tests there. This would help avoid consuming ASF resources directly. If ASF credentials aren’t available, a link to the successful run on their fork can also be included as a comment on the PR to confirm the test results. This was just an early idea I had back then—feel free to explore it further if it seems useful. -Ayush [1] https://issues.apache.org/jira/browse/INFRA-24353 On Thu, 24 Jul 2025 at 04:30, Chris Nauroth <cnaur...@apache.org> wrote: > > Hello everyone, > > For years, we've relied on specific contributors to run and verify the > integration tests for object store integrations like S3A, because the tests > require credentials for specific cloud providers. I'd like to explore if we > have any path forward today to bringing those tests into the pre-submit > automation. If successful, I'd like to apply that strategy to the GCS > integration tests, which are part of HADOOP-19343. > > To make this work, we'd need to either 1) run tests in a VM hosted in the > cloud provider, where credentials are vended natively from an adjacent > metadata server, or 2) export credentials so that the tests can run in any > VM outside the cloud provider (and be really, really, really careful to > secure the access to those exported credentials). > > Has anyone else already explored this recently? If not, I was thinking of > filing an INFRA ticket to discuss if they already have established patterns > for this. This is potentially relevant to other projects. (It was the code > review for FLINK-37247 that prompted me to start this conversation.) I > think it makes sense to solve it in Hadoop first and then extend it to > other projects. > > Chris Nauroth --------------------------------------------------------------------- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
