Hi everyone, I’ve been following the recent discussions around S3-compatible backends for local testing and CI, especially the challenges around S3 compatibility and credential-related testing.
I’d like to propose an approach to help address: https://github.com/apache/iceberg/issues/14638 I’ve opened an initial PR that switches the test backend from MinIO to SeaweedFS: https://github.com/apache/iceberg/pull/15577 The goal is to improve test reliability and better support scenarios that are currently difficult to validate. SeaweedFS is an S3-compatible storage system with support for IAM-style access control and STS flows. One reason I explored this direction is that it may help with testing credential-related scenarios (e.g., temporary credentials / vended credentials), which have been discussed recently. There are also a couple of areas that could potentially expand test coverage over time: - Table-oriented bucket layouts could make it easier to exercise Iceberg-specific storage patterns that are currently hard to simulate in CI - Support for IAM and STS flows could allow better validation of credential-related behaviors used by some catalog integrations The current PR is intentionally minimal and focuses only on getting the existing tests running. If this direction is useful, I’m planning to follow up with additional integration tests (e.g., around table bucket behaviors). I want to emphasize that this is mainly to address #14638 and improve coverage. If replacing the backend is too disruptive, I’m also happy to explore alternatives such as running multiple backends or limiting usage to specific scenarios. I’d really appreciate feedback on: - Whether switching the backend is a reasonable approach for #14638 - Any concerns around CI stability or maintenance - Whether running multiple backends would be preferable - Any specific scenarios we should prioritize testing For context, I’m the author of SeaweedFS. SeaweedFS is open source under the Apache 2.0 license. Thanks for your time and guidance! Chris
