Hi all, Wanted to revive this question - today, we saw some CI blockages [1] across Iceberg projects due to MinIO images being removed from Docker Hub, which adds another data point to moving away from MinIO.
Worth noting that have been similar threads to this one in the past [2], [3] - also, Polaris now uses RustFS for its quickstart, with RustFS and Floci for integration tests (separating those concerns) [4], and PyIceberg has moved to RustFS [5]. Wondering what the community now thinks about moving away from MinIO for quickstarts and integration tests? [1] Iceberg Java: https://github.com/apache/iceberg/pull/18071 [2] https://lists.apache.org/thread/vnw9jonmfcsz6bwojhfch1nmywyl50h3 [3] https://lists.apache.org/thread/prmpbfy6v20spsmts19hs8gbkkxo16vv [4] https://lists.apache.org/thread/8o31ly7cd8ov70opjbtg630qlhrfl5yh [5] https://github.com/apache/iceberg-python/pull/3928 Thanks, Sreesh On Fri, Mar 20, 2026 at 5:38 AM Chris Lu <[email protected]> wrote: > Hi everyone, > > I’ve been following the recent discussions around S3-compatible backends > for local testing and CI, especially the challenges around S3 compatibility > and credential-related testing. > > I’d like to propose an approach to help address: > https://github.com/apache/iceberg/issues/14638 > > I’ve opened an initial PR that switches the test backend from MinIO to > SeaweedFS: > https://github.com/apache/iceberg/pull/15577 > > The goal is to improve test reliability and better support scenarios that > are currently difficult to validate. > > SeaweedFS is an S3-compatible storage system with support for IAM-style > access control and STS flows. One reason I explored this direction is that > it may help with testing credential-related scenarios (e.g., temporary > credentials / vended credentials), which have been discussed recently. > > There are also a couple of areas that could potentially expand test > coverage over time: > > - Table-oriented bucket layouts could make it easier to exercise > Iceberg-specific storage patterns that are currently hard to simulate in CI > > - Support for IAM and STS flows could allow better validation of > credential-related behaviors used by some catalog integrations > > The current PR is intentionally minimal and focuses only on getting the > existing tests running. If this direction is useful, I’m planning to follow > up with additional integration tests (e.g., around table bucket behaviors). > > I want to emphasize that this is mainly to address #14638 and improve > coverage. If replacing the backend is too disruptive, I’m also happy to > explore alternatives such as running multiple backends or limiting usage to > specific scenarios. > > I’d really appreciate feedback on: > > - Whether switching the backend is a reasonable approach for #14638 > - Any concerns around CI stability or maintenance > - Whether running multiple backends would be preferable > - Any specific scenarios we should prioritize testing > > For context, I’m the author of SeaweedFS. SeaweedFS is open source under > the Apache 2.0 license. > > Thanks for your time and guidance! > > Chris >
