There's a PR up from amazon to add this to the s3a connector https://github.com/apache/hadoop/pull/7011
targeting a 3.4.2 release early next year, though they've not updated the PR as requested yet. 1. It doesn't give you the same semantics as posix create-no-overwrite call -you only get the error after the upload, not in create(). You should only be writing a very small file as part of your commit protocol, not something big. 2. Most( all) third party stores *do not* support this -but they don't fail with any errors. The only way to probe for the behaviour is actually to attempt to do it and see if overwrites are rejected On Tue, 26 Nov 2024 at 17:36, Nikhil Benesch <nikhil.bene...@gmail.com> wrote: > Hi all, > > With Amazon S3 announcing support for the If-Match header yesterday [0], > all the > major object store implementations now support a compare-and-swap > operation. > > As far as I can tell, this opens up the possibility of storing Iceberg > catalogs directly on object storage, without the need for a separate > metastore, > and without violating any of Iceberg's ACID guarantees. > > It seems the immediate next step is to build an independent Java or REST > catalog > backend to prove this concept out. Long term, though, the ideal would be to > have such a catalog backend be a first class citizen in the Iceberg > project. > > Is anyone else in the Iceberg community barking up this tree? I'm a long > term > Iceberg enthusiast, but new to the community. I'd very much appreciate any > pointers to current or past discussions on the topic. So far all I've been > able to turn up is some light chatter from myself and others on Bluesky and > Hacker News ([1][2][3]). > > Cheers, > Nikhil > > [0]: > https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-functionality-conditional-writes/ > [1]: https://bsky.app/profile/benesch.bsky.social/post/3lauesxg3ic2c > [2]: https://bsky.app/profile/eatonphil.bsky.social/post/3lbskq3jwk22e > [3]: https://news.ycombinator.com/item?id=42240370 >
