Hi I believe we still need to deprecate HadoopCatalog since the operation is still not safe on Hadoop. As raised by Jack Ye before, I suggest we consider having a StorageCatalog or ObjectStorageCatalog that can only be used with storage services supporting conditional writes. That would be a good approach.
On Wed, Nov 27, 2024, at 15:47, Nikhil Benesch wrote: > Makes sense! I'd be eager to chat more about this but I'm afraid I won't be at > re:Invent. Maybe we plan to circle back after re:Invent, once we see what AWS > announces? > > On Tue, Nov 26, 2024 at 2:58 PM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: >> >> Hi Nikhil >> >> Thanks for your message, very interesting. >> >> I think it would be great to involve the Polaris project here as well, >> as a REST Catalog implementation. >> The Polaris community is discussing storage/backend right now, so it >> would be the perfect timing to consider leveraging S3 conditional >> writes (as a plugin for instance first). >> >> I would be happy to connect and know more about your perspective about that. >> >> Thanks, >> Regards >> JB >> >> PS: I will be at AWS re:Invent next week, so maybe we can connect there. >> >> On Tue, Nov 26, 2024 at 6:35 PM Nikhil Benesch <nikhil.bene...@gmail.com> >> wrote: >> > >> > Hi all, >> > >> > With Amazon S3 announcing support for the If-Match header yesterday [0], >> > all the >> > major object store implementations now support a compare-and-swap >> > operation. >> > >> > As far as I can tell, this opens up the possibility of storing Iceberg >> > catalogs directly on object storage, without the need for a separate >> > metastore, >> > and without violating any of Iceberg's ACID guarantees. >> > >> > It seems the immediate next step is to build an independent Java or REST >> > catalog >> > backend to prove this concept out. Long term, though, the ideal would be to >> > have such a catalog backend be a first class citizen in the Iceberg >> > project. >> > >> > Is anyone else in the Iceberg community barking up this tree? I'm a long >> > term >> > Iceberg enthusiast, but new to the community. I'd very much appreciate any >> > pointers to current or past discussions on the topic. So far all I've been >> > able to turn up is some light chatter from myself and others on Bluesky and >> > Hacker News ([1][2][3]). >> > >> > Cheers, >> > Nikhil >> > >> > [0]: >> > https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-functionality-conditional-writes/ >> > [1]: https://bsky.app/profile/benesch.bsky.social/post/3lauesxg3ic2c >> > [2]: https://bsky.app/profile/eatonphil.bsky.social/post/3lbskq3jwk22e >> > [3]: https://news.ycombinator.com/item?id=42240370 -- Xuanwo https://xuanwo.io/
