Hi All, Xuanwo, I recall the reasoning against HadoopCatalog was the other way around: even though it is safe to use on HDFS, it is unsafe on object storage. I believe that this gap of functionalities of object stores seems to go away, so for me HadoopCatalog would even make more sense now than before. The name might not be straightforward as it's not just for Hadoop.
Regards, Gabor On Wed, Nov 27, 2024 at 9:02 AM Xuanwo <xua...@apache.org> wrote: > Hi > > I believe we still need to deprecate HadoopCatalog since the operation is > still not safe on Hadoop. As raised by Jack Ye before, I suggest we > consider having a StorageCatalog or ObjectStorageCatalog that can only be > used with storage services supporting conditional writes. That would be a > good approach. > > On Wed, Nov 27, 2024, at 15:47, Nikhil Benesch wrote: > > Makes sense! I'd be eager to chat more about this but I'm afraid I won't > be at > > re:Invent. Maybe we plan to circle back after re:Invent, once we see > what AWS > > announces? > > > > On Tue, Nov 26, 2024 at 2:58 PM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > >> > >> Hi Nikhil > >> > >> Thanks for your message, very interesting. > >> > >> I think it would be great to involve the Polaris project here as well, > >> as a REST Catalog implementation. > >> The Polaris community is discussing storage/backend right now, so it > >> would be the perfect timing to consider leveraging S3 conditional > >> writes (as a plugin for instance first). > >> > >> I would be happy to connect and know more about your perspective about > that. > >> > >> Thanks, > >> Regards > >> JB > >> > >> PS: I will be at AWS re:Invent next week, so maybe we can connect there. > >> > >> On Tue, Nov 26, 2024 at 6:35 PM Nikhil Benesch < > nikhil.bene...@gmail.com> wrote: > >> > > >> > Hi all, > >> > > >> > With Amazon S3 announcing support for the If-Match header yesterday > [0], all the > >> > major object store implementations now support a compare-and-swap > operation. > >> > > >> > As far as I can tell, this opens up the possibility of storing Iceberg > >> > catalogs directly on object storage, without the need for a separate > metastore, > >> > and without violating any of Iceberg's ACID guarantees. > >> > > >> > It seems the immediate next step is to build an independent Java or > REST catalog > >> > backend to prove this concept out. Long term, though, the ideal would > be to > >> > have such a catalog backend be a first class citizen in the Iceberg > project. > >> > > >> > Is anyone else in the Iceberg community barking up this tree? I'm a > long term > >> > Iceberg enthusiast, but new to the community. I'd very much > appreciate any > >> > pointers to current or past discussions on the topic. So far all I've > been > >> > able to turn up is some light chatter from myself and others on > Bluesky and > >> > Hacker News ([1][2][3]). > >> > > >> > Cheers, > >> > Nikhil > >> > > >> > [0]: > https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-functionality-conditional-writes/ > >> > [1]: https://bsky.app/profile/benesch.bsky.social/post/3lauesxg3ic2c > >> > [2]: > https://bsky.app/profile/eatonphil.bsky.social/post/3lbskq3jwk22e > >> > [3]: https://news.ycombinator.com/item?id=42240370 > > -- > Xuanwo > > https://xuanwo.io/ >
