Interesting. We already have file system tables [1] in Iceberg (HadoopCatalog implements this spec). We deprecated this recently and we don't have to deprecate it if object stores support atomic operations like this.
[1] https://iceberg.apache.org/spec/#file-system-tables - Ajantha On Wed, Nov 27, 2024 at 2:53 AM Nikhil Benesch <nikhil.bene...@gmail.com> wrote: > Ah, fascinating. Thanks very much for the pointer. > > Here's the thread introducing the proposal [0], for anyone else curious. > > [0]: https://lists.apache.org/thread/kh4n98w4z22sc8h2vot4q8n44vdtnltg > > On Tue, Nov 26, 2024 at 3:27 PM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > > > Hi Vignesh > > > > Thanks for the reminder, I remember we quickly discussed this during a > > community meeting. > > > > I will take a new look at the doc. > > > > Regards > > JB > > > > On Tue, Nov 26, 2024 at 9:19 PM Vignesh <vignesh.v...@gmail.com> wrote: > > > > > > Hi, > > > There was a proposal along the same lines, for the read portion few > weeks back by Ashvin. > > > > > > > https://docs.google.com/document/d/1yzLXSOtzBXyaWHfeVsWsMu4xmOH8rV6QyM5ZAnJZjMQ/edit?usp=drivesdk > > > > > > Thanks, > > > Vignesh. > > > > > > > > > On Tue, Nov 26, 2024, 11:59 AM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > > >> > > >> Hi Nikhil > > >> > > >> Thanks for your message, very interesting. > > >> > > >> I think it would be great to involve the Polaris project here as well, > > >> as a REST Catalog implementation. > > >> The Polaris community is discussing storage/backend right now, so it > > >> would be the perfect timing to consider leveraging S3 conditional > > >> writes (as a plugin for instance first). > > >> > > >> I would be happy to connect and know more about your perspective > about that. > > >> > > >> Thanks, > > >> Regards > > >> JB > > >> > > >> PS: I will be at AWS re:Invent next week, so maybe we can connect > there. > > >> > > >> On Tue, Nov 26, 2024 at 6:35 PM Nikhil Benesch < > nikhil.bene...@gmail.com> wrote: > > >> > > > >> > Hi all, > > >> > > > >> > With Amazon S3 announcing support for the If-Match header yesterday > [0], all the > > >> > major object store implementations now support a compare-and-swap > operation. > > >> > > > >> > As far as I can tell, this opens up the possibility of storing > Iceberg > > >> > catalogs directly on object storage, without the need for a > separate metastore, > > >> > and without violating any of Iceberg's ACID guarantees. > > >> > > > >> > It seems the immediate next step is to build an independent Java or > REST catalog > > >> > backend to prove this concept out. Long term, though, the ideal > would be to > > >> > have such a catalog backend be a first class citizen in the Iceberg > project. > > >> > > > >> > Is anyone else in the Iceberg community barking up this tree? I'm a > long term > > >> > Iceberg enthusiast, but new to the community. I'd very much > appreciate any > > >> > pointers to current or past discussions on the topic. So far all > I've been > > >> > able to turn up is some light chatter from myself and others on > Bluesky and > > >> > Hacker News ([1][2][3]). > > >> > > > >> > Cheers, > > >> > Nikhil > > >> > > > >> > [0]: > https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-functionality-conditional-writes/ > > >> > [1]: > https://bsky.app/profile/benesch.bsky.social/post/3lauesxg3ic2c > > >> > [2]: > https://bsky.app/profile/eatonphil.bsky.social/post/3lbskq3jwk22e > > >> > [3]: https://news.ycombinator.com/item?id=42240370 >
