Indeed, I got pointed at that feature on Bluesky earlier today [0]. I dredged up the mailing list discussion that occurred around its deprecation, and this exact point actually came up. There was some concern from Ryan that the complexity of keeping the file system tables around just wasn't worth it [1].
Assuming the deprecation of file system tables is generally desirable, I think there is probably a happy outcome here where a full fledged catalog implementation can be built directly on top of S3. [0]: https://bsky.app/profile/jankaul.bsky.social/post/3lbuuazta2k2c [1]: https://lists.apache.org/thread/oyo28xlwlsj5rlvq73tm3z780f2poddg On Wed, Nov 27, 2024 at 2:05 AM Ajantha Bhat <ajanthab...@gmail.com> wrote: > > Interesting. > > We already have file system tables [1] in Iceberg (HadoopCatalog implements > this spec). > We deprecated this recently and we don't have to deprecate it if object > stores support atomic operations like this. > > [1] https://iceberg.apache.org/spec/#file-system-tables > > - Ajantha > > On Wed, Nov 27, 2024 at 2:53 AM Nikhil Benesch <nikhil.bene...@gmail.com> > wrote: >> >> Ah, fascinating. Thanks very much for the pointer. >> >> Here's the thread introducing the proposal [0], for anyone else curious. >> >> [0]: https://lists.apache.org/thread/kh4n98w4z22sc8h2vot4q8n44vdtnltg >> >> On Tue, Nov 26, 2024 at 3:27 PM Jean-Baptiste Onofré <j...@nanthrax.net> >> wrote: >> > >> > Hi Vignesh >> > >> > Thanks for the reminder, I remember we quickly discussed this during a >> > community meeting. >> > >> > I will take a new look at the doc. >> > >> > Regards >> > JB >> > >> > On Tue, Nov 26, 2024 at 9:19 PM Vignesh <vignesh.v...@gmail.com> wrote: >> > > >> > > Hi, >> > > There was a proposal along the same lines, for the read portion few >> > > weeks back by Ashvin. >> > > >> > > https://docs.google.com/document/d/1yzLXSOtzBXyaWHfeVsWsMu4xmOH8rV6QyM5ZAnJZjMQ/edit?usp=drivesdk >> > > >> > > Thanks, >> > > Vignesh. >> > > >> > > >> > > On Tue, Nov 26, 2024, 11:59 AM Jean-Baptiste Onofré <j...@nanthrax.net> >> > > wrote: >> > >> >> > >> Hi Nikhil >> > >> >> > >> Thanks for your message, very interesting. >> > >> >> > >> I think it would be great to involve the Polaris project here as well, >> > >> as a REST Catalog implementation. >> > >> The Polaris community is discussing storage/backend right now, so it >> > >> would be the perfect timing to consider leveraging S3 conditional >> > >> writes (as a plugin for instance first). >> > >> >> > >> I would be happy to connect and know more about your perspective about >> > >> that. >> > >> >> > >> Thanks, >> > >> Regards >> > >> JB >> > >> >> > >> PS: I will be at AWS re:Invent next week, so maybe we can connect there. >> > >> >> > >> On Tue, Nov 26, 2024 at 6:35 PM Nikhil Benesch >> > >> <nikhil.bene...@gmail.com> wrote: >> > >> > >> > >> > Hi all, >> > >> > >> > >> > With Amazon S3 announcing support for the If-Match header yesterday >> > >> > [0], all the >> > >> > major object store implementations now support a compare-and-swap >> > >> > operation. >> > >> > >> > >> > As far as I can tell, this opens up the possibility of storing Iceberg >> > >> > catalogs directly on object storage, without the need for a separate >> > >> > metastore, >> > >> > and without violating any of Iceberg's ACID guarantees. >> > >> > >> > >> > It seems the immediate next step is to build an independent Java or >> > >> > REST catalog >> > >> > backend to prove this concept out. Long term, though, the ideal would >> > >> > be to >> > >> > have such a catalog backend be a first class citizen in the Iceberg >> > >> > project. >> > >> > >> > >> > Is anyone else in the Iceberg community barking up this tree? I'm a >> > >> > long term >> > >> > Iceberg enthusiast, but new to the community. I'd very much >> > >> > appreciate any >> > >> > pointers to current or past discussions on the topic. So far all I've >> > >> > been >> > >> > able to turn up is some light chatter from myself and others on >> > >> > Bluesky and >> > >> > Hacker News ([1][2][3]). >> > >> > >> > >> > Cheers, >> > >> > Nikhil >> > >> > >> > >> > [0]: >> > >> > https://aws.amazon.com/about-aws/whats-new/2024/11/amazon-s3-functionality-conditional-writes/ >> > >> > [1]: https://bsky.app/profile/benesch.bsky.social/post/3lauesxg3ic2c >> > >> > [2]: https://bsky.app/profile/eatonphil.bsky.social/post/3lbskq3jwk22e >> > >> > [3]: https://news.ycombinator.com/item?id=42240370
