Thanks all for the replies. The Flink and Spark implementations are indeed using the CachingCatalog wrapper, but as far as I can see, they are the only users of it.
I rather wanted to understand why this wrapper isn't applied inside the RESTCatalog class directly. In our implementation, we also wrap the catalog instance in the CachingCatalog. However, since this functionality is not included in the RESTCatalog class, clients have to be aware of this wrapper. It looks like different cache implementations exist in different projects and I wonder if this should be implemented in the core library instead to improve the user experience. To put it directly: does it make sense to include this functionality directly into the RESTCatalog class? Regards, Anatolii. On Tue, Sep 30, 2025 at 11:08 PM Amogh Jahagirdar <[email protected]> wrote: > +1 to what others mentioned, we already have the abstractions to wrap a > CachingCatalog around a RESTCatalog implementation. Should just be able to > do CachingCatalog.wrap in your application or if you're using an engine, > enable the cache through one of the configs. > > Thanks, > Amogh Jahagirdar > > On Tue, Sep 30, 2025 at 1:34 PM Gábor Kaszab <[email protected]> > wrote: > >> Hi Anatolii, >> >> You can wrap RESTCatalog with CachingCatalog and as Yufei said Spark >> already does this. On top of this I'm actively working on an enhancement I >> call freshness-aware table loading where on the client side there is a >> table cache and results could be answered using this cache in case the >> server detects that the table is unchanged since the last load table call. >> In case you are interested, here you can find more details >> <https://docs.google.com/document/d/1rnVSP_iv2I47giwfAe-Z3DYhKkKwWCVvCkC9rEvtaLA/edit?usp=sharing> >> . >> >> Best Regards, >> Gabor >> >> Yufei Gu <[email protected]> ezt írta (időpont: 2025. szept. 30., K, >> 20:59): >> >>> Hi Anatolii, >>> >>> Looking through the code, the RESTCatalog is already wrapped by >>> a CachingCatalog, >>> https://github.com/apache/iceberg/blob/35cda2adc26ac374246c094d0a50bc7606e03346/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkCatalog.java#L759-L759. >>> Is that what are you looking for? >>> >>> Yufei >>> >>> >>> On Tue, Sep 30, 2025 at 7:53 AM Anatolii Popov >>> <[email protected]> wrote: >>> >>>> Hi all, >>>> >>>> While working with the REST Catalog, I noticed that there doesn't >>>> appear to be a client-side table metadata cache implementation similar to >>>> what is available for the Flink and Spark catalogs, where a catalog >>>> instance can be wrapped in a CachingCatalog through configuration. >>>> >>>> I am wondering if there has been any discussion about providing a >>>> client-side caching layer for the REST Catalog as well, perhaps for >>>> environments where the REST server itself doesn't implement caching or when >>>> further reducing calls to the catalog from the client is desirable. >>>> >>>> Would it make sense to have a similar CachingCatalog wrapper for the >>>> REST Catalog to provide a consistent caching mechanism across different >>>> catalog types? >>>> >>>> If this is a desirable feature and not already being worked on, I would >>>> be happy to contribute to its implementation. >>>> >>>> Thanks, >>>> Anatolii. >>>> >>> -- Anatolii Popov Senior Software Developer, *Aiven OY* m: +358505126242 w: aiven.io e: [email protected] <https://www.facebook.com/aivencloud> <https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io>
