How does HTTP caching handle desynchronized clocks between clients and the server?
- At *t0*, the client gets the latest table version. - At *t1*, the server makes a new commit. - At *t2*, the client sends a request with a timestamp *t2*, but due to desynchronization, it refers to *t0*. The server may reply with 304 Not Modified, causing the client to think its cache is up-to-date and miss the commit at *t1*. Yufei On Fri, Nov 15, 2024 at 6:37 AM Gabor Kaszab <gaborkas...@apache.org> wrote: > Hi All, > > First of all it's great to see that there are others who could benefit > from giving a solution to this problem. I appreciate all the comments and > feedback so far. > There were a number of different opinions, so let me start with > summarizing the different topics that came up: > > *New endpoint vs using an existing endpoint:* > Based on the answers (Fokko, Yufei) I had the impression that we should be > careful when adding new REST endpoints, and we should examine the re-use of > existing endpoints first. Let's do that then, and in case we don't find it > feasible then we can still fall back to any of my initial proposals > (isLatest() or metadataLocation()). > > *Granularity of freshness checks:* > It was brought up (Dmitri) that we might not want to do the metadata > freshness checks solely based on metadata location, but we should consider > doing more granular freshness checks. I personally don't see much benefit > of designing this solution like that, TBH, but seeing some use-cases could > help us understand the motivation here. > Let me share my opinion on some of the arguments: > > "A change in metadata location does not necessarily mean a change in > metadata content" > > AFAIK whenever Iceberg creates a new metadata file there is some change in > the metadata itself. There might not be a new snapshot, though in the cases > of e.g. a schema/partition evolution. But even in these cases triggering a > table reload could make sense to me (e.g. answering SHOW CREATE TABLE and > similar queries). Additionally, I'd assume the number of metadata location > changes that don't create a new snapshot is too negligible to optimize for. > Dmitri, let me know if I misunderstood something. > > "it may still be beneficial to permit the client to ask for changes to > specific areas of metadata" > > This seems like a use-case that the partial metadata loading proposal > could solve. To identify the need to load a specific part of the metadata > with partial metadata loading seems an overkill to design with my proposal, > if this is what you have in mind. Also I found that the partial metadata > loading proposal faces serious headwinds, so I wouldn't rely on it at the > moment. > > *Re-using tableExists* > I think there is a consensus here that tableExists returning a metadata > location could work but seems more like a workaround and could be > misleading for the users. > > *Partial metadata loading could solve this:* > (Yufei) I agree, it would be perfect for my use-case and I'm following the > discussion on the proposal. However, for me it seems, as I wrote above, > that the proposal faces serious headwinds now and I honestly wouldn't > expect a solution in the short term. But solving the freshness problems is > a more urgent thing to solve, not just for myself and Impala but apparently > to many other stakeholders in the community according to the interest on > this thread. > Hence, I propose to come up with a separate solution for freshness checks, > and we can still move to using partial metadata loading once that's out. > > *Use HTTPCache and If-Modified-Since with loadTable* > This solution seems to do the trick for us. Let me do some research myself > to see if there are any difficulties implementing this. Currently, I have > more questions than answers wrt this approach :) > - The initial problem is to answer freshness questions for the cached > tables on the client side. If we introduce HttpCaching wouldn't we > introduce the same problem but on a different level of representation. We'd > then need to decide the freshness/staleness of the cached data in the HTTP > layer. > - If we cache the HTTP responses for a loadTable then we essentially cache > the content of the metadata.jsons including the snapshot and metadata log > and everything, plus the snapshot list (and I think the manifests for the > latest snapshot). I believe that the size of this can easily reach the low > megabytes range in memory, so in total keeping them in the HTTP Cache for > all the tables we have queried can easily mean that we keep a couple of GBs > in memory just for this purpose. > For engines that already cache table metadata wouldn't this mean that we > will cache some parts of the metadata redundantly? > - How would we decide what is the max-age of a cached table metadata in > the HTTP Cache? Would it be configurable so that each engine could use > whatever it prefers? > > Sorry if any of the questions doesn't make sense, I just want to make sure > I understand all the aspects of this approach. > > An additional topic I have in mind: > *REST catalog vs other catalogs:* > Now we are focusing our discussion on the REST spec, but I think it would > be beneficial to extend our focus and cover other catalog implementations > too. I don't think that this problem of data freshness is specific to REST > catalog, it could affect any table in any other catalog too. > > I'll continue my investigation wrt the proposals, I just wanted to flush > out and sum up what we have now before the weekend. > > Regards, > Gabor > > > On Fri, Nov 15, 2024 at 10:16 AM Jean-Baptiste Onofré <j...@nanthrax.net> > wrote: > >> Hi, >> >> I like the idea and it makes sense. As soon as it's clearly stated in >> the spec (using If-Modified-Since header and 304 status code), it >> looks good to me. >> >> Thanks ! >> Regards >> JB >> >> On Fri, Nov 15, 2024 at 1:58 AM Taeyun Kim <taeyun....@innowireless.com> >> wrote: >> > >> > Hi, >> > >> > (Apologies if this email is a duplicate. This is my third attempt.) >> > >> > I also need a way to ensure that my table data is up-to-date. For now, >> I’m handling this by setting an expiration period after which I fetch the >> data again, regardless of its freshness. >> > >> > Here are my thoughts on the current suggestions. Please correct me if >> I've misunderstood any of the points. >> > >> > - isLatest(): This function could be inefficient since it would require >> an additional round-trip to fetch the metadata if it’s not up-to-date. This >> would result in two round-trips overall, which seems suboptimal. >> > - metadataLocation(): This has a similar issue as isLatest(). BTW, >> according to the REST catalog API documentation for LoadTableResult schema, >> it states, "Clients can check whether metadata has changed by comparing >> metadata locations after the table has been created." ( >> https://github.com/apache/iceberg/blob/3659ded18d50206576985339bd55cd82f5e200cc/open-api/rest-catalog-open-api.yaml#L3175) >> This suggests that if the metadata location has changed, the metadata can >> be considered updated. >> > - tableExists(): Based on the name, this function seems to serve a >> different purpose. >> > >> > Here is my suggestion: >> > >> > Since HTTP has built-in caching features ( >> https://developer.mozilla.org/en-US/docs/Web/HTTP/Caching), and REST >> catalogs operate over HTTP, it seems natural to leverage HTTP caching >> mechanisms. For example, HTTP includes the If-Modified-Since header and the >> 304 Not Modified status code. Using this approach, we could achieve data >> freshness with a single round-trip, fetching updated data only if there are >> modifications. >> > >> > What do you think about defining the spec in this direction? >> > >> > Thank you. >> > >> > >> > >> > >> > -----Original Message----- >> > From: "Yufei Gu" <flyrain...@gmail.com> >> > To: <dev@iceberg.apache.org>; >> > Cc: >> > Sent: 2024-11-13 (수) 03:43:24 (UTC+09:00) >> > Subject: Re: [DISCUSS] REST: Way to query if metadata pointer is the >> latest >> > >> > >> > >> > Hi Gamber, >> > >> > Thanks for the proposal! Impala isn’t unique in needing this—I've seen >> similar requirements from other engines. >> > >> > As others pointed out, using the “tableExists” endpoint seems like a >> workaround. I don't consider it a permanent way forward. We could address >> this by either modifying the current load table endpoint or introducing a >> new one, but ideally, we should avoid adding endpoints for every specific >> need. With that, partial metadata loading seems like a strong approach >> here, we will need certain agreement though. I'd suggest the community >> consider the use cases seriously. We need a way forward. >> > >> > I’m also not too concerned about using metadata file paths to verify >> the latest table version; clients can simply extract metadata filenames, >> which include the UUID. >> > >> > Yufei >> > >> > >> > >> > >> > On Tue, Nov 12, 2024 at 7:46 AM Jean-Baptiste Onofré <j...@nanthrax.net> >> wrote: >> > >> > Hi Fokko >> > >> > I like the idea, but I think it's more a workaround and could be >> > confusing for users :) >> > >> > Regards >> > JB >> > >> > On Tue, Nov 12, 2024 at 2:53 PM Fokko Driesprong <fo...@apache.org> >> wrote: >> > > >> > > Hey Gabor, >> > > >> > > Thanks for raising this. While reading this, my first thought is to >> leverage the `tableExists` operation: >> > > >> https://github.com/apache/iceberg/blob/e3f39972863f891481ad9f5a559ffef093976bd7/open-api/rest-catalog-open-api.yaml#L1129-L1160 >> > > >> > > This doesn't return anything today, but we could return a payload to >> the latest metadata.json. >> > > >> > > Looking forward to what others think. >> > > >> > > Kind regards, >> > > Fokko >> > > >> > > >> > > >> > > >> > > Op di 12 nov 2024 om 14:33 schreef Shani Elharrar >> <sh...@upsolver.com.invalid>: >> > >> >> > >> I recommend option (b), provided there is no partial metadata >> loading. We implemented option (b) internally to facilitate partial >> metadata loading, as we have tables with hundreds of thousands of >> snapshots. This results in metadata that occupies approximately 500 MB in >> memory (excluding the JsonNodes), which is a significant load for some of >> our services. >> > >> >> > >> Shani. >> > >> >> > >> On 12 Nov 2024, at 14:12, Gabor Kaszab <gaborkas...@apache.org> >> wrote: >> > >> >> > >> Hey Iceberg Community, >> > >> >> > >> Background: >> > >> Impala is designed in a way to cache the Iceberg table metadata >> (BaseTable objects in practice) for faster access. Currently, Impala is >> tightly coupled with HMS and in turn with the HiveCatalog, and in order to >> keep the cached table objects up-to-date there is a notification mechanism >> driven by HMS to notify Impala about any changes in the table metadata. >> > >> The Impala community is actively looking for ways to decouple HMS >> from Impala and provide a way to use Impala without the need for HMS, and >> get the Iceberg table metadata from other catalog Implementations mainly >> focusing now on REST catalogs. >> > >> >> > >> Problem to solve: >> > >> We identified a particular missing functionality in the current REST >> spec: For engines that cache table metadata currently there is no way to >> check if that table metadata is up-to-date or not, and whether the engine >> should reload the metadata for that table or not without getting a whole >> table object from the catalog. For this I think the REST catalog (but in >> fact I think this could apply to any other catalogs) should be able to >> answer a question like: >> > >> "Hi Catalog, I have this version of this table, is it up-to-date?" >> > >> >> > >> Proposal: >> > >> I've been following the discussion about partial metadata loading >> that could be also used to answer the above question, but I have the >> impression now that the conversation stopped making any progress. >> > >> So instead of waiting for partial metadata loading I propose to have >> an addition to the REST spec now to answer the question I raised above: >> > >> >> > >> a) boolean isLatest(TableIdentifier ident, String metadataLocation); >> > >> b) String metadataLocation(TableIdentifier ident); >> > >> >> > >> Any of the above 2 approaches could help engines to decide if they >> have to invalidate/reload particular table metadata in the cache. I >> personally would go for option a) but would be open to hear other opinions. >> > >> >> > >> I'd like to know if the community could support me extending the >> REST spec with any of the 2 options. >> > >> >> > >> Regards, >> > >> Gabor >> > >> >> > >> >> >
