Oops. In the first paragraph, I meant “when we added the endpoint to load a VIEW, metadata-location was correctly marked as required."
On Thu, Feb 29, 2024 at 4:18 PM Ryan Blue <b...@tabular.io> wrote: > Once again, I’m catching up late and might have a helpful perspective. > > I think there was a mistake in the OpenAPI spec for loading tables and the > metadata-location is not listed as required. I don’t recall that being > intentional, but maybe it was? Maybe for a different reason? Either way, > when we added the endpoint to load a catalog, metadata-location was > correctly marked as required. > > Whatever the reason for the field being optional, *the intent was never > to change requirements from Iceberg* that metadata is written to files > and atomic operations guarantee a linear history. > > I’m glad to clear up the confusion on that. Right now, *catalogs must > write metadata files for Iceberg tables and should guarantee a linear > history*. > > You may be able to get away with bending those rules (what Dan refers to > as not compliant), but that’s unintentional. We may also choose to relax > the requirement for metadata files in the future — I see support for the > idea and have considered proposing it also. But for now, it’s a > requirement, even if you don’t have to send the location to the client > (though note that the client has a hard dependency on it!). > > Ryan > > On Thu, Feb 29, 2024 at 4:06 PM Daniel Weeks <daniel.c.we...@gmail.com> > wrote: > >> 1. I agree, this is what the spec currently requires >> >> 2. I agree, it's up for consideration >> >> 3. I agree, I think if an implementation didn't adhere to the current >> spec requirements, I would say it's out of spec (not sure I'd go as far as >> to say it's a different kind of table entirely). >> >> Just to expand on #3, we will find lots of cases where implementations >> deviate (likely unintentionally) from the rest/table spec and I feel like >> the goal is to identify those cases and steer them back into compliance >> with the spec as opposed to immediately claiming they're something entirely >> different. The overarching goal is to improve openness and >> interoperability. >> >> My main point is that there isn't an inherent incompatibility between the >> REST spec and the Iceberg spec. The preservation of the storage >> representation was discussed and intentional during the design/development >> of the REST spec. >> >> -Dan >> >> >> On Thu, Feb 29, 2024 at 3:40 PM Jack Ye <yezhao...@gmail.com> wrote: >> >>> > For example, I cannot validate the atomic behaviors Glue claims, but I >>> wouldn't assert that it is non-compliant because of that. >>> >>> I think these are not comparable claims because the API scope is >>> completely different, but I don't think it's worth arguing in depth. Let's >>> try to see if we can have some consensus. >>> >>> Based on what you said above, do you agree with the following 3 points? >>> >>> 1. Today, a table/view in any catalog including a REST spec-compatible >>> catalog is an Iceberg table/view if and only if it points to a JSON >>> metadata file in storage. This concept is a part of the Iceberg table/view >>> spec. There is a debate to be had for if we want to remove this requirement >>> or not. The argument for it (as Yufei said) is to use other storage for >>> better performance. The argument against it (as Amogh said) is to keep >>> Iceberg open source friendly through the JSON format. >>> >>> 2. Today, a table/view in any catalog including a REST spec-compatible >>> catalog is an Iceberg table/view if and only if it behind the scene >>> performs the atomic metadata file swap for every commit. This concept is a >>> part of the Iceberg table/view spec. We should consider removing this >>> requirement in the Iceberg table/view spec. >>> >>> 3. A table/view in an Iceberg REST spec-compatible catalog may or may >>> not be an Iceberg table/view. The REST spec does not enforce this, and this >>> stance will remain true going forward. For example, it could use the >>> Iceberg table/view metadata structure but does not store the metadata in >>> JSON file, or not use the metadata file swap commit procedure, or both, and >>> in those cases it is not an Iceberg table/view. More extremely, it might be >>> a totally different kind of table that is only surfaced through the REST >>> models. >>> >>> -Jack >>> >>> On Thu, Feb 29, 2024 at 2:13 PM Daniel Weeks <daniel.c.we...@gmail.com> >>> wrote: >>> >>>> > In that case are tables in a REST-compliant catalog still an Iceberg >>>> table? I don't think so, because it is a table that only partially follows >>>> the Iceberg table spec. >>>> >>>> If the catalog is REST compliant and complies with the Iceberg spec, >>>> they are still Iceberg tables. I can see there is an argument that if the >>>> catalog is REST compliant but does not follow the commit requirements (or >>>> aspects of the Iceberg spec), that you cannot call those Iceberg tables. >>>> But the assertion that Iceberg tables in a REST catalog are de facto >>>> non-compliant is incorrect. >>>> >>>> > I like the idea about validation for format compliance. But don't >>>> think you can technically validate this. You can validate the static table >>>> to see if it has all the Iceberg metadata components, but you can not >>>> validate the internal behavior of the service during a commit to see if it >>>> really atomically swapped a metadata file. >>>> >>>> Just because you cannot see/validate the implementation doesn't mean >>>> that it is non-compliant. For example, I cannot validate the atomic >>>> behaviors Glue claims, but I wouldn't assert that it is non-compliant >>>> because of that. >>>> >>>> I do think there is a discussion to be had about if/when we might >>>> adjust the storage/swap requirements, but to reinforce Amogh's point, >>>> removing those requirements would impact the openness and accessibility of >>>> Iceberg, which I feel would hamper adoption. >>>> >>>> -Dan >>>> >>>> >>>> >>>> On Thu, Feb 29, 2024 at 1:53 PM Yufei Gu <flyrain...@gmail.com> wrote: >>>> >>>>> We've periodically discussed removing the storage requirement and I >>>>>> think there's a path forward to do that and would agree that >>>>>> standardizing >>>>>> on REST, but I wouldn't say the justification for making this push is >>>>>> that >>>>>> REST is not compliant so we can just ignore the table spec requirements. >>>>>> There are a few more things to consider, which is that not everything >>>>>> can use REST currently and making a hard cut away from file based >>>>>> metadata >>>>>> could bifurcate access to Iceberg data. There are also aspects to the >>>>>> spec >>>>>> that reference the metadata paths (like metadata log, though it's >>>>>> optional), but would likely need to be addressed. >>>>> >>>>> >>>>> This is a bit off-topic. It makes sense to me to remove the storage >>>>> requirement moving foward. The metadata.json file isn't necessary in the >>>>> Rest catalog. For example, the rest catalog may not have the permission to >>>>> write to the table owner's storage. It still can save it as a file of >>>>> course, but doesn't quite make sense. Putting it in a key-value store or >>>>> RDMS could be a better option. >>>>> >>>>> Given that we are going to remove the storage requirement. Should we >>>>> avoid the file path in the current design for things like view spec? A >>>>> solution like table identifier + version uuid may serve the purpose. >>>>> >>>>> Yufei >>>>> >>>>> >>>>> On Thu, Feb 29, 2024 at 1:29 PM Jack Ye <yezhao...@gmail.com> wrote: >>>>> >>>>>> > There's no exemption that says if you're using REST you don't need >>>>>> to follow the spec. Why do you think that's the case? >>>>>> >>>>>> In that case are tables in a REST-compliant catalog still an Iceberg >>>>>> table? I don't think so, because it is a table that only partially >>>>>> follows >>>>>> the Iceberg table spec. >>>>>> >>>>>> I like the idea about validation for format compliance. But don't >>>>>> think you can technically validate this. You can validate the static >>>>>> table >>>>>> to see if it has all the Iceberg metadata components, but you can not >>>>>> validate the internal behavior of the service during a commit to see if >>>>>> it >>>>>> really atomically swapped a metadata file. >>>>>> >>>>>> So I think at minimum we should update the table/view spec to remove >>>>>> the metadata file swap requirement. The Iceberg table/view spec should >>>>>> be a >>>>>> pure format spec that specifies how the file is laid out in storage. >>>>>> >>>>>> -Jack >>>>>> >>>>>> On Thu, Feb 29, 2024 at 1:22 PM Amogh Jahagirdar <am...@tabular.io> >>>>>> wrote: >>>>>> >>>>>>> I want to echo Dan's point that just because there is a separate >>>>>>> spec for a REST Catalog does not mean that implementations can deviate >>>>>>> from >>>>>>> the spec's definition of the commit protocol or metadata layout, and >>>>>>> still >>>>>>> be considered "spec compliant". >>>>>>> >>>>>>> > Secondly, once we do that, we should declare REST spec as the >>>>>>> official catalog spec to interact with Iceberg tables. Otherwise at >>>>>>> least I >>>>>>> will be very tempted to just break the atomic pointer swap pattern and >>>>>>> store the entire metadata using the Glue Table object to achieve much >>>>>>> better performance and also Glue native feature integrations, and I >>>>>>> think >>>>>>> other players will be equally motivated to do something similar. That >>>>>>> will >>>>>>> lead to even more chaos in the Iceberg catalog space. >>>>>>> >>>>>>> On this, a second point I want to make is around the openness of >>>>>>> this ecosystem. We all already know that openness (the file formats, the >>>>>>> metadata layout, the spec itself) is a fundamental tenant of the >>>>>>> project. >>>>>>> If we take the provided example of removing the metadata JSON file and >>>>>>> moving it to some other storage, I think that goes against this >>>>>>> principle >>>>>>> since a JSON file is quite open by definition. Going back to the first >>>>>>> point, I think a catalog which has such a behavior would *not* be >>>>>>> considered spec compliant. Another reason this is important is if we >>>>>>> think >>>>>>> about what's healthiest for all users of Iceberg, is to have a healthy >>>>>>> list >>>>>>> of options for catalog choices. Storing the metadata JSON in non-open >>>>>>> ways >>>>>>> can make users lives harder for trying out new catalogs since now the >>>>>>> metadata would be stored in their own way, and the users will have a >>>>>>> harder >>>>>>> time accessing their own data. >>>>>>> >>>>>>> A last point I'd like to make is I think there's a good discussion >>>>>>> to be had on how do we validate that a REST Catalog implementation is >>>>>>> spec >>>>>>> compliant. I think that's really beneficial for the ecosystem as a >>>>>>> whole. >>>>>>> Before that, I think first though we'd want to conclude on this topic >>>>>>> itself. >>>>>>> >>>>>>> On Thu, Feb 29, 2024 at 12:29 PM Daniel Weeks < >>>>>>> daniel.c.we...@gmail.com> wrote: >>>>>>> >>>>>>>> > REST spec-compliant catalog does not need to follow the Iceberg >>>>>>>> spec to commit or store metadata >>>>>>>> >>>>>>>> If the REST implementation doesn't follow the Iceberg spec for >>>>>>>> commit requirements, it's not compliant with the spec. There's no >>>>>>>> exemption that says if you're using REST you don't need to follow the >>>>>>>> spec. Why do you think that's the case? >>>>>>>> >>>>>>>> I don't believe there's a reason to say that the REST spec needs to >>>>>>>> enforce the commit requirements either, that's a requirement of the >>>>>>>> Iceberg >>>>>>>> spec and still needs to be complied with. >>>>>>>> >>>>>>>> -Dan >>>>>>>> >>>>>>>> On Thu, Feb 29, 2024 at 12:19 PM Jack Ye <yezhao...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> > The implementation of the spec can either be compliant or not. >>>>>>>>> >>>>>>>>> This is exactly the problem we are talking about right? Just to >>>>>>>>> give an example, we cannot technically say that tables/views in the >>>>>>>>> Tabular >>>>>>>>> catalog are Iceberg tables/views, because a REST spec-compliant >>>>>>>>> catalog >>>>>>>>> does not need to follow the Iceberg spec to commit or store metadata. >>>>>>>>> Even >>>>>>>>> if you say it is, there is no way to really prove that, because the >>>>>>>>> REST >>>>>>>>> spec does not enforce it. >>>>>>>>> >>>>>>>>> JB, what do you mean by participating on the Catalog RFC? Is there >>>>>>>>> already an ongoing RFC? >>>>>>>>> >>>>>>>>> -Jack >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Feb 29, 2024 at 12:08 PM Jean-Baptiste Onofré < >>>>>>>>> j...@nanthrax.net> wrote: >>>>>>>>> >>>>>>>>>> Hi Dan, >>>>>>>>>> >>>>>>>>>> I agree with your statement about REST Spec is not an implement >>>>>>>>>> but I strongly disagree with your statement "impl of the spec can >>>>>>>>>> either be >>>>>>>>>> compliant or not". >>>>>>>>>> >>>>>>>>>> The REST Catalog spec impl should be consistent with the REST >>>>>>>>>> Spec. That's why a reference implementation in Iceberg would be a >>>>>>>>>> must, >>>>>>>>>> with a TCK. >>>>>>>>>> >>>>>>>>>> The REST Spec should bridge/give access to Table/View metadata. I >>>>>>>>>> think it would make sense to have a resource to GET the Table/View >>>>>>>>>> metadata, also supporting PUT to update. >>>>>>>>>> JSON Schema and eventually JSON RPC could help on some area here >>>>>>>>>> (compliant with OpenAPI). >>>>>>>>>> >>>>>>>>>> In another thread, I propose to work on a Catalog RFC, exactly to >>>>>>>>>> target this. I think it would make sense to have the REST/Catalog >>>>>>>>>> RFC as >>>>>>>>>> the main catalog API, so it has to be both consistent (giving access >>>>>>>>>> to >>>>>>>>>> table/view metadata) and extensible (via OpenAPI Extensions for >>>>>>>>>> instance). >>>>>>>>>> >>>>>>>>>> So, I agree with Jack: the minimum would be to have JSON metadata >>>>>>>>>> exposed by the REST Spec. >>>>>>>>>> >>>>>>>>>> @Jack, short term I'm in favor of your proposal, long term, I >>>>>>>>>> propose to participate on the Catalog RFC (REST Spec). WDYT ? >>>>>>>>>> >>>>>>>>>> Thanks ! >>>>>>>>>> Regards >>>>>>>>>> JB >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Le jeu. 29 févr. 2024 à 20:47, Daniel Weeks < >>>>>>>>>> daniel.c.we...@gmail.com> a écrit : >>>>>>>>>> >>>>>>>>>>> Hey Jack, >>>>>>>>>>> >>>>>>>>>>> I'm not sure I agree with the framing of this argument. The >>>>>>>>>>> REST Spec defines a protocol, not an implementation. >>>>>>>>>>> >>>>>>>>>>> The implementation of the spec can either be compliant or not. >>>>>>>>>>> So a REST Implementation that adheres to all the requirements >>>>>>>>>>> (atomic >>>>>>>>>>> location swap, json representation, etc.), would be compliant. >>>>>>>>>>> There's no >>>>>>>>>>> requirement around who performs these operations and with REST, >>>>>>>>>>> that is >>>>>>>>>>> delegated to the server. The optional metadata location doesn't >>>>>>>>>>> mean that >>>>>>>>>>> there isn't a metadata location, just that it may not be exposed >>>>>>>>>>> directly >>>>>>>>>>> in the response. >>>>>>>>>>> >>>>>>>>>>> Therefore, an implementation where you just store the table >>>>>>>>>>> metadata in a Glue Table object, would not be compliant, currently. >>>>>>>>>>> >>>>>>>>>>> We've periodically discussed removing the storage requirement >>>>>>>>>>> and I think there's a path forward to do that and would agree that >>>>>>>>>>> standardizing on REST, but I wouldn't say the justification for >>>>>>>>>>> making this >>>>>>>>>>> push is that REST is not compliant so we can just ignore the table >>>>>>>>>>> spec >>>>>>>>>>> requirements. >>>>>>>>>>> >>>>>>>>>>> There are a few more things to consider, which is that not >>>>>>>>>>> everything can use REST currently and making a hard cut away from >>>>>>>>>>> file >>>>>>>>>>> based metadata could bifurcate access to Iceberg data. There are >>>>>>>>>>> also >>>>>>>>>>> aspects to the spec that reference the metadata paths (like >>>>>>>>>>> metadata log, >>>>>>>>>>> though it's optional), but would likely need to be addressed. >>>>>>>>>>> >>>>>>>>>>> -Dan >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Feb 29, 2024 at 11:13 AM Jack Ye <yezhao...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi everyone, >>>>>>>>>>>> >>>>>>>>>>>> Just want to pull this specific topic out of the materialized >>>>>>>>>>>> view discussion thread. I noticed this during the MV discussion, >>>>>>>>>>>> and I >>>>>>>>>>>> think it is important to clarify this not just for the MV topic, >>>>>>>>>>>> but also >>>>>>>>>>>> for the ongoing discussion to consolidate all the different >>>>>>>>>>>> catalogs. >>>>>>>>>>>> >>>>>>>>>>>> *How the table/view spec defines Iceberg table/view* >>>>>>>>>>>> >>>>>>>>>>>> If we look into the table/view spec, the optimistic >>>>>>>>>>>> concurrency section >>>>>>>>>>>> <https://iceberg.apache.org/spec/#optimistic-concurrency> >>>>>>>>>>>> requires the existence of a metadata file, and the atomic swap of >>>>>>>>>>>> the >>>>>>>>>>>> metadata file ensures serializable isolation. This implies 2 >>>>>>>>>>>> things: >>>>>>>>>>>> 1. the metadata file in a storage that holds the information >>>>>>>>>>>> described in the rest of the spec. >>>>>>>>>>>> 2. there is an object in a catalog that holds the pointer of >>>>>>>>>>>> the metadata file. What object and what catalog is implementation >>>>>>>>>>>> dependent, but these generalized concepts are always intact. >>>>>>>>>>>> >>>>>>>>>>>> The JSON serialization parts of the spec plus the reader >>>>>>>>>>>> requirements also implies that the metadata file is in JSON format. >>>>>>>>>>>> >>>>>>>>>>>> So when we talk about an Iceberg table/view that is compliant >>>>>>>>>>>> with the spec, it is the combination of all these 5 requirements: >>>>>>>>>>>> 1. there is an object in the catalog representing this >>>>>>>>>>>> table/view >>>>>>>>>>>> 2. there is a pointer to a JSON metadata file in the object >>>>>>>>>>>> 3. the JSON metadata file exists in storage and contains the >>>>>>>>>>>> table/view metadata content >>>>>>>>>>>> 4. the metadata content is compliant with the standard >>>>>>>>>>>> described in the spec >>>>>>>>>>>> 5. serializable isolation is achieved by atomic swap of the >>>>>>>>>>>> object pointer >>>>>>>>>>>> >>>>>>>>>>>> *How non-REST catalogs are compliant with the table/view spec* >>>>>>>>>>>> >>>>>>>>>>>> An implementation of the Iceberg table/view is essentially >>>>>>>>>>>> specifying: >>>>>>>>>>>> 1. what is the exact implementation of the catalog, e.g. JDBC, >>>>>>>>>>>> Hive metastore (HMS), Glue, etc. >>>>>>>>>>>> 2. what is the object that represents a table, e.g. a row in >>>>>>>>>>>> the "iceberg_tables" table in JDBC, a Table object in HMS/Glue, >>>>>>>>>>>> etc. >>>>>>>>>>>> 3. how is the JSON metadata file pointer stored, e.g. a column >>>>>>>>>>>> in the table's row in JDBC, metadata_location key in the Table's >>>>>>>>>>>> parameter >>>>>>>>>>>> map in HMS/Glue, etc. >>>>>>>>>>>> 4. how the atomic swap is implemented, e.g. SQL atomic update >>>>>>>>>>>> in JDBC, conditional parameter update in HMS, conditional version >>>>>>>>>>>> update in >>>>>>>>>>>> Glue, etc. >>>>>>>>>>>> >>>>>>>>>>>> *How the REST spec is NOT compliant with the table/view spec* >>>>>>>>>>>> >>>>>>>>>>>> The REST spec technically does not match the following >>>>>>>>>>>> table/view spec requirements: >>>>>>>>>>>> 2. there is a pointer to a JSON metadata file in the object >>>>>>>>>>>> 3. the JSON metadata file exists in storage and contains the >>>>>>>>>>>> table/view metadata content >>>>>>>>>>>> 5. serializable isolation is achieved by atomic swap of the >>>>>>>>>>>> object pointer >>>>>>>>>>>> >>>>>>>>>>>> The key parts in REST spec that are not compliant are: >>>>>>>>>>>> 1. metadata-location field is optional in LoadTableResponse >>>>>>>>>>>> <https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L2721-L2728> >>>>>>>>>>>> 2. pointer swap is not enforced in the UpdateTable >>>>>>>>>>>> <https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L658> >>>>>>>>>>>> operation >>>>>>>>>>>> >>>>>>>>>>>> Therefore, it opens the door for a REST service to be >>>>>>>>>>>> completely not dependent on a JSON metadata file, store the Iceberg >>>>>>>>>>>> table/view metadata not as a file, and achieve much better >>>>>>>>>>>> performance >>>>>>>>>>>> characteristics than other catalogs. This technically gives a >>>>>>>>>>>> unique >>>>>>>>>>>> advantage for REST catalog adopters that is not there for non-REST >>>>>>>>>>>> catalogs >>>>>>>>>>>> like HMS and Glue. >>>>>>>>>>>> >>>>>>>>>>>> *How can we fix this?* >>>>>>>>>>>> >>>>>>>>>>>> I suggest the following: >>>>>>>>>>>> >>>>>>>>>>>> Firstly, I think it is good that we try to remove the >>>>>>>>>>>> requirements of JSON metadata file pointer and atomic pointer >>>>>>>>>>>> swap. We know >>>>>>>>>>>> these requirements have perf limitations based on production usage, >>>>>>>>>>>> especially when the metadata file is large. If that is the >>>>>>>>>>>> direction, we >>>>>>>>>>>> should make it official by changing the table/view spec to say >>>>>>>>>>>> that those >>>>>>>>>>>> requirements are catalog level implementation details that are no >>>>>>>>>>>> longer >>>>>>>>>>>> required. >>>>>>>>>>>> >>>>>>>>>>>> Secondly, once we do that, we should declare REST spec as the >>>>>>>>>>>> official catalog spec to interact with Iceberg tables. Otherwise >>>>>>>>>>>> at least I >>>>>>>>>>>> will be very tempted to just break the atomic pointer swap pattern >>>>>>>>>>>> and >>>>>>>>>>>> store the entire metadata using the Glue Table object to achieve >>>>>>>>>>>> much >>>>>>>>>>>> better performance and also Glue native feature integrations, and >>>>>>>>>>>> I think >>>>>>>>>>>> other players will be equally motivated to do something similar. >>>>>>>>>>>> That will >>>>>>>>>>>> lead to even more chaos in the Iceberg catalog space. >>>>>>>>>>>> >>>>>>>>>>>> With REST spec as the official catalog spec, we can actually >>>>>>>>>>>> support non-REST catalogs by using the HTTP execution chain >>>>>>>>>>>> handler. Dan >>>>>>>>>>>> has already done a prototype here >>>>>>>>>>>> <https://github.com/apache/iceberg/commit/619127ff69f89e43a1edef2ea94c3dd439396a8d#diff-869264a83ba9ca657e7defefaa16ad196b0de9fce6c87f97533db77f29e44762> >>>>>>>>>>>> that is based on this discussion >>>>>>>>>>>> <https://github.com/apache/iceberg/pull/8091#issuecomment-1647189146> >>>>>>>>>>>> in the past about using AWS Lambda as an alternative HTTP client >>>>>>>>>>>> for REST >>>>>>>>>>>> catalog. The same approach can be used to talk to >>>>>>>>>>>> HMS/Glue/JDBC/... while >>>>>>>>>>>> users will only interact with the RESTCatalog as the entry point. >>>>>>>>>>>> >>>>>>>>>>>> I think this can provide a good path forward overall for the >>>>>>>>>>>> catalog consolidation story, interested to know what others think. >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> Jack Ye >>>>>>>>>>>> >>>>>>>>>>>> > > -- > Ryan Blue > Tabular > -- Ryan Blue Tabular
