> I feel like the goal is to identify those cases and steer them back into compliance with the spec
+100000 > as opposed to immediately claiming they're something entirely different In case this comment is talking about my last sentence "More extremely, it might be a totally different kind of table that is only surfaced through the REST models." I don't mean the Iceberg tables/view in a REST compatible catalog are entirely different. I mean there is another more extreme use case, where the REST catalog can surface other non-Iceberg tables (e.g. Hive Parquet tables) through the same REST model, which is a use case I mentioned previously that is an interesting application of REST that we see some users are interested in. Which is also why the metadata location is important to guide those use cases. > when we added the endpoint to load a VIEW, metadata-location was correctly marked as required hmmm interesting, you are right, I did not notice the difference between table and view. Should we change that for tables then? -Jack On Thu, Feb 29, 2024 at 4:20 PM Ryan Blue <b...@tabular.io> wrote: > Oops. In the first paragraph, I meant “when we added the endpoint to load > a VIEW, metadata-location was correctly marked as required." > > On Thu, Feb 29, 2024 at 4:18 PM Ryan Blue <b...@tabular.io> wrote: > >> Once again, I’m catching up late and might have a helpful perspective. >> >> I think there was a mistake in the OpenAPI spec for loading tables and >> the metadata-location is not listed as required. I don’t recall that >> being intentional, but maybe it was? Maybe for a different reason? Either >> way, when we added the endpoint to load a catalog, metadata-location was >> correctly marked as required. >> >> Whatever the reason for the field being optional, *the intent was never >> to change requirements from Iceberg* that metadata is written to files >> and atomic operations guarantee a linear history. >> >> I’m glad to clear up the confusion on that. Right now, *catalogs must >> write metadata files for Iceberg tables and should guarantee a linear >> history*. >> >> You may be able to get away with bending those rules (what Dan refers to >> as not compliant), but that’s unintentional. We may also choose to relax >> the requirement for metadata files in the future — I see support for the >> idea and have considered proposing it also. But for now, it’s a >> requirement, even if you don’t have to send the location to the client >> (though note that the client has a hard dependency on it!). >> >> Ryan >> >> On Thu, Feb 29, 2024 at 4:06 PM Daniel Weeks <daniel.c.we...@gmail.com> >> wrote: >> >>> 1. I agree, this is what the spec currently requires >>> >>> 2. I agree, it's up for consideration >>> >>> 3. I agree, I think if an implementation didn't adhere to the current >>> spec requirements, I would say it's out of spec (not sure I'd go as far as >>> to say it's a different kind of table entirely). >>> >>> Just to expand on #3, we will find lots of cases where implementations >>> deviate (likely unintentionally) from the rest/table spec and I feel like >>> the goal is to identify those cases and steer them back into compliance >>> with the spec as opposed to immediately claiming they're something entirely >>> different. The overarching goal is to improve openness and >>> interoperability. >>> >>> My main point is that there isn't an inherent incompatibility between >>> the REST spec and the Iceberg spec. The preservation of the storage >>> representation was discussed and intentional during the design/development >>> of the REST spec. >>> >>> -Dan >>> >>> >>> On Thu, Feb 29, 2024 at 3:40 PM Jack Ye <yezhao...@gmail.com> wrote: >>> >>>> > For example, I cannot validate the atomic behaviors Glue claims, but >>>> I wouldn't assert that it is non-compliant because of that. >>>> >>>> I think these are not comparable claims because the API scope is >>>> completely different, but I don't think it's worth arguing in depth. Let's >>>> try to see if we can have some consensus. >>>> >>>> Based on what you said above, do you agree with the following 3 points? >>>> >>>> 1. Today, a table/view in any catalog including a REST spec-compatible >>>> catalog is an Iceberg table/view if and only if it points to a JSON >>>> metadata file in storage. This concept is a part of the Iceberg table/view >>>> spec. There is a debate to be had for if we want to remove this requirement >>>> or not. The argument for it (as Yufei said) is to use other storage for >>>> better performance. The argument against it (as Amogh said) is to keep >>>> Iceberg open source friendly through the JSON format. >>>> >>>> 2. Today, a table/view in any catalog including a REST spec-compatible >>>> catalog is an Iceberg table/view if and only if it behind the scene >>>> performs the atomic metadata file swap for every commit. This concept is a >>>> part of the Iceberg table/view spec. We should consider removing this >>>> requirement in the Iceberg table/view spec. >>>> >>>> 3. A table/view in an Iceberg REST spec-compatible catalog may or may >>>> not be an Iceberg table/view. The REST spec does not enforce this, and this >>>> stance will remain true going forward. For example, it could use the >>>> Iceberg table/view metadata structure but does not store the metadata in >>>> JSON file, or not use the metadata file swap commit procedure, or both, and >>>> in those cases it is not an Iceberg table/view. More extremely, it might be >>>> a totally different kind of table that is only surfaced through the REST >>>> models. >>>> >>>> -Jack >>>> >>>> On Thu, Feb 29, 2024 at 2:13 PM Daniel Weeks <daniel.c.we...@gmail.com> >>>> wrote: >>>> >>>>> > In that case are tables in a REST-compliant catalog still an Iceberg >>>>> table? I don't think so, because it is a table that only partially follows >>>>> the Iceberg table spec. >>>>> >>>>> If the catalog is REST compliant and complies with the Iceberg spec, >>>>> they are still Iceberg tables. I can see there is an argument that if the >>>>> catalog is REST compliant but does not follow the commit requirements (or >>>>> aspects of the Iceberg spec), that you cannot call those Iceberg tables. >>>>> But the assertion that Iceberg tables in a REST catalog are de facto >>>>> non-compliant is incorrect. >>>>> >>>>> > I like the idea about validation for format compliance. But don't >>>>> think you can technically validate this. You can validate the static table >>>>> to see if it has all the Iceberg metadata components, but you can not >>>>> validate the internal behavior of the service during a commit to see if it >>>>> really atomically swapped a metadata file. >>>>> >>>>> Just because you cannot see/validate the implementation doesn't mean >>>>> that it is non-compliant. For example, I cannot validate the atomic >>>>> behaviors Glue claims, but I wouldn't assert that it is non-compliant >>>>> because of that. >>>>> >>>>> I do think there is a discussion to be had about if/when we might >>>>> adjust the storage/swap requirements, but to reinforce Amogh's point, >>>>> removing those requirements would impact the openness and accessibility of >>>>> Iceberg, which I feel would hamper adoption. >>>>> >>>>> -Dan >>>>> >>>>> >>>>> >>>>> On Thu, Feb 29, 2024 at 1:53 PM Yufei Gu <flyrain...@gmail.com> wrote: >>>>> >>>>>> We've periodically discussed removing the storage requirement and I >>>>>>> think there's a path forward to do that and would agree that >>>>>>> standardizing >>>>>>> on REST, but I wouldn't say the justification for making this push is >>>>>>> that >>>>>>> REST is not compliant so we can just ignore the table spec requirements. >>>>>>> There are a few more things to consider, which is that not >>>>>>> everything can use REST currently and making a hard cut away from file >>>>>>> based metadata could bifurcate access to Iceberg data. There are also >>>>>>> aspects to the spec that reference the metadata paths (like metadata >>>>>>> log, >>>>>>> though it's optional), but would likely need to be addressed. >>>>>> >>>>>> >>>>>> This is a bit off-topic. It makes sense to me to remove the storage >>>>>> requirement moving foward. The metadata.json file isn't necessary in the >>>>>> Rest catalog. For example, the rest catalog may not have the permission >>>>>> to >>>>>> write to the table owner's storage. It still can save it as a file of >>>>>> course, but doesn't quite make sense. Putting it in a key-value store or >>>>>> RDMS could be a better option. >>>>>> >>>>>> Given that we are going to remove the storage requirement. Should we >>>>>> avoid the file path in the current design for things like view spec? A >>>>>> solution like table identifier + version uuid may serve the purpose. >>>>>> >>>>>> Yufei >>>>>> >>>>>> >>>>>> On Thu, Feb 29, 2024 at 1:29 PM Jack Ye <yezhao...@gmail.com> wrote: >>>>>> >>>>>>> > There's no exemption that says if you're using REST you don't need >>>>>>> to follow the spec. Why do you think that's the case? >>>>>>> >>>>>>> In that case are tables in a REST-compliant catalog still an Iceberg >>>>>>> table? I don't think so, because it is a table that only partially >>>>>>> follows >>>>>>> the Iceberg table spec. >>>>>>> >>>>>>> I like the idea about validation for format compliance. But don't >>>>>>> think you can technically validate this. You can validate the static >>>>>>> table >>>>>>> to see if it has all the Iceberg metadata components, but you can not >>>>>>> validate the internal behavior of the service during a commit to see if >>>>>>> it >>>>>>> really atomically swapped a metadata file. >>>>>>> >>>>>>> So I think at minimum we should update the table/view spec to remove >>>>>>> the metadata file swap requirement. The Iceberg table/view spec should >>>>>>> be a >>>>>>> pure format spec that specifies how the file is laid out in storage. >>>>>>> >>>>>>> -Jack >>>>>>> >>>>>>> On Thu, Feb 29, 2024 at 1:22 PM Amogh Jahagirdar <am...@tabular.io> >>>>>>> wrote: >>>>>>> >>>>>>>> I want to echo Dan's point that just because there is a separate >>>>>>>> spec for a REST Catalog does not mean that implementations can deviate >>>>>>>> from >>>>>>>> the spec's definition of the commit protocol or metadata layout, and >>>>>>>> still >>>>>>>> be considered "spec compliant". >>>>>>>> >>>>>>>> > Secondly, once we do that, we should declare REST spec as the >>>>>>>> official catalog spec to interact with Iceberg tables. Otherwise at >>>>>>>> least I >>>>>>>> will be very tempted to just break the atomic pointer swap pattern and >>>>>>>> store the entire metadata using the Glue Table object to achieve much >>>>>>>> better performance and also Glue native feature integrations, and I >>>>>>>> think >>>>>>>> other players will be equally motivated to do something similar. That >>>>>>>> will >>>>>>>> lead to even more chaos in the Iceberg catalog space. >>>>>>>> >>>>>>>> On this, a second point I want to make is around the openness of >>>>>>>> this ecosystem. We all already know that openness (the file formats, >>>>>>>> the >>>>>>>> metadata layout, the spec itself) is a fundamental tenant of the >>>>>>>> project. >>>>>>>> If we take the provided example of removing the metadata JSON file and >>>>>>>> moving it to some other storage, I think that goes against this >>>>>>>> principle >>>>>>>> since a JSON file is quite open by definition. Going back to the first >>>>>>>> point, I think a catalog which has such a behavior would *not* be >>>>>>>> considered spec compliant. Another reason this is important is if we >>>>>>>> think >>>>>>>> about what's healthiest for all users of Iceberg, is to have a healthy >>>>>>>> list >>>>>>>> of options for catalog choices. Storing the metadata JSON in non-open >>>>>>>> ways >>>>>>>> can make users lives harder for trying out new catalogs since now the >>>>>>>> metadata would be stored in their own way, and the users will have a >>>>>>>> harder >>>>>>>> time accessing their own data. >>>>>>>> >>>>>>>> A last point I'd like to make is I think there's a good discussion >>>>>>>> to be had on how do we validate that a REST Catalog implementation is >>>>>>>> spec >>>>>>>> compliant. I think that's really beneficial for the ecosystem as a >>>>>>>> whole. >>>>>>>> Before that, I think first though we'd want to conclude on this topic >>>>>>>> itself. >>>>>>>> >>>>>>>> On Thu, Feb 29, 2024 at 12:29 PM Daniel Weeks < >>>>>>>> daniel.c.we...@gmail.com> wrote: >>>>>>>> >>>>>>>>> > REST spec-compliant catalog does not need to follow the Iceberg >>>>>>>>> spec to commit or store metadata >>>>>>>>> >>>>>>>>> If the REST implementation doesn't follow the Iceberg spec for >>>>>>>>> commit requirements, it's not compliant with the spec. There's no >>>>>>>>> exemption that says if you're using REST you don't need to follow the >>>>>>>>> spec. Why do you think that's the case? >>>>>>>>> >>>>>>>>> I don't believe there's a reason to say that the REST spec needs >>>>>>>>> to enforce the commit requirements either, that's a requirement of the >>>>>>>>> Iceberg spec and still needs to be complied with. >>>>>>>>> >>>>>>>>> -Dan >>>>>>>>> >>>>>>>>> On Thu, Feb 29, 2024 at 12:19 PM Jack Ye <yezhao...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> > The implementation of the spec can either be compliant or not. >>>>>>>>>> >>>>>>>>>> This is exactly the problem we are talking about right? Just to >>>>>>>>>> give an example, we cannot technically say that tables/views in the >>>>>>>>>> Tabular >>>>>>>>>> catalog are Iceberg tables/views, because a REST spec-compliant >>>>>>>>>> catalog >>>>>>>>>> does not need to follow the Iceberg spec to commit or store >>>>>>>>>> metadata. Even >>>>>>>>>> if you say it is, there is no way to really prove that, because the >>>>>>>>>> REST >>>>>>>>>> spec does not enforce it. >>>>>>>>>> >>>>>>>>>> JB, what do you mean by participating on the Catalog RFC? Is >>>>>>>>>> there already an ongoing RFC? >>>>>>>>>> >>>>>>>>>> -Jack >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Feb 29, 2024 at 12:08 PM Jean-Baptiste Onofré < >>>>>>>>>> j...@nanthrax.net> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Dan, >>>>>>>>>>> >>>>>>>>>>> I agree with your statement about REST Spec is not an implement >>>>>>>>>>> but I strongly disagree with your statement "impl of the spec can >>>>>>>>>>> either be >>>>>>>>>>> compliant or not". >>>>>>>>>>> >>>>>>>>>>> The REST Catalog spec impl should be consistent with the REST >>>>>>>>>>> Spec. That's why a reference implementation in Iceberg would be a >>>>>>>>>>> must, >>>>>>>>>>> with a TCK. >>>>>>>>>>> >>>>>>>>>>> The REST Spec should bridge/give access to Table/View metadata. >>>>>>>>>>> I think it would make sense to have a resource to GET the Table/View >>>>>>>>>>> metadata, also supporting PUT to update. >>>>>>>>>>> JSON Schema and eventually JSON RPC could help on some area here >>>>>>>>>>> (compliant with OpenAPI). >>>>>>>>>>> >>>>>>>>>>> In another thread, I propose to work on a Catalog RFC, exactly >>>>>>>>>>> to target this. I think it would make sense to have the >>>>>>>>>>> REST/Catalog RFC as >>>>>>>>>>> the main catalog API, so it has to be both consistent (giving >>>>>>>>>>> access to >>>>>>>>>>> table/view metadata) and extensible (via OpenAPI Extensions for >>>>>>>>>>> instance). >>>>>>>>>>> >>>>>>>>>>> So, I agree with Jack: the minimum would be to have JSON >>>>>>>>>>> metadata exposed by the REST Spec. >>>>>>>>>>> >>>>>>>>>>> @Jack, short term I'm in favor of your proposal, long term, I >>>>>>>>>>> propose to participate on the Catalog RFC (REST Spec). WDYT ? >>>>>>>>>>> >>>>>>>>>>> Thanks ! >>>>>>>>>>> Regards >>>>>>>>>>> JB >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Le jeu. 29 févr. 2024 à 20:47, Daniel Weeks < >>>>>>>>>>> daniel.c.we...@gmail.com> a écrit : >>>>>>>>>>> >>>>>>>>>>>> Hey Jack, >>>>>>>>>>>> >>>>>>>>>>>> I'm not sure I agree with the framing of this argument. The >>>>>>>>>>>> REST Spec defines a protocol, not an implementation. >>>>>>>>>>>> >>>>>>>>>>>> The implementation of the spec can either be compliant or not. >>>>>>>>>>>> So a REST Implementation that adheres to all the requirements >>>>>>>>>>>> (atomic >>>>>>>>>>>> location swap, json representation, etc.), would be compliant. >>>>>>>>>>>> There's no >>>>>>>>>>>> requirement around who performs these operations and with REST, >>>>>>>>>>>> that is >>>>>>>>>>>> delegated to the server. The optional metadata location doesn't >>>>>>>>>>>> mean that >>>>>>>>>>>> there isn't a metadata location, just that it may not be exposed >>>>>>>>>>>> directly >>>>>>>>>>>> in the response. >>>>>>>>>>>> >>>>>>>>>>>> Therefore, an implementation where you just store the table >>>>>>>>>>>> metadata in a Glue Table object, would not be compliant, currently. >>>>>>>>>>>> >>>>>>>>>>>> We've periodically discussed removing the storage requirement >>>>>>>>>>>> and I think there's a path forward to do that and would agree that >>>>>>>>>>>> standardizing on REST, but I wouldn't say the justification for >>>>>>>>>>>> making this >>>>>>>>>>>> push is that REST is not compliant so we can just ignore the table >>>>>>>>>>>> spec >>>>>>>>>>>> requirements. >>>>>>>>>>>> >>>>>>>>>>>> There are a few more things to consider, which is that not >>>>>>>>>>>> everything can use REST currently and making a hard cut away from >>>>>>>>>>>> file >>>>>>>>>>>> based metadata could bifurcate access to Iceberg data. There are >>>>>>>>>>>> also >>>>>>>>>>>> aspects to the spec that reference the metadata paths (like >>>>>>>>>>>> metadata log, >>>>>>>>>>>> though it's optional), but would likely need to be addressed. >>>>>>>>>>>> >>>>>>>>>>>> -Dan >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Feb 29, 2024 at 11:13 AM Jack Ye <yezhao...@gmail.com> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi everyone, >>>>>>>>>>>>> >>>>>>>>>>>>> Just want to pull this specific topic out of the materialized >>>>>>>>>>>>> view discussion thread. I noticed this during the MV discussion, >>>>>>>>>>>>> and I >>>>>>>>>>>>> think it is important to clarify this not just for the MV topic, >>>>>>>>>>>>> but also >>>>>>>>>>>>> for the ongoing discussion to consolidate all the different >>>>>>>>>>>>> catalogs. >>>>>>>>>>>>> >>>>>>>>>>>>> *How the table/view spec defines Iceberg table/view* >>>>>>>>>>>>> >>>>>>>>>>>>> If we look into the table/view spec, the optimistic >>>>>>>>>>>>> concurrency section >>>>>>>>>>>>> <https://iceberg.apache.org/spec/#optimistic-concurrency> >>>>>>>>>>>>> requires the existence of a metadata file, and the atomic swap of >>>>>>>>>>>>> the >>>>>>>>>>>>> metadata file ensures serializable isolation. This implies 2 >>>>>>>>>>>>> things: >>>>>>>>>>>>> 1. the metadata file in a storage that holds the information >>>>>>>>>>>>> described in the rest of the spec. >>>>>>>>>>>>> 2. there is an object in a catalog that holds the pointer of >>>>>>>>>>>>> the metadata file. What object and what catalog is implementation >>>>>>>>>>>>> dependent, but these generalized concepts are always intact. >>>>>>>>>>>>> >>>>>>>>>>>>> The JSON serialization parts of the spec plus the reader >>>>>>>>>>>>> requirements also implies that the metadata file is in JSON >>>>>>>>>>>>> format. >>>>>>>>>>>>> >>>>>>>>>>>>> So when we talk about an Iceberg table/view that is compliant >>>>>>>>>>>>> with the spec, it is the combination of all these 5 requirements: >>>>>>>>>>>>> 1. there is an object in the catalog representing this >>>>>>>>>>>>> table/view >>>>>>>>>>>>> 2. there is a pointer to a JSON metadata file in the object >>>>>>>>>>>>> 3. the JSON metadata file exists in storage and contains the >>>>>>>>>>>>> table/view metadata content >>>>>>>>>>>>> 4. the metadata content is compliant with the standard >>>>>>>>>>>>> described in the spec >>>>>>>>>>>>> 5. serializable isolation is achieved by atomic swap of the >>>>>>>>>>>>> object pointer >>>>>>>>>>>>> >>>>>>>>>>>>> *How non-REST catalogs are compliant with the table/view spec* >>>>>>>>>>>>> >>>>>>>>>>>>> An implementation of the Iceberg table/view is essentially >>>>>>>>>>>>> specifying: >>>>>>>>>>>>> 1. what is the exact implementation of the catalog, e.g. JDBC, >>>>>>>>>>>>> Hive metastore (HMS), Glue, etc. >>>>>>>>>>>>> 2. what is the object that represents a table, e.g. a row in >>>>>>>>>>>>> the "iceberg_tables" table in JDBC, a Table object in HMS/Glue, >>>>>>>>>>>>> etc. >>>>>>>>>>>>> 3. how is the JSON metadata file pointer stored, e.g. a column >>>>>>>>>>>>> in the table's row in JDBC, metadata_location key in the Table's >>>>>>>>>>>>> parameter >>>>>>>>>>>>> map in HMS/Glue, etc. >>>>>>>>>>>>> 4. how the atomic swap is implemented, e.g. SQL atomic update >>>>>>>>>>>>> in JDBC, conditional parameter update in HMS, conditional version >>>>>>>>>>>>> update in >>>>>>>>>>>>> Glue, etc. >>>>>>>>>>>>> >>>>>>>>>>>>> *How the REST spec is NOT compliant with the table/view spec* >>>>>>>>>>>>> >>>>>>>>>>>>> The REST spec technically does not match the following >>>>>>>>>>>>> table/view spec requirements: >>>>>>>>>>>>> 2. there is a pointer to a JSON metadata file in the object >>>>>>>>>>>>> 3. the JSON metadata file exists in storage and contains the >>>>>>>>>>>>> table/view metadata content >>>>>>>>>>>>> 5. serializable isolation is achieved by atomic swap of the >>>>>>>>>>>>> object pointer >>>>>>>>>>>>> >>>>>>>>>>>>> The key parts in REST spec that are not compliant are: >>>>>>>>>>>>> 1. metadata-location field is optional in LoadTableResponse >>>>>>>>>>>>> <https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L2721-L2728> >>>>>>>>>>>>> 2. pointer swap is not enforced in the UpdateTable >>>>>>>>>>>>> <https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L658> >>>>>>>>>>>>> operation >>>>>>>>>>>>> >>>>>>>>>>>>> Therefore, it opens the door for a REST service to be >>>>>>>>>>>>> completely not dependent on a JSON metadata file, store the >>>>>>>>>>>>> Iceberg >>>>>>>>>>>>> table/view metadata not as a file, and achieve much better >>>>>>>>>>>>> performance >>>>>>>>>>>>> characteristics than other catalogs. This technically gives a >>>>>>>>>>>>> unique >>>>>>>>>>>>> advantage for REST catalog adopters that is not there for >>>>>>>>>>>>> non-REST catalogs >>>>>>>>>>>>> like HMS and Glue. >>>>>>>>>>>>> >>>>>>>>>>>>> *How can we fix this?* >>>>>>>>>>>>> >>>>>>>>>>>>> I suggest the following: >>>>>>>>>>>>> >>>>>>>>>>>>> Firstly, I think it is good that we try to remove the >>>>>>>>>>>>> requirements of JSON metadata file pointer and atomic pointer >>>>>>>>>>>>> swap. We know >>>>>>>>>>>>> these requirements have perf limitations based on production >>>>>>>>>>>>> usage, >>>>>>>>>>>>> especially when the metadata file is large. If that is the >>>>>>>>>>>>> direction, we >>>>>>>>>>>>> should make it official by changing the table/view spec to say >>>>>>>>>>>>> that those >>>>>>>>>>>>> requirements are catalog level implementation details that are no >>>>>>>>>>>>> longer >>>>>>>>>>>>> required. >>>>>>>>>>>>> >>>>>>>>>>>>> Secondly, once we do that, we should declare REST spec as the >>>>>>>>>>>>> official catalog spec to interact with Iceberg tables. Otherwise >>>>>>>>>>>>> at least I >>>>>>>>>>>>> will be very tempted to just break the atomic pointer swap >>>>>>>>>>>>> pattern and >>>>>>>>>>>>> store the entire metadata using the Glue Table object to achieve >>>>>>>>>>>>> much >>>>>>>>>>>>> better performance and also Glue native feature integrations, and >>>>>>>>>>>>> I think >>>>>>>>>>>>> other players will be equally motivated to do something similar. >>>>>>>>>>>>> That will >>>>>>>>>>>>> lead to even more chaos in the Iceberg catalog space. >>>>>>>>>>>>> >>>>>>>>>>>>> With REST spec as the official catalog spec, we can actually >>>>>>>>>>>>> support non-REST catalogs by using the HTTP execution chain >>>>>>>>>>>>> handler. Dan >>>>>>>>>>>>> has already done a prototype here >>>>>>>>>>>>> <https://github.com/apache/iceberg/commit/619127ff69f89e43a1edef2ea94c3dd439396a8d#diff-869264a83ba9ca657e7defefaa16ad196b0de9fce6c87f97533db77f29e44762> >>>>>>>>>>>>> that is based on this discussion >>>>>>>>>>>>> <https://github.com/apache/iceberg/pull/8091#issuecomment-1647189146> >>>>>>>>>>>>> in the past about using AWS Lambda as an alternative HTTP client >>>>>>>>>>>>> for REST >>>>>>>>>>>>> catalog. The same approach can be used to talk to >>>>>>>>>>>>> HMS/Glue/JDBC/... while >>>>>>>>>>>>> users will only interact with the RESTCatalog as the entry point. >>>>>>>>>>>>> >>>>>>>>>>>>> I think this can provide a good path forward overall for the >>>>>>>>>>>>> catalog consolidation story, interested to know what others think. >>>>>>>>>>>>> >>>>>>>>>>>>> Best, >>>>>>>>>>>>> Jack Ye >>>>>>>>>>>>> >>>>>>>>>>>>> >> >> -- >> Ryan Blue >> Tabular >> > > > -- > Ryan Blue > Tabular >
