Hi Dan, I agree with your statement about REST Spec is not an implement but I strongly disagree with your statement "impl of the spec can either be compliant or not".
The REST Catalog spec impl should be consistent with the REST Spec. That's why a reference implementation in Iceberg would be a must, with a TCK. The REST Spec should bridge/give access to Table/View metadata. I think it would make sense to have a resource to GET the Table/View metadata, also supporting PUT to update. JSON Schema and eventually JSON RPC could help on some area here (compliant with OpenAPI). In another thread, I propose to work on a Catalog RFC, exactly to target this. I think it would make sense to have the REST/Catalog RFC as the main catalog API, so it has to be both consistent (giving access to table/view metadata) and extensible (via OpenAPI Extensions for instance). So, I agree with Jack: the minimum would be to have JSON metadata exposed by the REST Spec. @Jack, short term I'm in favor of your proposal, long term, I propose to participate on the Catalog RFC (REST Spec). WDYT ? Thanks ! Regards JB Le jeu. 29 févr. 2024 à 20:47, Daniel Weeks <daniel.c.we...@gmail.com> a écrit : > Hey Jack, > > I'm not sure I agree with the framing of this argument. The REST Spec > defines a protocol, not an implementation. > > The implementation of the spec can either be compliant or not. So a REST > Implementation that adheres to all the requirements (atomic location swap, > json representation, etc.), would be compliant. There's no requirement > around who performs these operations and with REST, that is delegated to > the server. The optional metadata location doesn't mean that there isn't a > metadata location, just that it may not be exposed directly in the response. > > Therefore, an implementation where you just store the table metadata in a > Glue Table object, would not be compliant, currently. > > We've periodically discussed removing the storage requirement and I think > there's a path forward to do that and would agree that standardizing on > REST, but I wouldn't say the justification for making this push is that > REST is not compliant so we can just ignore the table spec requirements. > > There are a few more things to consider, which is that not everything can > use REST currently and making a hard cut away from file based metadata > could bifurcate access to Iceberg data. There are also aspects to the spec > that reference the metadata paths (like metadata log, though it's > optional), but would likely need to be addressed. > > -Dan > > > > On Thu, Feb 29, 2024 at 11:13 AM Jack Ye <yezhao...@gmail.com> wrote: > >> Hi everyone, >> >> Just want to pull this specific topic out of the materialized view >> discussion thread. I noticed this during the MV discussion, and I think it >> is important to clarify this not just for the MV topic, but also for the >> ongoing discussion to consolidate all the different catalogs. >> >> *How the table/view spec defines Iceberg table/view* >> >> If we look into the table/view spec, the optimistic concurrency section >> <https://iceberg.apache.org/spec/#optimistic-concurrency> requires the >> existence of a metadata file, and the atomic swap of the metadata file >> ensures serializable isolation. This implies 2 things: >> 1. the metadata file in a storage that holds the information described in >> the rest of the spec. >> 2. there is an object in a catalog that holds the pointer of the metadata >> file. What object and what catalog is implementation dependent, but these >> generalized concepts are always intact. >> >> The JSON serialization parts of the spec plus the reader requirements >> also implies that the metadata file is in JSON format. >> >> So when we talk about an Iceberg table/view that is compliant with the >> spec, it is the combination of all these 5 requirements: >> 1. there is an object in the catalog representing this table/view >> 2. there is a pointer to a JSON metadata file in the object >> 3. the JSON metadata file exists in storage and contains the table/view >> metadata content >> 4. the metadata content is compliant with the standard described in the >> spec >> 5. serializable isolation is achieved by atomic swap of the object pointer >> >> *How non-REST catalogs are compliant with the table/view spec* >> >> An implementation of the Iceberg table/view is essentially specifying: >> 1. what is the exact implementation of the catalog, e.g. JDBC, Hive >> metastore (HMS), Glue, etc. >> 2. what is the object that represents a table, e.g. a row in the >> "iceberg_tables" table in JDBC, a Table object in HMS/Glue, etc. >> 3. how is the JSON metadata file pointer stored, e.g. a column in the >> table's row in JDBC, metadata_location key in the Table's parameter map in >> HMS/Glue, etc. >> 4. how the atomic swap is implemented, e.g. SQL atomic update in JDBC, >> conditional parameter update in HMS, conditional version update in Glue, >> etc. >> >> *How the REST spec is NOT compliant with the table/view spec* >> >> The REST spec technically does not match the following table/view spec >> requirements: >> 2. there is a pointer to a JSON metadata file in the object >> 3. the JSON metadata file exists in storage and contains the table/view >> metadata content >> 5. serializable isolation is achieved by atomic swap of the object pointer >> >> The key parts in REST spec that are not compliant are: >> 1. metadata-location field is optional in LoadTableResponse >> <https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L2721-L2728> >> 2. pointer swap is not enforced in the UpdateTable >> <https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L658> >> operation >> >> Therefore, it opens the door for a REST service to be completely not >> dependent on a JSON metadata file, store the Iceberg table/view metadata >> not as a file, and achieve much better performance characteristics than >> other catalogs. This technically gives a unique advantage for REST catalog >> adopters that is not there for non-REST catalogs like HMS and Glue. >> >> *How can we fix this?* >> >> I suggest the following: >> >> Firstly, I think it is good that we try to remove the requirements of >> JSON metadata file pointer and atomic pointer swap. We know these >> requirements have perf limitations based on production usage, especially >> when the metadata file is large. If that is the direction, we should make >> it official by changing the table/view spec to say that those requirements >> are catalog level implementation details that are no longer required. >> >> Secondly, once we do that, we should declare REST spec as the official >> catalog spec to interact with Iceberg tables. Otherwise at least I will be >> very tempted to just break the atomic pointer swap pattern and store the >> entire metadata using the Glue Table object to achieve much better >> performance and also Glue native feature integrations, and I think other >> players will be equally motivated to do something similar. That will lead >> to even more chaos in the Iceberg catalog space. >> >> With REST spec as the official catalog spec, we can actually support >> non-REST catalogs by using the HTTP execution chain handler. Dan has >> already done a prototype here >> <https://github.com/apache/iceberg/commit/619127ff69f89e43a1edef2ea94c3dd439396a8d#diff-869264a83ba9ca657e7defefaa16ad196b0de9fce6c87f97533db77f29e44762> >> that is based on this discussion >> <https://github.com/apache/iceberg/pull/8091#issuecomment-1647189146> in >> the past about using AWS Lambda as an alternative HTTP client for REST >> catalog. The same approach can be used to talk to HMS/Glue/JDBC/... while >> users will only interact with the RESTCatalog as the entry point. >> >> I think this can provide a good path forward overall for the catalog >> consolidation story, interested to know what others think. >> >> Best, >> Jack Ye >> >>
