Re: [DISCUSS] Describing REST Server capabilities

@Walaa: Those are all good feedback points that are appreciated. I realized that the capabilities probably add more complexity than necessary. As you mentioned, in the end we really want to know what endpoints a server supports. I've created a new design doc and opened a separate discussion thread

Re: [DISCUSS] Describing REST Server capabilities

Catching up here. >From Eduard's doc [1], it seems that at the end of the day, the capability boils down to whether an end point is implemented by the server or not. Therefore, I feel we could simplify things by skipping the categorization/grouping (e.g., tables, views, udfs, etc) and just allow s

Re: [DISCUSS] Describing REST Server capabilities

> endpoint version should bump (e.g. GET /v1/namespaces to GET /v2/namespaces) when there is a significant backwards incompatible change. That makes sense to me too. > (2) version the entire catalog spec. A released catalog spec version will contain a list of configs it supports, and also a set o

Re: [DISCUSS] Describing REST Server capabilities

One thing to clarify, regarding per-endpoint versioning, my understanding is that endpoint version should bump (e.g. GET /v1/namespaces to GET /v2/namespaces) when there is a significant backwards incompatible change. -Jack On Tue, Jul 30, 2024 at 7:56 PM Jack Ye wrote: > > are you talking abou

Re: [DISCUSS] Describing REST Server capabilities

> are you talking about the endpoint path like "/v1/" No, I mean in general the catalog spec version, which is marked currently as 0.0.1: https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml#L27 And in my mind it would be you periodically release a version of the catalo

Re: [DISCUSS] Describing REST Server capabilities

> (2) version the entire catalog spec. A released catalog spec version will contain a list of configs it supports, and also a set of APIs and all features embedded in the APIs. A server will report the specific catalog version it adheres to, and then document the nuances. Jack, just to clarify, a

Re: [DISCUSS] Describing REST Server capabilities

Since the catalog sync was canceled this week, I find maybe it is better to reply here for my latest take on this topic. I think we have 2 discussions intertwined here, that I would like to decouple if possible. (1) is it worth having a concept of capabilities to control client behaviors? (2) sup

Re: [DISCUSS] Describing REST Server capabilities

Hey everyone, I've written up https://docs.google.com/document/d/1F1xh6SJhS-opgWRe1pPvWh01j8VHNHRocfCCFttKNf0/edit to provide an easier way of giving feedback to the proposal. Please take a look so that we can discuss how we'd like to handle the default fallback behavior (*tables* vs *everything t

Re: [DISCUSS] Describing REST Server capabilities

So I would argue to define the current set of APIs and specs as the default > if the `capabilities` field is missing. There have been two sides to this in prior discussions. Having *tables* as the default vs having what's *currently in the spec* as the default. The argument for having *tables* as

Re: [DISCUSS] Describing REST Server capabilities

On 15.07.24 16:10, Eduard Tudenhöfner wrote: Current servers do not send a `capabilities` field at all. You're suggesting to use a new `rest-default-capabilities` property to let newer clients assume `1`.  Once the table/view/etc-spec capabilities are needed, those newer clients

Re: [DISCUSS] Describing REST Server capabilities

> > Current servers do not send a `capabilities` field at all. You're > suggesting to use a new `rest-default-capabilities` property to let newer > clients assume `1`. Once the table/view/etc-spec capabilities are needed, > those newer clients would assume table-spec v1. That's wrong IMO. That s

Re: [DISCUSS] Describing REST Server capabilities

Sorry, I don't understand the two suggestions, especially when used in combination. Current servers do not send a `capabilities` field at all. You're suggesting to use a new `rest-default-capabilities` property to let newer clients assume `1`.  Once the table/view/etc-spec capabilities are need

Re: [DISCUSS] Describing REST Server capabilities

I would suggest adding *table-spec / view-spec / udf-spec *capabilities later when new requirements/updates get added. The current implementation wouldn't make any use of these capabilities, so I don't see a good enough reason to add them at this point. The PR currently says: "tables -> default ca

Re: [DISCUSS] Describing REST Server capabilities

Hi, I still have concerns regarding the missing table-spec/view-spec capabilities. Newer clients can send create/update requests with requirements/updates of newer Iceberg table/view/udf specs to a server that doesn't support those spec versions - the outcome is rather undefined. What should

Re: [DISCUSS] Describing REST Server capabilities

Let's remove the *remote-signing* capability for now and go with *tables / views / multi-table-commit*. As I mentioned earlier, we can always add it when there's a clear benefit. Eduard On Fri, Jul 12, 2024 at 5:09 PM Dmitri Bourlatchkov wrote: > After more thinking about the "remote signing" c

Re: [DISCUSS] Describing REST Server capabilities

After more thinking about the "remote signing" capability flag, I am still not sure it is actually useful for making decisions on the client side. Granted, the client may have s3.remote-signing-enabled=true set independently of the server and then use the remote signing call paths. However, in thi

Re: [DISCUSS] Describing REST Server capabilities

> > But why does this logic not apply to per-endpoint versioning? Isn't it > also nice to just fail at client side instead of calling server and getting > a "generic 501"? Yes of course that would be nice, but that would be at the cost of having finer-grained capabilities which we want to avoid b

Re: [DISCUSS] Describing REST Server capabilities

> > Are there any other concerns with the proposal or should we start a VOTE > thread? We should summarize the consensus since the thread is quite long and then check if anyone has additional points to add before we proceed to voting. - Ajantha On Fri, Jul 12, 2024 at 9:09 AM Jack Ye wrote:

Re: [DISCUSS] Describing REST Server capabilities

> While a server can control this behavior and instruct the client to use remote signing, technically nothing is preventing a client from configuring s3.remote-signing-enabled=true. In such a case it seems more appropriate to indicate that this capability isn't supported rather than a generic 501,

Re: [DISCUSS] Describing REST Server capabilities

Sorry I will take a look at the new comments later today. -Jack On Wed, Jul 10, 2024, 11:42 PM Eduard Tudenhöfner wrote: > Are there any other concerns with the proposal or should we start a VOTE > thread? > > Eduard > > On Wed, Jul 10, 2024 at 5:20 PM Dmitri Bourlatchkov > wrote: > >> Re: rem

Re: [DISCUSS] Describing REST Server capabilities

Are there any other concerns with the proposal or should we start a VOTE thread? Eduard On Wed, Jul 10, 2024 at 5:20 PM Dmitri Bourlatchkov wrote: > Re: remote signing, I agree that it does not look like a server capability >> that a client can / should discover. It is more like something that

Re: [DISCUSS] Describing REST Server capabilities

Re: remote signing, I agree that it does not look like a server capability > that a client can / should discover. It is more like something that the > server instructs / configures the client to do. While a server can control this behavior and instruct the client to use remote signing, technicall

Re: [DISCUSS] Describing REST Server capabilities

Hey everyone, I've added a few inline comments below. > Re: remote signing, I agree that it does not look like a server capability > that a client can / should discover. It is more like something that the > server instructs / configures the client to do. While a server can control this behavi

Re: [DISCUSS] Describing REST Server capabilities

Yes I agree that sounds like a valid use case. So the criteria so far is that capabilities are used for: - controlling client-side fallback behavior - failing expensive operations early if we know it will eventually fail due to missing capability Do we agree if this is the criteria we should use?

Re: [DISCUSS] Describing REST Server capabilities

> does it make a difference if I declare the capability or not? I think that it does in other cases. Multi-table commits, for example, are a building block for multi-statement transactions. If a service doesn't support multi-table commits then we ideally want clients to know that ahead of time so

Re: [DISCUSS] Describing REST Server capabilities

Re: remote signing, I agree that it does not look like a server capability that a client can / should discover. It is more like something that the server instructs / configures the client to do. Cheers, Dmitri. On Tue, Jul 9, 2024 at 12:05 PM Jack Ye wrote: > I was reconciling the discussion ye

Re: [DISCUSS] Describing REST Server capabilities

I was reconciling the discussion yesterday, one point that was interesting to me was that we agreed the purpose of these capabilities is to "control client-side fallback behavior", or at least the client should behave differently based on these capabilities. However, this seems to be only needed so

Re: [DISCUSS] Describing REST Server capabilities

Hi Eduard, > I've also added the 501 error to the response of the respective endpoints but worth mentioning that *HEAD* / *GET *requests must not return a 501 (this implies that the server impl would e.g. return a *404* in such a case)

Re: [DISCUSS] Describing REST Server capabilities

Hey everyone, I watched the catalog sync recording today and updated the PR to remove fine-grained capabilities like *register-table / table-metrics*. The current capabilities (with versioning information) in the PR are: - tables - views - r

Re: [DISCUSS] Describing REST Server capabilities

Hi Eduard, It makes sense to return 501 for servers which don't implement all endpoints. It means that the server will at least have to implement empty endpoints if needed (that makes sense to me). I think we should focus on only "identified capabilities". I think that I proposed before that the

Re: [DISCUSS] Describing REST Server capabilities

On 04.07.24 10:32, Eduard Tudenhöfner wrote: For servers that only *partially* implement endpoints under a capability the spec requires the server to throw a *501 Not Implemented*. This was suggested by Jack and it seems reasonable to do that. SGTM Regarding the inclusion of table-spec / *

Re: [DISCUSS] Describing REST Server capabilities

I have clarified the wording in #9940 around the requirement on having to implement all endpoints under a particular capability. For servers that only *partially* implement endpoints under a capability the spec requires the server to throw a *501 Not I

Re: [DISCUSS] Describing REST Server capabilities

> > Spec is an interesting topic we did not discuss. Robert, how do you > envision this to be used? > In my mind, if a new table format v3 is launched, there are 2 approaches > we can go with, taking CreateTable as an example: > (1) increment the related operation version, which means that POST > /

Re: [DISCUSS] Describing REST Server capabilities

The opposite is, as I expressed, that a client would _not_ use functionality that worked before. In other words: the capabilities-change would become a _breaking_ change - not backwards compatible. On 02.07.24 15:02, Eduard Tudenhöfner wrote: I couldn't make it to the catalog sync meeting y

Re: [DISCUSS] Describing REST Server capabilities

I think that we really need to let a service express which table/view/udf/etc-spec versions it supports. Otherwise, as you noted, a client would only receive some error. If a "table-spec v4" aware client knows that the service just knows about "table-spec v2", it can just use that version - pre

Re: [DISCUSS] Describing REST Server capabilities

Spec is an interesting topic we did not discuss. Robert, how do you envision this to be used? In my mind, if a new table format v3 is launched, there are 2 approaches we can go with, taking CreateTable as an example: (1) increment the related operation version, which means that POST /v2/{prefix}/

Re: [DISCUSS] Describing REST Server capabilities

I couldn't make it to the catalog sync meeting yesterday but I watched the recording today (thanks for providing that). The missing piece is how (new, capabilities-aware) clients handle the case > when a service does _not_ return the capabilities field (absent). My > proposal would be that a clien

Re: [DISCUSS] Describing REST Server capabilities

We also talked briefly about whether clients/server must fail on unknown attributes/values or not. Please correct me if I'm wrong, but I think that the consensus in the meeting yesterday was clients must fail. (more comments inline below) On 02.07.24 04:44, Jack Ye wrote: Each operation will

Re: [DISCUSS] Describing REST Server capabilities

Let me try to summarize what my understanding is after the sync, and we can see if we agree: Each operation will be versioned separately, starting at v1 which is the current version on the spec for all operations. When there are significant changes to the request response model, or highly backward

Re: [DISCUSS] Describing REST Server capabilities

Ignore my previous email - fat thumbed... Here is the full version: I think most of us agree that the server should announce its exact capabilities, so the clients don't need to guess. The debate is around how granular this definition should be. If we do it on service level, then the client need

Re: [DISCUSS] Describing REST Server capabilities

I think most of us agree that the server should announce its exact capabilities, so the clients don't need to guess. The debate is around how granular this definition should be. If we do it on service level, then the client needs to examine each and every service it is using whether it has the spe

Re: [DISCUSS] Describing REST Server capabilities

IMO that would be a list of "capability" to "set of versions" tuples. The reason to have a "set of (integer) version" is that you have to plan for the future, now. I also think we do need "logical" capabilities to express for example which table/view/etc specs a service supports and to express

Re: [DISCUSS] Describing REST Server capabilities

On 27.06.24 19:05, Micah Kornfield wrote: Maybe it pays to prototype the individual end point approach to demonstrate its relative complexity? The math is pretty simple: you need to duplicate all endpoints, all request/response schema types, all tests, duplicate and/or adopt client code. No

Re: [DISCUSS] Describing REST Server capabilities

> > If it means returning a response with 20-30ish hard-coded entries, and the > client is configured based on that, that seems totally reasonable to me. It reads to me that a lot of the debate is around the complexity of one approach for the other. Maybe it pays to prototype the individual end

Re: [DISCUSS] Describing REST Server capabilities

I feel Alex is already tapping into the more complex territory I do not want to go into, because as he says, a "capability" is logical, and it can be a set of overlapping endpoints, small features in some endpoints, etc. We already saw that in the original PR we tried to say "pagination" is a capab

Re: [DISCUSS] Describing REST Server capabilities

Hi all, So far we've been thinking of capabilities as equivalent to a set of endpoints. That's a rather technical definition. It also brings one important limitation: one endpoint can only be "governed" by one capability. Granted, most capabilities do require implementing specific endpoints. But

Re: [DISCUSS] Describing REST Server capabilities

Hi Jack I like Robert's proposal. Back to the topics, I think grouping with tags is more "flexible" (it was what we included in the REST spec proposal as well). Regards JB On Wed, Jun 26, 2024 at 6:26 PM Jack Ye wrote: > > It seems like there are 2 sub-topics here: > 1. should we group operatio

Re: [DISCUSS] Describing REST Server capabilities

IMO a capability is a coarse-grained way of describing that a catalog supports X, Y, Z. In order to support X it needs to implement the particular endpoints under X, otherwise it doesn't fully support X. >From a user's perspective this makes it easy to understand whether a catalog fully supports e

Re: [DISCUSS] Describing REST Server capabilities

I don't have a very strong opinion in the groups vs. single services debate, but I lean towards grouping, as that makes the result of the service human readable too. I expect that small/incomplete services will be the exception in the long run, and they can highlight the implemented services in the

Re: [DISCUSS] Describing REST Server capabilities

> evaluate a REST service and if it's good for their use case Feels like you are talking from the perspective of people choosing a vendor product. I believe most vendors will offer near-full capability. But I am coming from an angle of small organizations that are building REST servers just for op

Re: [DISCUSS] Describing REST Server capabilities

I'm in favor of grouping by tags. The way I look at this, there are 2 primary considerations: 1.) The client/server protocol complexity tradeoffs. On the first consideration, unless I'm missing something the client side becomes significantly more complex; if this has been sketched out earlier i

Re: [DISCUSS] Describing REST Server capabilities

It seems like there are 2 sub-topics here: 1. should we group operations with tags, or should we do this per-operation/endpoint? 2. how should we do the capability/versioning for each unit (either per tag or per operation) Shall we first conclude on 1? For 1, my take is that we will need to do it

Re: [DISCUSS] Describing REST Server capabilities

I think Robert's approach is a reasonable compromise here. If we wanted a "per operation/endpoint" versioning, I think I'd prefer Micah's OpenAPI spec based approach because it's more standardized, but I feel adds a lot of client complexity. -Dan On Wed, Jun 26, 2024 at 6:59 AM Robert Stupp w

Re: [DISCUSS] Describing REST Server capabilities

(I think, compatibility deserves a separate thread - it's a "huge" topic) Based on experience, we decided on the following with Nessie: * Unknown fields/attributes in a structure _DO_ cause (de)serialization failures. * "Stable API versions" - endpoint additions and/or added query parame

Re: [DISCUSS] Describing REST Server capabilities

Hi everyone, A few considerations: - I think we should explicitly state which client/service interoperability we are aiming for. I expect that we want to support both old client -> new server, and new client -> old server communications. - I agree with Jack, that we should think about versions in

Re: [DISCUSS] Describing REST Server capabilities

Hi everyone, I feel I do not see a good answer to why not just simply version each API? When using tag, it means I have to offer capabilities per-tagged group. However, I could for example just offer loadTable and nothing else in a catalog, and that should still be Iceberg REST compliant. And I th

Re: [DISCUSS] Describing REST Server capabilities

I don't have strong opinions either way here, just thought it was worth raising some concerns over possible evolution here. Some responses inline, but if capabilities seem to meet the requirement at hand, then it does potentially seem the simplest mechanism. I think we also want to avoid relyanc

Re: [DISCUSS] Describing REST Server capabilities

Hey Micah, I think what we're trying to achieve is strike a balance between client complexity and ability to support multiple server-side capabilities. One challenge we've run into is if a client performs an operation (e.g. listViews), but receives a 403 code, it's not clear whether the client do

Re: [DISCUSS] Describing REST Server capabilities

We had a separate discussion with Dan on the *oauth2* flag last week and came to the same conclusion that removing the *oauth2* capability is probably the best for now. This is mainly because we can't really act on the *oauth2* capability right now, because the */tokens* endpoint is called before w

Re: [DISCUSS] Describing REST Server capabilities

Let's remove the oauth2 tag for now until we figure out how to move forward there. That makes sense to me. On Fri, Jun 21, 2024 at 9:30 AM Dmitri Bourlatchkov wrote: > Hi Eduard, > > The capabilities PR looks good to me overall. I have a concern with the > "oauth2" tag name though. > > I also co

Re: [DISCUSS] Describing REST Server capabilities

Hi Eduard, The capabilities PR looks good to me overall. I have a concern with the "oauth2" tag name though. I also commented [1] in GH but the comment appears to be closed by default :) I believe the term "oauth2" is confusing in this context with respect to RFC 6749 [2] as discussed in depth o

Re: [DISCUSS] Describing REST Server capabilities

Hi Ryan, I think I wasn't clear (sorry about that): by "catalog-level-versioning" capability, I don't mean to actually define any specific version, it's more to indicate to the client how the catalog behaving in terms of versioning (per table/views or global to catalog). It's a "pure" capability i

Re: [DISCUSS] Describing REST Server capabilities

> > The general idea behind a capability is that if e.g. a server supports > *views*, then that server must implement all endpoints grouped under that > capability. I haven't thought deeply about this, but is there a reason to be prescriptive about this by grouping endpoints in capabilities? Ano

Re: [DISCUSS] Describing REST Server capabilities

I think the capabilities proposal is intended to let people build in a different way than a versioning system would. It's probably valuable to think through the differences between the approaches. The capabilities that are proposed let catalogs declare sets of features that are supported, like sup

Re: [DISCUSS] Describing REST Server capabilities

Hi Eduard, That makes sense. Thanks. Maybe we can already anticipate a little and add a "catalog-level versioning" capability as it's a feature supported by Nessie catalog for instance ? We can also imagine a more generic capability like "version scope". Regards JB On Thu, Jun 20, 2024 at 3:47 

Re: [DISCUSS] Describing REST Server capabilities

Hey JB, If adding UDFs would require adding new endpoints, then you'd also add a *udf *capability when adding UDF support to the REST catalog. That way a client knows whether it's safe to call the UDF endpoints on a given server. Eduard On Thu, Jun 20, 2024 at 1:59 PM Jean-Baptiste Onofré wrote

Re: [DISCUSS] Describing REST Server capabilities

Hi Eduard It looks good to me. I have a question however :) Later, Imagine, we add UDF support in Iceberg. Does it mean that you will need to update REST Spec (ConfigResponse/capabilities) to add this capability ? For consistency, I think it makes sense as I don't think we often add new capabilit

[DISCUSS] Describing REST Server capabilities

Hey everyone, I'd like to bring up the discussion around describing REST server capabilities via the */config* endpoint. There is PR #9940 that describes the OpenAPI spec changes. Mainly we'd like to have a *capabilities* field in the *ConfigResponse*