Ugh, I tried to send this on Saturday but ended up having it blocked because I used my personal gmail. Here's my original email. I think I'm saying much the same thing as Dan.
I was talking to Dan about the REST capabilities header earlier this week and he convinced me that one problem with the generic header is the association with endpoints. If we say that the "read-restrictions" capability requires certain behavior of the load table endpoint, then what happens if we later realize that other endpoints should also participate? We would need additional headers and this is the key idea behind Dan's point that we would need to associate capabilities with specific endpoints -- and there's not a great way to document that. In that discussion, I had an idea that I think might be a good third option. Adding a capability is a way for clients to opt into new forward-breaking changes. But we already have a way to introduce forward-breaking changes: versioned endpoints. With a new endpoint, clients are required to understand payloads and behave certain ways. If we introduced a v2 loadTable endpoint, we could add read restrictions and mandate that clients must fail if there are unsupported restrictions. There are other reasons to introduce a v2 endpoint as well: we are making table location optional in v4, we have discussed making snapshots optional (an empty list means an empty table!), and we could default the snapshot loading behavior to refs. I think we should seriously consider adding a v2 endpoint since that's the way that we already have to introduce new required functionality. Then we don't have to worry about either too many custom headers or associating values from a generic header with endpoints and behavior. I think this is much cleaner than either head-based solution. On Thu, Jun 4, 2026 at 8:45 AM Daniel Weeks <[email protected]> wrote: > Thanks Prashant, > > I added some comments to the various PRs, but wanted to follow up here as > well. > > I have several concerns regarding the complexity this approach would > introduce. While this may seem simple initially, a number of artifacts > emerge when you start to dig into it: > - Capabilities are not just what the client supports, it's what the engine > supports and how the environment has been configured in some cases (like > read restrictions) > - A global header means that if associated behaviors apply only to a > subset of endpoints, it's not clear where the behavior is targeted > - Evolution is already being discussed through the addition of versioning > to the labels > - This can lead to a proliferation of capabilities that have behaviors > orthogonal (or at least nuanced variations) to what is defined in the > OpenAPI spec > > I feel like what we're actually doing is trying to work around introducing > new versioned endpoints (e.g. v2 loadTable). We already have versioning in > the path, but we have yet to justify introducing a new version. Now we're > taking extreme steps and introducing new features to continue down that > path. > > I think it's worth looking at what it would take to introduce a v2 > loadTable endpoint and correct for a number of issues with the original > endpoint (creds duplicated in properties, required metadata location) and > introduce new response objects so that it's clear the calling client > clearly understands the endpoint. > > I'm not fundamentally opposed to client capabilities at this point, but it > feels like we're creating complexity that isn't narrowly targeted. > > -Dan > > On Wed, May 27, 2026 at 11:50 PM Prashant Singh <[email protected]> > wrote: > >> Hi Dan, >> >> Wanted to follow up specifically on your concerns since we re-discussed >> this at the 05/26 sync (you weren't there for >> the final stretch). Recording: >> https://www.youtube.com/watch?v=-KEesN1udyY >> >> *On the per-endpoint vs. generic header pattern*: the room leaned >> generic, mostly on the argument that >> X-Iceberg-Access-Delegation is doing a different job - it's a >> per-request directive ("vend credentials for this >> call"), not a static declaration of what the client understands. >> Modeling capabilities the same way conflates two >> patterns: one is runtime preference, the other is a build-time fact >> about the client. >> >> *On the OpenAPI tooling concern (the "spec within a spec" point **from >> sync)*: the values are still a constrained enum in the >> OpenAPI schema - same shape as X-Iceberg-Access-Delegation. Generated >> clients validate values; servers get a closed >> list; adding a capability is a one-line enum edit. There's no parallel >> registry - it's a normal header whose schema >> happens to be enum-typed. >> >> * On bundling all capabilities per request*: Russell raised the same - >> could a client send only the relevant subset per >> endpoint? The sync's view was that since capabilities don't change >> between calls, per-endpoint subsetting adds client >> bookkeeping without much benefit. Open to revisiting if you have a use >> case in mind where conditional advertisement >> matters (e.g., a client that wants to suppress a capability it >> technically supports). >> >> Looking forward to hearing from you. >> >> Thanks, >> Prashant >> >> On Mon, May 25, 2026 at 12:47 PM Prashant Singh <[email protected]> >> wrote: >> >>> Thanks everyone for the feedback. Apologies for the delayed response >>> on the >>> list, I wasn't receiving mailing list emails and had only been >>> responding on >>> the PR so far. >>> >>> * On versioning (Sung):* >>> >>> I don't think we need versioning if we bake into the spec that clients >>> MUST >>> fail when they encounter something they don't understand within a >>> capability's >>> payload. The capability signal means "I understand this payload >>> exists and I >>> will do the right thing with it" — not "I understand every possible >>> value >>> within it forever." The fail-closed contract gives us forward >>> compatibility >>> without version suffixes. >>> >>> *On single generic header vs. per-feature headers (Dan, Russell):* >>> >>> Re-listening to the sync, I don't think we landed on a hard conclusion >>> either >>> way. I went with the generic approach based on how other protocols >>> handle >>> this. >>> >>> *Precedents*: >>> >>> - Anthropic's API : single *anthropic-beta* header with >>> comma-separated feature >>> tokens; different endpoints consume different tokens, client sends the >>> union. >>> - Delta Sharing : single *delta-sharing-capabilities* header mixing >>> different >>> feature dimensions, server acts only on what's relevant per endpoint. >>> >>> One distinction that helped me think about it: per-feature headers like >>> X-Iceberg-Access-Delegation are directives for a specific request - >>> "vend >>> credentials for this call." A capabilities header is different - it's a >>> passive, static advertisement of what the client understands. Bundling >>> these >>> into one header keeps them distinct from per-request directives. >>> >>> The practical benefit: adding a new capability is a one-line enum >>> addition. >>> >>> I don't have a strong preference between the two approaches and am >>> happy to >>> revise the PR based on community consensus. Russell, to your point >>> about >>> scoping it to relevant endpoints - Sung raised the same on the PR and >>> I'm open >>> to that. >>> >>> Happy to discuss more in the upcoming sync (05/26). >>> >>> Thanks, >>> Prashant >>> >>> >>> >>> On Fri, May 22, 2026 at 1:15 PM Russell Spitzer < >>> [email protected]> wrote: >>> >>>> This seems matched to what I thought we went over, I'm also not sure >>>> why the shape *capabilities : {foo, bar}* necessarily mean the client >>>> has to send all the capabilities with every request. Can the client just >>>> decide to send *capabilities:{foo}* to those endpoints that care about >>>> foo? >>>> >>>> On Tue, May 19, 2026 at 2:54 PM Daniel Weeks <[email protected]> wrote: >>>> >>>>> Thanks for following up on this, Prashant. >>>>> >>>>> My recollection of the conversation was not to add a single header >>>>> that allows an arbitrary number of capabilities set, but rather to follow >>>>> the pattern established with the `x-iceberg-access-delegation' and use >>>>> specific headers applicable to endpoints where they would apply. >>>>> >>>>> I'm not convinced we want to have all the capabilities bundled up and >>>>> sent for every request (though this might be easier to implement in some >>>>> cases). There may be scenarios where you want to advertise capabilities >>>>> or >>>>> not depending on the client configuration and would make managing the set >>>>> of properties more difficult. >>>>> >>>>> After reviewing the conversation, this doesn't quite correspond to how >>>>> we ended the discussion. >>>>> >>>>> -Dan >>>>> >>>>> On Tue, May 19, 2026 at 7:06 AM Sung Yun <[email protected]> wrote: >>>>> >>>>>> Thanks Prashant, I’m supportive of introducing a generic client >>>>>> capabilities header. I agree with the direction of treating this >>>>>> primarily >>>>>> as capability advertisement for compatibility and response shaping rather >>>>>> than as a trust mechanism. >>>>>> >>>>>> I left a couple comments on the PR. One thing I’m wondering is >>>>>> whether we should discuss and define the versioning and compatibility >>>>>> model >>>>>> for capability header values as part of this proposal. >>>>>> >>>>>> Capabilities like `read-restrictions` seem very likely to evolve over >>>>>> time in ways that older clients may not be able to safely consume. I’m >>>>>> curious whether the community thinks capability values should represent >>>>>> versioned compatibility contracts (like read-restrictions.v1) and whether >>>>>> we want to define any expectations around cross-version compatibility >>>>>> behavior now as we introduce the header. >>>>>> >>>>>> Sung >>>>>> >>>>>> On 2026/05/18 18:51:52 Prashant Singh wrote: >>>>>> > Hi all, >>>>>> > >>>>>> > I'd like to propose adding a new HTTP header, >>>>>> > X-Iceberg-Client-Capabilities, >>>>>> > to the REST catalog spec. This was brought up at the Read >>>>>> Restrictions >>>>>> > community sync on May 12, 2026; I'm bringing it to the broader >>>>>> list now >>>>>> > for >>>>>> > wider feedback. >>>>>> > >>>>>> > PR: https://github.com/apache/iceberg/pull/16394 >>>>>> > Recording: https://youtu.be/b9p6mI-k-0I >>>>>> > >>>>>> > *Context* >>>>>> > >>>>>> > Iceberg's REST protocol keeps evolving — server-side scan >>>>>> > planning, remote signing, and the in-flight ReadRestrictions >>>>>> proposal >>>>>> > (PR #13879) are all features that catalogs may return but older >>>>>> clients >>>>>> > can't handle. Today there's no standard way for a client to declare >>>>>> > "I understand X" to the server. One direction discussed at the >>>>>> community >>>>>> > sync was to introduce a single generic capability header rather >>>>>> than >>>>>> > per-feature negotiations; this thread is to gather broader input >>>>>> on that >>>>>> > proposal. >>>>>> > >>>>>> > *Proposal* >>>>>> > >>>>>> > Send X-Iceberg-Client-Capabilities on every catalog request: >>>>>> > >>>>>> > ex: X-Iceberg-Client-Capabilities: >>>>>> > vended-credentials,remote-signing,scan-planning >>>>>> > >>>>>> > The Java SDK adds it via HTTPClient.baseHeaders — the same path >>>>>> used for >>>>>> > X-Client-Version and X-Client-Git-Commit-Short today. Other client >>>>>> > implementations (PyIceberg, Rust, Go, etc.) can mirror the same >>>>>> enum. >>>>>> > >>>>>> > Initial capability set for Java SDK: >>>>>> > - vended-credentials >>>>>> > - remote-signing >>>>>> > - scan-planning >>>>>> > >>>>>> > read-restrictions will be added once PR #13879 lands. >>>>>> > >>>>>> > >>>>>> > * Design notes worth feedback* >>>>>> > 1. Forward-compat hint, not security. The header is informational >>>>>> — >>>>>> > clients >>>>>> > can trivially spoof its value, so servers MUST NOT base trust >>>>>> or >>>>>> > authorization decisions on it. Trust establishment (mTLS, >>>>>> OAuth, etc.) >>>>>> > is out of scope. The spec parameter description states this >>>>>> explicitly. >>>>>> > >>>>>> > 2. enum: constraint on the schema. Mirrors >>>>>> X-Iceberg-Access-Delegation. >>>>>> > Adding >>>>>> > a new capability is a one-line spec edit; generated clients can >>>>>> > validate >>>>>> > values; servers get a machine-readable closed list of >>>>>> recognized >>>>>> > values. >>>>>> > >>>>>> > 3. Skipped for /v1/oauth/tokens. OAuth token endpoints may be >>>>>> served by >>>>>> > external IdPs (Okta, Auth0, etc.) where Iceberg-specific >>>>>> headers are >>>>>> > noise. Mirrors how X-Iceberg-Access-Delegation is handled. >>>>>> > >>>>>> > *Links* >>>>>> > >>>>>> > - PR: https://github.com/apache/iceberg/pull/16394 >>>>>> > - May 12 sync recording: https://youtu.be/b9p6mI-k-0I >>>>>> > - Related: PR #13879 < >>>>>> https://github.com/apache/iceberg/pull/13879> >>>>>> > >>>>>> > Thanks, >>>>>> > Prashant >>>>>> > >>>>>> >>>>>
