Thank you for sending this to the dev list Dan. I was having connectivity issues during the previous meeting and didn’t quite wrap my head around the proposal at that time.
+1 to creating a v2 loadTable api.
I worry (maybe needlessly) about the pattern a generic global capabilities header might set and how future spec changes might try to leverage this header. It feels itchy that a client could make a request for something, but also inform the server that it doesn’t have the capability to handle the response. I know it’s a slippery slope argument, but it doesn’t seem like a reason not to be concerned about that potential use.
I am of the opinion currently that now establishing the pattern of using versioned endpoints to extend Iceberg functionality is a greater step towards maturity of the project.
Best Regards, Kurtis C. Wright Ugh, I tried to send this on Saturday but ended up having it blocked because I used my personal gmail. Here's my original email. I think I'm saying much the same thing as Dan.
I was talking to Dan about the REST capabilities header earlier this week and he convinced me that one problem with the generic header is the association with endpoints. If we say that the "read-restrictions" capability requires certain behavior of the load table endpoint, then what happens if we later realize that other endpoints should also participate? We would need additional headers and this is the key idea behind Dan's point that we would need to associate capabilities with specific endpoints -- and there's not a great way to document that.
In that discussion, I had an idea that I think might be a good third option. Adding a capability is a way for clients to opt into new forward-breaking changes. But we already have a way to introduce forward-breaking changes: versioned endpoints. With a new endpoint, clients are required to understand payloads and behave certain ways. If we introduced a v2 loadTable endpoint, we could add read restrictions and mandate that clients must fail if there are unsupported restrictions.
There are other reasons to introduce a v2 endpoint as well: we are making table location optional in v4, we have discussed making snapshots optional (an empty list means an empty table!), and we could default the snapshot loading behavior to refs.
I think we should seriously consider adding a v2 endpoint since that's the way that we already have to introduce new required functionality. Then we don't have to worry about either too many custom headers or associating values from a generic header with endpoints and behavior. I think this is much cleaner than either head-based solution. Thanks Prashant,
I added some comments to the various PRs, but wanted to follow up here as well.
I have several concerns regarding the complexity this approach would introduce. While this may seem simple initially, a number of artifacts emerge when you start to dig into it: - Capabilities are not just what the client supports, it's what the engine supports and how the environment has been configured in some cases (like read restrictions) - A global header means that if associated behaviors apply only to a subset of endpoints, it's not clear where the behavior is targeted - Evolution is already being discussed through the addition of versioning to the labels - This can lead to a proliferation of capabilities that have behaviors orthogonal (or at least nuanced variations) to what is defined in the OpenAPI spec
I feel like what we're actually doing is trying to work around introducing new versioned endpoints (e.g. v2 loadTable). We already have versioning in the path, but we have yet to justify introducing a new version. Now we're taking extreme steps and introducing new features to continue down that path.
I think it's worth looking at what it would take to introduce a v2 loadTable endpoint and correct for a number of issues with the original endpoint (creds duplicated in properties, required metadata location) and introduce new response objects so that it's clear the calling client clearly understands the endpoint.
I'm not fundamentally opposed to client capabilities at this point, but it feels like we're creating complexity that isn't narrowly targeted.
-Dan Hi Dan, Wanted to follow up specifically on your concerns since we re-discussed this at the 05/26 sync (you weren't there for the final stretch). Recording: https://www.youtube.com/watch?v=-KEesN1udyY On the per-endpoint vs. generic header pattern: the room leaned generic, mostly on the argument that X-Iceberg-Access-Delegation is doing a different job - it's a per-request directive ("vend credentials for this call"), not a static declaration of what the client understands. Modeling capabilities the same way conflates two patterns: one is runtime preference, the other is a build-time fact about the client. On the OpenAPI tooling concern (the "spec within a spec" point from sync): the values are still a constrained enum in the OpenAPI schema - same shape as X-Iceberg-Access-Delegation. Generated clients validate values; servers get a closed list; adding a capability is a one-line enum edit. There's no parallel registry - it's a normal header whose schema happens to be enum-typed. On bundling all capabilities per request: Russell raised the same - could a client send only the relevant subset per endpoint? The sync's view was that since capabilities don't change between calls, per-endpoint subsetting adds client bookkeeping without much benefit. Open to revisiting if you have a use case in mind where conditional advertisement matters (e.g., a client that wants to suppress a capability it technically supports). Looking forward to hearing from you. Thanks, Prashant
Thanks everyone for the feedback. Apologies for the delayed response on the list, I wasn't receiving mailing list emails and had only been responding on the PR so far. On versioning (Sung): I don't think we need versioning if we bake into the spec that clients MUST fail when they encounter something they don't understand within a capability's payload. The capability signal means "I understand this payload exists and I will do the right thing with it" — not "I understand every possible value within it forever." The fail-closed contract gives us forward compatibility without version suffixes. On single generic header vs. per-feature headers (Dan, Russell): Re-listening to the sync, I don't think we landed on a hard conclusion either way. I went with the generic approach based on how other protocols handle this. Precedents: - Anthropic's API : single anthropic-beta header with comma-separated feature tokens; different endpoints consume different tokens, client sends the union. - Delta Sharing : single delta-sharing-capabilities header mixing different feature dimensions, server acts only on what's relevant per endpoint. One distinction that helped me think about it: per-feature headers like X-Iceberg-Access-Delegation are directives for a specific request - "vend credentials for this call." A capabilities header is different - it's a passive, static advertisement of what the client understands. Bundling these into one header keeps them distinct from per-request directives. The practical benefit: adding a new capability is a one-line enum addition. I don't have a strong preference between the two approaches and am happy to revise the PR based on community consensus. Russell, to your point about scoping it to relevant endpoints - Sung raised the same on the PR and I'm open to that.
Happy to discuss more in the upcoming sync (05/26).
Thanks,
Prashant
This seems matched to what I thought we went over, I'm also not sure why the shape capabilities : {foo, bar} necessarily mean the client has to send all the capabilities with every request. Can the client just decide to send capabilities:{foo} to those endpoints that care about foo?
Thanks for following up on this, Prashant.
My recollection of the conversation was not to add a single header that allows an arbitrary number of capabilities set, but rather to follow the pattern established with the `x-iceberg-access-delegation' and use specific headers applicable to endpoints where they would apply.
I'm not convinced we want to have all the capabilities bundled up and sent for every request (though this might be easier to implement in some cases). There may be scenarios where you want to advertise capabilities or not depending on the client configuration and would make managing the set of properties more difficult.
After reviewing the conversation, this doesn't quite correspond to how we ended the discussion.
-Dan
Thanks Prashant, I’m supportive of introducing a generic client capabilities header. I agree with the direction of treating this primarily as capability advertisement for compatibility and response shaping rather than as a trust mechanism.
I left a couple comments on the PR. One thing I’m wondering is whether we should discuss and define the versioning and compatibility model for capability header values as part of this proposal.
Capabilities like `read-restrictions` seem very likely to evolve over time in ways that older clients may not be able to safely consume. I’m curious whether the community thinks capability values should represent versioned compatibility contracts (like read-restrictions.v1) and whether we want to define any expectations around cross-version compatibility behavior now as we introduce the header.
Sung
On 2026/05/18 18:51:52 Prashant Singh wrote:
> Hi all,
>
> I'd like to propose adding a new HTTP header,
> X-Iceberg-Client-Capabilities,
> to the REST catalog spec. This was brought up at the Read Restrictions
> community sync on May 12, 2026; I'm bringing it to the broader list now
> for
> wider feedback.
>
> PR: https://github.com/apache/iceberg/pull/16394
> Recording: https://youtu.be/b9p6mI-k-0I
>
> *Context*
>
> Iceberg's REST protocol keeps evolving — server-side scan
> planning, remote signing, and the in-flight ReadRestrictions proposal
> (PR #13879) are all features that catalogs may return but older clients
> can't handle. Today there's no standard way for a client to declare
> "I understand X" to the server. One direction discussed at the community
> sync was to introduce a single generic capability header rather than
> per-feature negotiations; this thread is to gather broader input on that
> proposal.
>
> *Proposal*
>
> Send X-Iceberg-Client-Capabilities on every catalog request:
>
> ex: X-Iceberg-Client-Capabilities:
> vended-credentials,remote-signing,scan-planning
>
> The Java SDK adds it via HTTPClient.baseHeaders — the same path used for
> X-Client-Version and X-Client-Git-Commit-Short today. Other client
> implementations (PyIceberg, Rust, Go, etc.) can mirror the same enum.
>
> Initial capability set for Java SDK:
> - vended-credentials
> - remote-signing
> - scan-planning
>
> read-restrictions will be added once PR #13879 lands.
>
>
> * Design notes worth feedback*
> 1. Forward-compat hint, not security. The header is informational —
> clients
> can trivially spoof its value, so servers MUST NOT base trust or
> authorization decisions on it. Trust establishment (mTLS, OAuth, etc.)
> is out of scope. The spec parameter description states this explicitly.
>
> 2. enum: constraint on the schema. Mirrors X-Iceberg-Access-Delegation.
> Adding
> a new capability is a one-line spec edit; generated clients can
> validate
> values; servers get a machine-readable closed list of recognized
> values.
>
> 3. Skipped for /v1/oauth/tokens. OAuth token endpoints may be served by
> external IdPs (Okta, Auth0, etc.) where Iceberg-specific headers are
> noise. Mirrors how X-Iceberg-Access-Delegation is handled.
>
> *Links*
>
> - PR: https://github.com/apache/iceberg/pull/16394
> - May 12 sync recording: https://youtu.be/b9p6mI-k-0I
> - Related: PR #13879 <https://github.com/apache/iceberg/pull/13879>
>
> Thanks,
> Prashant
>
|
