There are a few issues all mixed together so I want to try to separate them:
1. How do we introduce forward-incompatible changes? 2. What does support for read restrictions mean in this context? 3. What does “trusted” or “untrusted” mean? 4. Why would we make read restriction support a forward-incompatible change? How do we introduce forward-incompatible changes? First, what I mean by “forward-incompatible” is that an older client can’t correctly interact with a newer server. Making table location optional is a good example: older clients require it and will fail if a server doesn’t send it. This is what Dan and I talked about last week and where I think we have the most alignment. I think that the capability header was really an alternative way to make a forward-incompatible change. Using the same location example, we could continue using v1 but signal that a client can handle a missing location with a capability, like supports-optional-location. With this simpler example I think it is more clear that this isn’t a good solution. Why deal with the problem of signaling support generally when it is unclear what this means for loadView or loadFunction? I think we all probably agree that for cases like location, a v2 endpoint is the right way forward. What does support for read restrictions mean in this context? The support that Dan and I are talking about is not evaluating and applying read restrictions. It is parsing and failing if there are restrictions that won’t be applied. The minimal commitment is to parse the “read-restrictions” field and fail if there are any column masks or row filters. That just means checking for a key and failing if either list is not empty. What does “trusted” or “untrusted” mean? Trusted means that an administrator trusts that a client is not malicious. A malicious client can say it supports and will even enforce read restrictions, but may do something else because it’s malicious. We really only care about the trusted case where the client is trusted to not lie or purposely violate requirements. But it might *accidentally* violate requirements! Why would we make read restriction support a forward-incompatible change? The reason to make read restrictions support a forward-incompatible change (v2 rather than v1) is to avoid the accidental violation. I think that administrators will use the “is this client malicious” definition of trust and could overlook client versions and abilities. By putting read restrictions in the v2 endpoint, we are saying that the client has to guarantee that it will look for read restrictions and fail if any are not enforced. This closes the possibility that a client that doesn’t support read restrictions will be trusted (by the “is this client malicious” definition) and receive them. A client that doesn’t support read restrictions will use the v1 endpoint. Independently, I think it makes sense that additional requirements that we way constitute “correct” behavior — applying masks and filters — are forward-incompatible changes. Trust aside, if we have additional requirements or change the contract, I think that’s a forward-breaking change. I know I didn’t address points directly, but I thought it would be helpful to outline my logic and cover the relevant questions. I think we can also address this in the discussion on Tuesday. Ryan On Fri, Jun 5, 2026 at 8:26 AM Daniel Weeks <[email protected]> wrote: > Christian, > > I think we're conflating two things: knowing about read restrictions and > fully supporting read restrictions. > > A major point of contention regarding why read restrictions cause issues > as a forward-compatible change is that clients don't know about the new > fields and may not fail/apply correctly. > > The claim: "If using v2 also requires implementing read-restrictions, > every client that just wants those cleanups is forced to handle > restrictions too." is overstating the effort. They just need to fail if > they see that field provided. It doesn't force implementation; it forces > knowledge about the behavior. > > In many ways, I think we're trying to design the API based on our > reference client's behavior. But each endpoint should really stand on its > own and if there's a behavior change clients need to know about, it should > be explicit. > > I'm pretty opposed to introducing side effect behaviors when we have a > path to do it explicitly. > > -Dan > > > On Thu, Jun 4, 2026 at 11:32 PM Christian Thiel < > [email protected]> wrote: > >> I want to push back on one idea in the thread: the sentiment that a v2 >> loadTable should "mandate that clients must fail if there are unsupported >> restrictions." A v2 endpoint will mostly be adopted for the unrelated >> cleanups — optional location, optional snapshots, creds out of properties. >> If using v2 also requires implementing read-restrictions, every client that >> just wants those cleanups is forced to handle restrictions too. That's >> coupling, not opt-in, and it'll drag on v2 adoption. So read-restrictions >> shouldn't be the leading motivation for v2 — I think it can just as well be >> introduced in v1. >> >> On versioned endpoints as the path to maturity: versioning is the right >> tool for genuinely forward-breaking changes like the metadata cleanups, but >> if it becomes the default way to extend the spec we're looking at a version >> bump and a client migration for every additive field. >> >> And as Russell points out, the mandate doesn't actually bind an untrusted >> caller anyway — spec language can't. Enforcement of read-restrictions is an >> out-of-band trust decision the catalog makes about a principal/engine. A >> trusted client honors them whether they arrive over v1, a header, or v2; an >> untrusted one ignores them regardless. So the capability flag is advisory >> at best, and a v2 endpoint is redundant for this feature with the trust >> decision the catalog already has to make. >> >> Kurtis — I share your unease with the global header; a single signal that >> mixes "what I'm requesting" with "what I can handle" is exactly the >> conflation that makes it awkward. On versioned endpoints as the path to >> maturity, though, I'd be a bit careful: versioning is the right tool for >> genuinely forward-breaking changes like the metadata cleanups, but if it >> becomes the default way to extend the spec we're looking at a version bump >> and a client migration for every additive field. >> >> So I'd back Russell's recommendation: ship read-restrictions in v1 and >> pursue v2 in parallel for the genuinely forward-breaking metadata changes — >> that's where mandating client understanding actually buys something. >> >> Christian >> >> On Fri, 5 Jun 2026 at 03:05, Russell Spitzer <[email protected]> >> wrote: >> >>> I don't really have an issue with a v2 endpoint, there are some good >>> reasons on Ryan's list (optional location, optional snapshots, creds >>> out of properties) but I don't see how a v2 actually changes the read >>> restrictions story. >>> >>> From my perspective read restrictions aren't a client capability the >>> client gets to opt into in the first place. Either the admin has set >>> up the catalog to send restrictions for a particular principal/engine >>> or it hasn't, the client doesn't really get a vote. Both proposals >>> are trying to use the protocol to establish that the caller will >>> honor restrictions, the header does it by advertising support and v2 >>> does it by mandating it in the spec, but neither one actually does >>> that. >>> >>> A client that says "I support read-restrictions" can ignore >>> them, a client calling v2 can ignore them, spec language doesn't bind >>> an untrusted caller. The catalog still has to decide out of band >>> whether this principal is trusted to receive a restriction-bearing >>> response, and once that decision has been made the header and the >>> endpoint version don't really add anything. >>> >>> So the header is advisory at best (a "fail fast if misconfigured" >>> hint), and a v2 endpoint, for this particular feature, is redundant >>> with whatever admin configuration is already required. I don't think >>> we should be encoding a trust signal into the protocol that the >>> protocol can't actually enforce. >>> >>> I recommend we move ahead with read-restrictions without >>> the capability flag in the V1 interface and do V2 work in parallel. >>> >>> On Thu, Jun 4, 2026 at 5:56 PM <[email protected]> wrote: >>> >>>> Thank you for sending this to the dev list Dan. I was having >>>> connectivity issues during the previous meeting and didn’t quite wrap my >>>> head around the proposal at that time. >>>> >>>> +1 to creating a v2 loadTable api. >>>> >>>> I worry (maybe needlessly) about the pattern a generic global >>>> capabilities header might set and how future spec changes might try to >>>> leverage this header. It feels itchy that a client could make a request for >>>> something, but also inform the server that it doesn’t have the capability >>>> to handle the response. I know it’s a slippery slope argument, but it >>>> doesn’t seem like a reason not to be concerned about that potential use. >>>> >>>> I am of the opinion currently that now establishing the pattern of >>>> using versioned endpoints to extend Iceberg functionality is a greater step >>>> towards maturity of the project. >>>> >>>> Best Regards, >>>> Kurtis C. Wright >>>> >>>> On Jun 4, 2026, at 15:48, Ryan Blue <[email protected]> wrote: >>>> >>>> >>>> Ugh, I tried to send this on Saturday but ended up having it blocked >>>> because I used my personal gmail. Here's my original email. I think I'm >>>> saying much the same thing as Dan. >>>> >>>> I was talking to Dan about the REST capabilities header earlier this >>>> week and he convinced me that one problem with the generic header is the >>>> association with endpoints. If we say that the "read-restrictions" >>>> capability requires certain behavior of the load table endpoint, then what >>>> happens if we later realize that other endpoints should also participate? >>>> We would need additional headers and this is the key idea behind Dan's >>>> point that we would need to associate capabilities with specific endpoints >>>> -- and there's not a great way to document that. >>>> >>>> In that discussion, I had an idea that I think might be a good third >>>> option. Adding a capability is a way for clients to opt into new >>>> forward-breaking changes. But we already have a way to introduce >>>> forward-breaking changes: versioned endpoints. With a new endpoint, clients >>>> are required to understand payloads and behave certain ways. If we >>>> introduced a v2 loadTable endpoint, we could add read restrictions and >>>> mandate that clients must fail if there are unsupported restrictions. >>>> >>>> There are other reasons to introduce a v2 endpoint as well: we are >>>> making table location optional in v4, we have discussed making snapshots >>>> optional (an empty list means an empty table!), and we could default the >>>> snapshot loading behavior to refs. >>>> >>>> I think we should seriously consider adding a v2 endpoint since that's >>>> the way that we already have to introduce new required functionality. Then >>>> we don't have to worry about either too many custom headers or associating >>>> values from a generic header with endpoints and behavior. I think this is >>>> much cleaner than either head-based solution. >>>> >>>> On Thu, Jun 4, 2026 at 8:45 AM Daniel Weeks <[email protected]> wrote: >>>> >>>>> Thanks Prashant, >>>>> >>>>> I added some comments to the various PRs, but wanted to follow up here >>>>> as well. >>>>> >>>>> I have several concerns regarding the complexity this approach would >>>>> introduce. While this may seem simple initially, a number of artifacts >>>>> emerge when you start to dig into it: >>>>> - Capabilities are not just what the client supports, it's what the >>>>> engine supports and how the environment has been configured in some cases >>>>> (like read restrictions) >>>>> - A global header means that if associated behaviors apply only to a >>>>> subset of endpoints, it's not clear where the behavior is targeted >>>>> - Evolution is already being discussed through the addition of >>>>> versioning to the labels >>>>> - This can lead to a proliferation of capabilities that have behaviors >>>>> orthogonal (or at least nuanced variations) to what is defined in the >>>>> OpenAPI spec >>>>> >>>>> I feel like what we're actually doing is trying to work around >>>>> introducing new versioned endpoints (e.g. v2 loadTable). We already have >>>>> versioning in the path, but we have yet to justify introducing a new >>>>> version. Now we're taking extreme steps and introducing new features to >>>>> continue down that path. >>>>> >>>>> I think it's worth looking at what it would take to introduce a v2 >>>>> loadTable endpoint and correct for a number of issues with the original >>>>> endpoint (creds duplicated in properties, required metadata location) and >>>>> introduce new response objects so that it's clear the calling client >>>>> clearly understands the endpoint. >>>>> >>>>> I'm not fundamentally opposed to client capabilities at this point, >>>>> but it feels like we're creating complexity that isn't narrowly targeted. >>>>> >>>>> -Dan >>>>> >>>>> On Wed, May 27, 2026 at 11:50 PM Prashant Singh < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Dan, >>>>>> >>>>>> Wanted to follow up specifically on your concerns since we >>>>>> re-discussed this at the 05/26 sync (you weren't there for >>>>>> the final stretch). Recording: >>>>>> https://www.youtube.com/watch?v=-KEesN1udyY >>>>>> >>>>>> *On the per-endpoint vs. generic header pattern*: the room leaned >>>>>> generic, mostly on the argument that >>>>>> X-Iceberg-Access-Delegation is doing a different job - it's a >>>>>> per-request directive ("vend credentials for this >>>>>> call"), not a static declaration of what the client understands. >>>>>> Modeling capabilities the same way conflates two >>>>>> patterns: one is runtime preference, the other is a build-time fact >>>>>> about the client. >>>>>> >>>>>> *On the OpenAPI tooling concern (the "spec within a spec" point **from >>>>>> sync)*: the values are still a constrained enum in the >>>>>> OpenAPI schema - same shape as X-Iceberg-Access-Delegation. >>>>>> Generated clients validate values; servers get a closed >>>>>> list; adding a capability is a one-line enum edit. There's no >>>>>> parallel registry - it's a normal header whose schema >>>>>> happens to be enum-typed. >>>>>> >>>>>> * On bundling all capabilities per request*: Russell raised the >>>>>> same - could a client send only the relevant subset per >>>>>> endpoint? The sync's view was that since capabilities don't change >>>>>> between calls, per-endpoint subsetting adds client >>>>>> bookkeeping without much benefit. Open to revisiting if you have a >>>>>> use case in mind where conditional advertisement >>>>>> matters (e.g., a client that wants to suppress a capability it >>>>>> technically supports). >>>>>> >>>>>> Looking forward to hearing from you. >>>>>> >>>>>> Thanks, >>>>>> Prashant >>>>>> >>>>>> On Mon, May 25, 2026 at 12:47 PM Prashant Singh < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Thanks everyone for the feedback. Apologies for the delayed >>>>>>> response on the >>>>>>> list, I wasn't receiving mailing list emails and had only been >>>>>>> responding on >>>>>>> the PR so far. >>>>>>> >>>>>>> * On versioning (Sung):* >>>>>>> >>>>>>> I don't think we need versioning if we bake into the spec that >>>>>>> clients MUST >>>>>>> fail when they encounter something they don't understand within a >>>>>>> capability's >>>>>>> payload. The capability signal means "I understand this payload >>>>>>> exists and I >>>>>>> will do the right thing with it" — not "I understand every >>>>>>> possible value >>>>>>> within it forever." The fail-closed contract gives us forward >>>>>>> compatibility >>>>>>> without version suffixes. >>>>>>> >>>>>>> *On single generic header vs. per-feature headers (Dan, Russell):* >>>>>>> >>>>>>> Re-listening to the sync, I don't think we landed on a hard >>>>>>> conclusion either >>>>>>> way. I went with the generic approach based on how other protocols >>>>>>> handle >>>>>>> this. >>>>>>> >>>>>>> *Precedents*: >>>>>>> >>>>>>> - Anthropic's API : single *anthropic-beta* header with >>>>>>> comma-separated feature >>>>>>> tokens; different endpoints consume different tokens, client sends >>>>>>> the union. >>>>>>> - Delta Sharing : single *delta-sharing-capabilities* header >>>>>>> mixing different >>>>>>> feature dimensions, server acts only on what's relevant per >>>>>>> endpoint. >>>>>>> >>>>>>> One distinction that helped me think about it: per-feature headers >>>>>>> like >>>>>>> X-Iceberg-Access-Delegation are directives for a specific request >>>>>>> - "vend >>>>>>> credentials for this call." A capabilities header is different - >>>>>>> it's a >>>>>>> passive, static advertisement of what the client understands. >>>>>>> Bundling these >>>>>>> into one header keeps them distinct from per-request directives. >>>>>>> >>>>>>> The practical benefit: adding a new capability is a one-line enum >>>>>>> addition. >>>>>>> >>>>>>> I don't have a strong preference between the two approaches and am >>>>>>> happy to >>>>>>> revise the PR based on community consensus. Russell, to your point >>>>>>> about >>>>>>> scoping it to relevant endpoints - Sung raised the same on the PR >>>>>>> and I'm open >>>>>>> to that. >>>>>>> >>>>>>> Happy to discuss more in the upcoming sync (05/26). >>>>>>> >>>>>>> Thanks, >>>>>>> Prashant >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, May 22, 2026 at 1:15 PM Russell Spitzer < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> This seems matched to what I thought we went over, I'm also not >>>>>>>> sure why the shape *capabilities : {foo, bar}* necessarily mean >>>>>>>> the client has to send all the capabilities with every request. Can the >>>>>>>> client just decide to send *capabilities:{foo}* to those endpoints >>>>>>>> that care about foo? >>>>>>>> >>>>>>>> On Tue, May 19, 2026 at 2:54 PM Daniel Weeks <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thanks for following up on this, Prashant. >>>>>>>>> >>>>>>>>> My recollection of the conversation was not to add a single header >>>>>>>>> that allows an arbitrary number of capabilities set, but rather to >>>>>>>>> follow >>>>>>>>> the pattern established with the `x-iceberg-access-delegation' and use >>>>>>>>> specific headers applicable to endpoints where they would apply. >>>>>>>>> >>>>>>>>> I'm not convinced we want to have all the capabilities bundled up >>>>>>>>> and sent for every request (though this might be easier to implement >>>>>>>>> in >>>>>>>>> some cases). There may be scenarios where you want to advertise >>>>>>>>> capabilities or not depending on the client configuration and would >>>>>>>>> make >>>>>>>>> managing the set of properties more difficult. >>>>>>>>> >>>>>>>>> After reviewing the conversation, this doesn't quite correspond to >>>>>>>>> how we ended the discussion. >>>>>>>>> >>>>>>>>> -Dan >>>>>>>>> >>>>>>>>> On Tue, May 19, 2026 at 7:06 AM Sung Yun <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Thanks Prashant, I’m supportive of introducing a generic client >>>>>>>>>> capabilities header. I agree with the direction of treating this >>>>>>>>>> primarily >>>>>>>>>> as capability advertisement for compatibility and response shaping >>>>>>>>>> rather >>>>>>>>>> than as a trust mechanism. >>>>>>>>>> >>>>>>>>>> I left a couple comments on the PR. One thing I’m wondering is >>>>>>>>>> whether we should discuss and define the versioning and >>>>>>>>>> compatibility model >>>>>>>>>> for capability header values as part of this proposal. >>>>>>>>>> >>>>>>>>>> Capabilities like `read-restrictions` seem very likely to evolve >>>>>>>>>> over time in ways that older clients may not be able to safely >>>>>>>>>> consume. I’m >>>>>>>>>> curious whether the community thinks capability values should >>>>>>>>>> represent >>>>>>>>>> versioned compatibility contracts (like read-restrictions.v1) and >>>>>>>>>> whether >>>>>>>>>> we want to define any expectations around cross-version compatibility >>>>>>>>>> behavior now as we introduce the header. >>>>>>>>>> >>>>>>>>>> Sung >>>>>>>>>> >>>>>>>>>> On 2026/05/18 18:51:52 Prashant Singh wrote: >>>>>>>>>> > Hi all, >>>>>>>>>> > >>>>>>>>>> > I'd like to propose adding a new HTTP header, >>>>>>>>>> > X-Iceberg-Client-Capabilities, >>>>>>>>>> > to the REST catalog spec. This was brought up at the Read >>>>>>>>>> Restrictions >>>>>>>>>> > community sync on May 12, 2026; I'm bringing it to the >>>>>>>>>> broader list now >>>>>>>>>> > for >>>>>>>>>> > wider feedback. >>>>>>>>>> > >>>>>>>>>> > PR: https://github.com/apache/iceberg/pull/16394 >>>>>>>>>> > Recording: https://youtu.be/b9p6mI-k-0I >>>>>>>>>> > >>>>>>>>>> > *Context* >>>>>>>>>> > >>>>>>>>>> > Iceberg's REST protocol keeps evolving — server-side scan >>>>>>>>>> > planning, remote signing, and the in-flight >>>>>>>>>> ReadRestrictions proposal >>>>>>>>>> > (PR #13879) are all features that catalogs may return but >>>>>>>>>> older clients >>>>>>>>>> > can't handle. Today there's no standard way for a client to >>>>>>>>>> declare >>>>>>>>>> > "I understand X" to the server. One direction discussed at >>>>>>>>>> the community >>>>>>>>>> > sync was to introduce a single generic capability header >>>>>>>>>> rather than >>>>>>>>>> > per-feature negotiations; this thread is to gather broader >>>>>>>>>> input on that >>>>>>>>>> > proposal. >>>>>>>>>> > >>>>>>>>>> > *Proposal* >>>>>>>>>> > >>>>>>>>>> > Send X-Iceberg-Client-Capabilities on every catalog request: >>>>>>>>>> > >>>>>>>>>> > ex: X-Iceberg-Client-Capabilities: >>>>>>>>>> > vended-credentials,remote-signing,scan-planning >>>>>>>>>> > >>>>>>>>>> > The Java SDK adds it via HTTPClient.baseHeaders — the same >>>>>>>>>> path used for >>>>>>>>>> > X-Client-Version and X-Client-Git-Commit-Short today. Other >>>>>>>>>> client >>>>>>>>>> > implementations (PyIceberg, Rust, Go, etc.) can mirror the >>>>>>>>>> same enum. >>>>>>>>>> > >>>>>>>>>> > Initial capability set for Java SDK: >>>>>>>>>> > - vended-credentials >>>>>>>>>> > - remote-signing >>>>>>>>>> > - scan-planning >>>>>>>>>> > >>>>>>>>>> > read-restrictions will be added once PR #13879 lands. >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > * Design notes worth feedback* >>>>>>>>>> > 1. Forward-compat hint, not security. The header is >>>>>>>>>> informational — >>>>>>>>>> > clients >>>>>>>>>> > can trivially spoof its value, so servers MUST NOT base >>>>>>>>>> trust or >>>>>>>>>> > authorization decisions on it. Trust establishment (mTLS, >>>>>>>>>> OAuth, etc.) >>>>>>>>>> > is out of scope. The spec parameter description states >>>>>>>>>> this explicitly. >>>>>>>>>> > >>>>>>>>>> > 2. enum: constraint on the schema. Mirrors >>>>>>>>>> X-Iceberg-Access-Delegation. >>>>>>>>>> > Adding >>>>>>>>>> > a new capability is a one-line spec edit; generated >>>>>>>>>> clients can >>>>>>>>>> > validate >>>>>>>>>> > values; servers get a machine-readable closed list of >>>>>>>>>> recognized >>>>>>>>>> > values. >>>>>>>>>> > >>>>>>>>>> > 3. Skipped for /v1/oauth/tokens. OAuth token endpoints may be >>>>>>>>>> served by >>>>>>>>>> > external IdPs (Okta, Auth0, etc.) where Iceberg-specific >>>>>>>>>> headers are >>>>>>>>>> > noise. Mirrors how X-Iceberg-Access-Delegation is handled. >>>>>>>>>> > >>>>>>>>>> > *Links* >>>>>>>>>> > >>>>>>>>>> > - PR: https://github.com/apache/iceberg/pull/16394 >>>>>>>>>> > - May 12 sync recording: https://youtu.be/b9p6mI-k-0I >>>>>>>>>> > - Related: PR #13879 < >>>>>>>>>> https://github.com/apache/iceberg/pull/13879> >>>>>>>>>> > >>>>>>>>>> > Thanks, >>>>>>>>>> > Prashant >>>>>>>>>> > >>>>>>>>>> >>>>>>>>>
