Thank you for the feedback everyone. Based on the 06/09 ReadRestrictions sync we're dropping the capabilities header and going with the V1/V2 path captured in the read-restrictions summary thread. See latest email for the summary (06/09): https://lists.apache.org/thread/wfgqlsnpyvt1t42fg6o4q48w8nc14oof
I updated the read restrictions spec PR with the wording we discussed during the sync, please have a look when you all get some time: https://github.com/apache/iceberg/pull/13879 Best, Prashant On Mon, Jun 8, 2026 at 2:29 PM Kurtis Wright <[email protected]> wrote: > I am also aligned with this. I think I was vague in my earlier email, I > would like to see a global capabilities header either discussed > separately of Read Restrictions spec change or dropped completely. > > On the compromise proposed by Daniel and highlighted by Russell. I am in > agreement on moving the enforcement spec language to MUST for v2 load > table, as long as Read Restrictions gets to move forward separate from v2 > being fully formalized and agreed upon. > > On Mon, Jun 8, 2026 at 12:13 PM Russell Spitzer <[email protected]> > wrote: > >> I would rather make the administrator role simpler here. But maybe we can >>> do both: we could allow read restrictions on the v1 path and require admins >>> to include enforcement in their trust decision. And on the v2 path we could >>> make support required for the future. Is that a good compromise? >> >> >> This sounds reasonable to me. I have no problem removing the capabilities >> header and just adding the read restrictions. Similar to our discussions on >> scan planning and using that for Auth, at best it lets the Catalog fail >> fast on an honestly but incorrectly configured client (I didn't think we >> needed a header there either.) >> >> On Fri, Jun 5, 2026 at 3:34 PM Daniel Weeks <[email protected]> wrote: >> >>> I generally agree with Ryan's framing. >>> >>> My issue is with stating that we need the client capabilities as is >>> currently proposed. Either this is a forward compatible feature and we >>> don't need capabilities, or it's not a forward compatible feature and we >>> need a v2 endpoint to signify the change in expected behavior. >>> >>> The discussion around using capabilities to augment endpoint behavior is >>> very concerning to me. There are fields we've added like Yufei mentioned, >>> but those are a different category in that they truly are >>> forward compatible; they do not require a change in behavior. >>> >>> I'm fine putting the onus on the administrator and the catalog to decide >>> when/what to return, but trying to share that responsibility between the >>> library, the engine integration and the catalog through the capabilities >>> makes the situation more complicated, but not more secure. >>> >>> I would add that I do think there are mechanisms to augment endpoint >>> behaviors in a structured and narrowly targeted way to allow for forward >>> compatibility, but the current proposal doesn't strike the right balance. >>> >>> -Dan >>> >>> >>> >>> On Fri, Jun 5, 2026 at 1:13 PM Ryan Blue <[email protected]> wrote: >>> >>>> I’m not sure moving the feature behind a v2 endpoint meaningfully >>>> reduces the diligence required from administrators. It may reduce one class >>>> of misconfiguration, but it does not eliminate the need for an explicit >>>> trust and compatibility decision. >>>> >>>> I agree with this. >>>> >>>> My rationale is that if we want to *require* a client behavior then it >>>> should be v2 or some other mechanism for signaling to the service that the >>>> client opts in. If we want read restrictions to be informational, then we >>>> can use v1. >>>> >>>> If we don't want this to be a requirement then it is not a >>>> compatibility breaking change. We would just have to change the spec >>>> language to state that clients SHOULD fail if restrictions aren’t enforced >>>> rather than MUST fail if restrictions aren’t enforced. >>>> >>>> Then that puts more responsibility on administrators. Administrators >>>> would have to verify that an engine is not malicious and actually supports >>>> read restrictions. >>>> >>>> Maybe an example would help? Say I’m an administrator setting up a new >>>> Trino cluster using a Docker image from the project. Because I’m using a >>>> public Docker image from the Trino community, have verified its hash, and >>>> am the one deploying it, I trust that it doesn’t have malicious code. I >>>> think as an admin, I would consider that “trusted” when I set up the >>>> identity for that cluster. If we don’t use a v2 endpoint (or other option) >>>> then I would also have to check the Trino version to see if it supports >>>> read restrictions and test them out. >>>> >>>> I would rather make the administrator role simpler here. But maybe we >>>> can do both: we could allow read restrictions on the v1 path and require >>>> admins to include enforcement in their trust decision. And on the v2 path >>>> we could make support required for the future. Is that a good compromise? >>>> >>>> Ryan >>>> >>>> On Fri, Jun 5, 2026 at 12:19 PM Yufei Gu <[email protected]> wrote: >>>> >>>>> We have added fields (e.g., storage-credentials and ETags) to >>>>> LoadTable before and generally haven't treated them as compatibility >>>>> issues. I think the difference here is that this change affects the >>>>> correctness contract for reading data. If a client does not understand or >>>>> cannot enforce the returned restrictions, ignoring the field could result >>>>> in behavior that the catalog administrator did not intend. >>>>> >>>>> That said, I'm not sure that preventing those violations is primarily >>>>> a protocol versioning concern. If a catalog is configured to return >>>>> restriction-bearing responses, it already needs an administrative >>>>> mechanism >>>>> to determine which principals or engines are expected to honor those >>>>> restrictions. >>>>> >>>>> One reason I'm not fully convinced by the accidental violation >>>>> argument is that, the operational burden seems roughly the same, >>>>> regardless >>>>> of whether restrictions are delivered through v1 or v2. For a >>>>> security-sensitive feature, I would expect admins to explicitly verify >>>>> that >>>>> a given engine or client can handle restrictions before enabling >>>>> restriction-bearing responses. In practice, checking "does this client >>>>> support read restrictions?" does not feel materially different from >>>>> checking "does this client support loadTable v2?". >>>>> >>>>> Because of that, I'm not sure moving the feature behind a v2 endpoint >>>>> meaningfully reduces the diligence required from administrators. It may >>>>> reduce one class of misconfiguration, but it does not eliminate the need >>>>> for an explicit trust and compatibility decision. If the goal is to avoid >>>>> accidental violations, the safer rule is that catalogs should not return >>>>> restriction-bearing responses, unless an admin has explicitly enabled them >>>>> for a given principal or engine, regardless of whether the request is made >>>>> through v1 or v2. >>>>> >>>>> Yufei >>>>> >>>>> >>>>> On Fri, Jun 5, 2026 at 11:05 AM Ryan Blue <[email protected]> wrote: >>>>> >>>>>> There are a few issues all mixed together so I want to try to >>>>>> separate them: >>>>>> >>>>>> 1. How do we introduce forward-incompatible changes? >>>>>> 2. What does support for read restrictions mean in this context? >>>>>> 3. What does “trusted” or “untrusted” mean? >>>>>> 4. Why would we make read restriction support a >>>>>> forward-incompatible change? >>>>>> >>>>>> How do we introduce forward-incompatible changes? >>>>>> >>>>>> First, what I mean by “forward-incompatible” is that an older client >>>>>> can’t correctly interact with a newer server. Making table location >>>>>> optional >>>>>> is a good example: older clients require it and will fail if a server >>>>>> doesn’t send it. >>>>>> >>>>>> This is what Dan and I talked about last week and where I think we >>>>>> have the most alignment. I think that the capability header was really an >>>>>> alternative way to make a forward-incompatible change. Using the same >>>>>> location example, we could continue using v1 but signal that a client can >>>>>> handle a missing location with a capability, like >>>>>> supports-optional-location. With this simpler example I think it is >>>>>> more clear that this isn’t a good solution. Why deal with the problem of >>>>>> signaling support generally when it is unclear what this means for >>>>>> loadView or loadFunction? >>>>>> >>>>>> I think we all probably agree that for cases like location, a v2 >>>>>> endpoint is the right way forward. >>>>>> >>>>>> What does support for read restrictions mean in this context? >>>>>> >>>>>> The support that Dan and I are talking about is not evaluating and >>>>>> applying read restrictions. It is parsing and failing if there are >>>>>> restrictions that won’t be applied. The minimal commitment is to parse >>>>>> the >>>>>> “read-restrictions” field and fail if there are any column masks or row >>>>>> filters. >>>>>> >>>>>> That just means checking for a key and failing if either list is not >>>>>> empty. >>>>>> >>>>>> What does “trusted” or “untrusted” mean? >>>>>> >>>>>> Trusted means that an administrator trusts that a client is not >>>>>> malicious. A malicious client can say it supports and will even enforce >>>>>> read restrictions, but may do something else because it’s malicious. We >>>>>> really only care about the trusted case where the client is trusted to >>>>>> not >>>>>> lie or purposely violate requirements. But it might *accidentally* >>>>>> violate >>>>>> requirements! >>>>>> >>>>>> Why would we make read restriction support a forward-incompatible >>>>>> change? >>>>>> >>>>>> The reason to make read restrictions support a forward-incompatible >>>>>> change (v2 rather than v1) is to avoid the accidental violation. I think >>>>>> that administrators will use the “is this client malicious” definition of >>>>>> trust and could overlook client versions and abilities. >>>>>> >>>>>> By putting read restrictions in the v2 endpoint, we are saying that >>>>>> the client has to guarantee that it will look for read restrictions and >>>>>> fail if any are not enforced. This closes the possibility that a client >>>>>> that doesn’t support read restrictions will be trusted (by the “is this >>>>>> client malicious” definition) and receive them. A client that doesn’t >>>>>> support read restrictions will use the v1 endpoint. >>>>>> >>>>>> Independently, I think it makes sense that additional requirements >>>>>> that we way constitute “correct” behavior — applying masks and filters — >>>>>> are forward-incompatible changes. Trust aside, if we have additional >>>>>> requirements or change the contract, I think that’s a forward-breaking >>>>>> change. >>>>>> >>>>>> I know I didn’t address points directly, but I thought it would be >>>>>> helpful to outline my logic and cover the relevant questions. I think we >>>>>> can also address this in the discussion on Tuesday. >>>>>> >>>>>> Ryan >>>>>> >>>>>> On Fri, Jun 5, 2026 at 8:26 AM Daniel Weeks <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Christian, >>>>>>> >>>>>>> I think we're conflating two things: knowing about read >>>>>>> restrictions and fully supporting read restrictions. >>>>>>> >>>>>>> A major point of contention regarding why read restrictions cause >>>>>>> issues as a forward-compatible change is that clients don't know about >>>>>>> the >>>>>>> new fields and may not fail/apply correctly. >>>>>>> >>>>>>> The claim: "If using v2 also requires implementing >>>>>>> read-restrictions, every client that just wants those cleanups is >>>>>>> forced to >>>>>>> handle restrictions too." is overstating the effort. They just need to >>>>>>> fail if they see that field provided. It doesn't force implementation; >>>>>>> it >>>>>>> forces knowledge about the behavior. >>>>>>> >>>>>>> In many ways, I think we're trying to design the API based on our >>>>>>> reference client's behavior. But each endpoint should really stand on >>>>>>> its >>>>>>> own and if there's a behavior change clients need to know about, it >>>>>>> should >>>>>>> be explicit. >>>>>>> >>>>>>> I'm pretty opposed to introducing side effect behaviors when we have >>>>>>> a path to do it explicitly. >>>>>>> >>>>>>> -Dan >>>>>>> >>>>>>> >>>>>>> On Thu, Jun 4, 2026 at 11:32 PM Christian Thiel < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> I want to push back on one idea in the thread: the sentiment that a >>>>>>>> v2 loadTable should "mandate that clients must fail if there are >>>>>>>> unsupported restrictions." A v2 endpoint will mostly be adopted for the >>>>>>>> unrelated cleanups — optional location, optional snapshots, creds out >>>>>>>> of >>>>>>>> properties. If using v2 also requires implementing read-restrictions, >>>>>>>> every >>>>>>>> client that just wants those cleanups is forced to handle restrictions >>>>>>>> too. >>>>>>>> That's coupling, not opt-in, and it'll drag on v2 adoption. So >>>>>>>> read-restrictions shouldn't be the leading motivation for v2 — I think >>>>>>>> it >>>>>>>> can just as well be introduced in v1. >>>>>>>> >>>>>>>> On versioned endpoints as the path to maturity: versioning is the >>>>>>>> right tool for genuinely forward-breaking changes like the metadata >>>>>>>> cleanups, but if it becomes the default way to extend the spec we're >>>>>>>> looking at a version bump and a client migration for every additive >>>>>>>> field. >>>>>>>> >>>>>>>> And as Russell points out, the mandate doesn't actually bind an >>>>>>>> untrusted caller anyway — spec language can't. Enforcement of >>>>>>>> read-restrictions is an out-of-band trust decision the catalog makes >>>>>>>> about >>>>>>>> a principal/engine. A trusted client honors them whether they arrive >>>>>>>> over >>>>>>>> v1, a header, or v2; an untrusted one ignores them regardless. So the >>>>>>>> capability flag is advisory at best, and a v2 endpoint is redundant for >>>>>>>> this feature with the trust decision the catalog already has to make. >>>>>>>> >>>>>>>> Kurtis — I share your unease with the global header; a single >>>>>>>> signal that mixes "what I'm requesting" with "what I can handle" is >>>>>>>> exactly >>>>>>>> the conflation that makes it awkward. On versioned endpoints as the >>>>>>>> path to >>>>>>>> maturity, though, I'd be a bit careful: versioning is the right tool >>>>>>>> for >>>>>>>> genuinely forward-breaking changes like the metadata cleanups, but if >>>>>>>> it >>>>>>>> becomes the default way to extend the spec we're looking at a version >>>>>>>> bump >>>>>>>> and a client migration for every additive field. >>>>>>>> >>>>>>>> So I'd back Russell's recommendation: ship read-restrictions in v1 >>>>>>>> and pursue v2 in parallel for the genuinely forward-breaking metadata >>>>>>>> changes — that's where mandating client understanding actually buys >>>>>>>> something. >>>>>>>> >>>>>>>> Christian >>>>>>>> >>>>>>>> On Fri, 5 Jun 2026 at 03:05, Russell Spitzer < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> I don't really have an issue with a v2 endpoint, there are some >>>>>>>>> good >>>>>>>>> reasons on Ryan's list (optional location, optional snapshots, >>>>>>>>> creds >>>>>>>>> out of properties) but I don't see how a v2 actually changes the >>>>>>>>> read >>>>>>>>> restrictions story. >>>>>>>>> >>>>>>>>> From my perspective read restrictions aren't a client capability >>>>>>>>> the >>>>>>>>> client gets to opt into in the first place. Either the admin has >>>>>>>>> set >>>>>>>>> up the catalog to send restrictions for a particular >>>>>>>>> principal/engine >>>>>>>>> or it hasn't, the client doesn't really get a vote. Both proposals >>>>>>>>> are trying to use the protocol to establish that the caller will >>>>>>>>> honor restrictions, the header does it by advertising support and >>>>>>>>> v2 >>>>>>>>> does it by mandating it in the spec, but neither one actually does >>>>>>>>> that. >>>>>>>>> >>>>>>>>> A client that says "I support read-restrictions" can ignore >>>>>>>>> them, a client calling v2 can ignore them, spec language doesn't >>>>>>>>> bind >>>>>>>>> an untrusted caller. The catalog still has to decide out of band >>>>>>>>> whether this principal is trusted to receive a restriction-bearing >>>>>>>>> response, and once that decision has been made the header and the >>>>>>>>> endpoint version don't really add anything. >>>>>>>>> >>>>>>>>> So the header is advisory at best (a "fail fast if misconfigured" >>>>>>>>> hint), and a v2 endpoint, for this particular feature, is redundant >>>>>>>>> with whatever admin configuration is already required. I don't >>>>>>>>> think >>>>>>>>> we should be encoding a trust signal into the protocol that the >>>>>>>>> protocol can't actually enforce. >>>>>>>>> >>>>>>>>> I recommend we move ahead with read-restrictions without >>>>>>>>> the capability flag in the V1 interface and do V2 work in parallel. >>>>>>>>> >>>>>>>>> On Thu, Jun 4, 2026 at 5:56 PM <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> Thank you for sending this to the dev list Dan. I was having >>>>>>>>>> connectivity issues during the previous meeting and didn’t quite >>>>>>>>>> wrap my >>>>>>>>>> head around the proposal at that time. >>>>>>>>>> >>>>>>>>>> +1 to creating a v2 loadTable api. >>>>>>>>>> >>>>>>>>>> I worry (maybe needlessly) about the pattern a generic global >>>>>>>>>> capabilities header might set and how future spec changes might try >>>>>>>>>> to >>>>>>>>>> leverage this header. It feels itchy that a client could make a >>>>>>>>>> request for >>>>>>>>>> something, but also inform the server that it doesn’t have the >>>>>>>>>> capability >>>>>>>>>> to handle the response. I know it’s a slippery slope argument, but it >>>>>>>>>> doesn’t seem like a reason not to be concerned about that potential >>>>>>>>>> use. >>>>>>>>>> >>>>>>>>>> I am of the opinion currently that now establishing the pattern >>>>>>>>>> of using versioned endpoints to extend Iceberg functionality is a >>>>>>>>>> greater >>>>>>>>>> step towards maturity of the project. >>>>>>>>>> >>>>>>>>>> Best Regards, >>>>>>>>>> Kurtis C. Wright >>>>>>>>>> >>>>>>>>>> On Jun 4, 2026, at 15:48, Ryan Blue <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Ugh, I tried to send this on Saturday but ended up having it >>>>>>>>>> blocked because I used my personal gmail. Here's my original email. >>>>>>>>>> I think >>>>>>>>>> I'm saying much the same thing as Dan. >>>>>>>>>> >>>>>>>>>> I was talking to Dan about the REST capabilities header earlier >>>>>>>>>> this week and he convinced me that one problem with the generic >>>>>>>>>> header is >>>>>>>>>> the association with endpoints. If we say that the >>>>>>>>>> "read-restrictions" >>>>>>>>>> capability requires certain behavior of the load table endpoint, >>>>>>>>>> then what >>>>>>>>>> happens if we later realize that other endpoints should also >>>>>>>>>> participate? >>>>>>>>>> We would need additional headers and this is the key idea behind >>>>>>>>>> Dan's >>>>>>>>>> point that we would need to associate capabilities with specific >>>>>>>>>> endpoints >>>>>>>>>> -- and there's not a great way to document that. >>>>>>>>>> >>>>>>>>>> In that discussion, I had an idea that I think might be a good >>>>>>>>>> third option. Adding a capability is a way for clients to opt into >>>>>>>>>> new >>>>>>>>>> forward-breaking changes. But we already have a way to introduce >>>>>>>>>> forward-breaking changes: versioned endpoints. With a new endpoint, >>>>>>>>>> clients >>>>>>>>>> are required to understand payloads and behave certain ways. If we >>>>>>>>>> introduced a v2 loadTable endpoint, we could add read restrictions >>>>>>>>>> and >>>>>>>>>> mandate that clients must fail if there are unsupported restrictions. >>>>>>>>>> >>>>>>>>>> There are other reasons to introduce a v2 endpoint as well: we >>>>>>>>>> are making table location optional in v4, we have discussed making >>>>>>>>>> snapshots optional (an empty list means an empty table!), and we >>>>>>>>>> could >>>>>>>>>> default the snapshot loading behavior to refs. >>>>>>>>>> >>>>>>>>>> I think we should seriously consider adding a v2 endpoint since >>>>>>>>>> that's the way that we already have to introduce new required >>>>>>>>>> functionality. Then we don't have to worry about either too many >>>>>>>>>> custom >>>>>>>>>> headers or associating values from a generic header with endpoints >>>>>>>>>> and >>>>>>>>>> behavior. I think this is much cleaner than either head-based >>>>>>>>>> solution. >>>>>>>>>> >>>>>>>>>> On Thu, Jun 4, 2026 at 8:45 AM Daniel Weeks <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks Prashant, >>>>>>>>>>> >>>>>>>>>>> I added some comments to the various PRs, but wanted to follow >>>>>>>>>>> up here as well. >>>>>>>>>>> >>>>>>>>>>> I have several concerns regarding the complexity this approach >>>>>>>>>>> would introduce. While this may seem simple initially, a number of >>>>>>>>>>> artifacts emerge when you start to dig into it: >>>>>>>>>>> - Capabilities are not just what the client supports, it's what >>>>>>>>>>> the engine supports and how the environment has been configured in >>>>>>>>>>> some >>>>>>>>>>> cases (like read restrictions) >>>>>>>>>>> - A global header means that if associated behaviors apply only >>>>>>>>>>> to a subset of endpoints, it's not clear where the behavior is >>>>>>>>>>> targeted >>>>>>>>>>> - Evolution is already being discussed through the addition of >>>>>>>>>>> versioning to the labels >>>>>>>>>>> - This can lead to a proliferation of capabilities that have >>>>>>>>>>> behaviors orthogonal (or at least nuanced variations) to what is >>>>>>>>>>> defined in >>>>>>>>>>> the OpenAPI spec >>>>>>>>>>> >>>>>>>>>>> I feel like what we're actually doing is trying to work around >>>>>>>>>>> introducing new versioned endpoints (e.g. v2 loadTable). We >>>>>>>>>>> already have >>>>>>>>>>> versioning in the path, but we have yet to justify introducing a new >>>>>>>>>>> version. Now we're taking extreme steps and introducing new >>>>>>>>>>> features to >>>>>>>>>>> continue down that path. >>>>>>>>>>> >>>>>>>>>>> I think it's worth looking at what it would take to introduce a >>>>>>>>>>> v2 loadTable endpoint and correct for a number of issues with the >>>>>>>>>>> original >>>>>>>>>>> endpoint (creds duplicated in properties, required metadata >>>>>>>>>>> location) and >>>>>>>>>>> introduce new response objects so that it's clear the calling client >>>>>>>>>>> clearly understands the endpoint. >>>>>>>>>>> >>>>>>>>>>> I'm not fundamentally opposed to client capabilities at this >>>>>>>>>>> point, but it feels like we're creating complexity that isn't >>>>>>>>>>> narrowly >>>>>>>>>>> targeted. >>>>>>>>>>> >>>>>>>>>>> -Dan >>>>>>>>>>> >>>>>>>>>>> On Wed, May 27, 2026 at 11:50 PM Prashant Singh < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi Dan, >>>>>>>>>>>> >>>>>>>>>>>> Wanted to follow up specifically on your concerns since we >>>>>>>>>>>> re-discussed this at the 05/26 sync (you weren't there for >>>>>>>>>>>> the final stretch). Recording: >>>>>>>>>>>> https://www.youtube.com/watch?v=-KEesN1udyY >>>>>>>>>>>> >>>>>>>>>>>> *On the per-endpoint vs. generic header pattern*: the room >>>>>>>>>>>> leaned generic, mostly on the argument that >>>>>>>>>>>> X-Iceberg-Access-Delegation is doing a different job - it's a >>>>>>>>>>>> per-request directive ("vend credentials for this >>>>>>>>>>>> call"), not a static declaration of what the client >>>>>>>>>>>> understands. Modeling capabilities the same way conflates two >>>>>>>>>>>> patterns: one is runtime preference, the other is a >>>>>>>>>>>> build-time fact about the client. >>>>>>>>>>>> >>>>>>>>>>>> *On the OpenAPI tooling concern (the "spec within a spec" >>>>>>>>>>>> point **from sync)*: the values are still a constrained enum >>>>>>>>>>>> in the >>>>>>>>>>>> OpenAPI schema - same shape as X-Iceberg-Access-Delegation. >>>>>>>>>>>> Generated clients validate values; servers get a closed >>>>>>>>>>>> list; adding a capability is a one-line enum edit. There's no >>>>>>>>>>>> parallel registry - it's a normal header whose schema >>>>>>>>>>>> happens to be enum-typed. >>>>>>>>>>>> >>>>>>>>>>>> * On bundling all capabilities per request*: Russell raised >>>>>>>>>>>> the same - could a client send only the relevant subset per >>>>>>>>>>>> endpoint? The sync's view was that since capabilities don't >>>>>>>>>>>> change between calls, per-endpoint subsetting adds client >>>>>>>>>>>> bookkeeping without much benefit. Open to revisiting if you >>>>>>>>>>>> have a use case in mind where conditional advertisement >>>>>>>>>>>> matters (e.g., a client that wants to suppress a capability >>>>>>>>>>>> it technically supports). >>>>>>>>>>>> >>>>>>>>>>>> Looking forward to hearing from you. >>>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Prashant >>>>>>>>>>>> >>>>>>>>>>>> On Mon, May 25, 2026 at 12:47 PM Prashant Singh < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Thanks everyone for the feedback. Apologies for the delayed >>>>>>>>>>>>> response on the >>>>>>>>>>>>> list, I wasn't receiving mailing list emails and had only >>>>>>>>>>>>> been responding on >>>>>>>>>>>>> the PR so far. >>>>>>>>>>>>> >>>>>>>>>>>>> * On versioning (Sung):* >>>>>>>>>>>>> >>>>>>>>>>>>> I don't think we need versioning if we bake into the spec >>>>>>>>>>>>> that clients MUST >>>>>>>>>>>>> fail when they encounter something they don't understand >>>>>>>>>>>>> within a capability's >>>>>>>>>>>>> payload. The capability signal means "I understand this >>>>>>>>>>>>> payload exists and I >>>>>>>>>>>>> will do the right thing with it" — not "I understand every >>>>>>>>>>>>> possible value >>>>>>>>>>>>> within it forever." The fail-closed contract gives us >>>>>>>>>>>>> forward compatibility >>>>>>>>>>>>> without version suffixes. >>>>>>>>>>>>> >>>>>>>>>>>>> *On single generic header vs. per-feature headers (Dan, >>>>>>>>>>>>> Russell):* >>>>>>>>>>>>> >>>>>>>>>>>>> Re-listening to the sync, I don't think we landed on a hard >>>>>>>>>>>>> conclusion either >>>>>>>>>>>>> way. I went with the generic approach based on how other >>>>>>>>>>>>> protocols handle >>>>>>>>>>>>> this. >>>>>>>>>>>>> >>>>>>>>>>>>> *Precedents*: >>>>>>>>>>>>> >>>>>>>>>>>>> - Anthropic's API : single *anthropic-beta* header with >>>>>>>>>>>>> comma-separated feature >>>>>>>>>>>>> tokens; different endpoints consume different tokens, client >>>>>>>>>>>>> sends the union. >>>>>>>>>>>>> - Delta Sharing : single *delta-sharing-capabilities* >>>>>>>>>>>>> header mixing different >>>>>>>>>>>>> feature dimensions, server acts only on what's relevant per >>>>>>>>>>>>> endpoint. >>>>>>>>>>>>> >>>>>>>>>>>>> One distinction that helped me think about it: per-feature >>>>>>>>>>>>> headers like >>>>>>>>>>>>> X-Iceberg-Access-Delegation are directives for a specific >>>>>>>>>>>>> request - "vend >>>>>>>>>>>>> credentials for this call." A capabilities header is >>>>>>>>>>>>> different - it's a >>>>>>>>>>>>> passive, static advertisement of what the client >>>>>>>>>>>>> understands. Bundling these >>>>>>>>>>>>> into one header keeps them distinct from per-request >>>>>>>>>>>>> directives. >>>>>>>>>>>>> >>>>>>>>>>>>> The practical benefit: adding a new capability is a one-line >>>>>>>>>>>>> enum addition. >>>>>>>>>>>>> >>>>>>>>>>>>> I don't have a strong preference between the two approaches >>>>>>>>>>>>> and am happy to >>>>>>>>>>>>> revise the PR based on community consensus. Russell, to your >>>>>>>>>>>>> point about >>>>>>>>>>>>> scoping it to relevant endpoints - Sung raised the same on >>>>>>>>>>>>> the PR and I'm open >>>>>>>>>>>>> to that. >>>>>>>>>>>>> >>>>>>>>>>>>> Happy to discuss more in the upcoming sync (05/26). >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Prashant >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Fri, May 22, 2026 at 1:15 PM Russell Spitzer < >>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> This seems matched to what I thought we went over, I'm also >>>>>>>>>>>>>> not sure why the shape *capabilities : {foo, bar}* necessarily >>>>>>>>>>>>>> mean the client has to send all the capabilities with every >>>>>>>>>>>>>> request. Can >>>>>>>>>>>>>> the client just decide to send *capabilities:{foo}* to those >>>>>>>>>>>>>> endpoints that care about foo? >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, May 19, 2026 at 2:54 PM Daniel Weeks < >>>>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks for following up on this, Prashant. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> My recollection of the conversation was not to add a single >>>>>>>>>>>>>>> header that allows an arbitrary number of capabilities set, but >>>>>>>>>>>>>>> rather to >>>>>>>>>>>>>>> follow the pattern established with the >>>>>>>>>>>>>>> `x-iceberg-access-delegation' and >>>>>>>>>>>>>>> use specific headers applicable to endpoints where they would >>>>>>>>>>>>>>> apply. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'm not convinced we want to have all the capabilities >>>>>>>>>>>>>>> bundled up and sent for every request (though this might be >>>>>>>>>>>>>>> easier to >>>>>>>>>>>>>>> implement in some cases). There may be scenarios where you >>>>>>>>>>>>>>> want to >>>>>>>>>>>>>>> advertise capabilities or not depending on the client >>>>>>>>>>>>>>> configuration >>>>>>>>>>>>>>> and would make managing the set of properties more difficult. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> After reviewing the conversation, this doesn't quite >>>>>>>>>>>>>>> correspond to how we ended the discussion. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> -Dan >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Tue, May 19, 2026 at 7:06 AM Sung Yun <[email protected]> >>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks Prashant, I’m supportive of introducing a generic >>>>>>>>>>>>>>>> client capabilities header. I agree with the direction of >>>>>>>>>>>>>>>> treating this >>>>>>>>>>>>>>>> primarily as capability advertisement for compatibility and >>>>>>>>>>>>>>>> response >>>>>>>>>>>>>>>> shaping rather than as a trust mechanism. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I left a couple comments on the PR. One thing I’m wondering >>>>>>>>>>>>>>>> is whether we should discuss and define the versioning and >>>>>>>>>>>>>>>> compatibility >>>>>>>>>>>>>>>> model for capability header values as part of this proposal. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Capabilities like `read-restrictions` seem very likely to >>>>>>>>>>>>>>>> evolve over time in ways that older clients may not be able to >>>>>>>>>>>>>>>> safely >>>>>>>>>>>>>>>> consume. I’m curious whether the community thinks capability >>>>>>>>>>>>>>>> values should >>>>>>>>>>>>>>>> represent versioned compatibility contracts (like >>>>>>>>>>>>>>>> read-restrictions.v1) and >>>>>>>>>>>>>>>> whether we want to define any expectations around cross-version >>>>>>>>>>>>>>>> compatibility behavior now as we introduce the header. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sung >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 2026/05/18 18:51:52 Prashant Singh wrote: >>>>>>>>>>>>>>>> > Hi all, >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > I'd like to propose adding a new HTTP header, >>>>>>>>>>>>>>>> > X-Iceberg-Client-Capabilities, >>>>>>>>>>>>>>>> > to the REST catalog spec. This was brought up at the >>>>>>>>>>>>>>>> Read Restrictions >>>>>>>>>>>>>>>> > community sync on May 12, 2026; I'm bringing it to the >>>>>>>>>>>>>>>> broader list now >>>>>>>>>>>>>>>> > for >>>>>>>>>>>>>>>> > wider feedback. >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > PR: https://github.com/apache/iceberg/pull/16394 >>>>>>>>>>>>>>>> > Recording: https://youtu.be/b9p6mI-k-0I >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > *Context* >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > Iceberg's REST protocol keeps evolving — server-side >>>>>>>>>>>>>>>> scan >>>>>>>>>>>>>>>> > planning, remote signing, and the in-flight >>>>>>>>>>>>>>>> ReadRestrictions proposal >>>>>>>>>>>>>>>> > (PR #13879) are all features that catalogs may return >>>>>>>>>>>>>>>> but older clients >>>>>>>>>>>>>>>> > can't handle. Today there's no standard way for a client >>>>>>>>>>>>>>>> to declare >>>>>>>>>>>>>>>> > "I understand X" to the server. One direction discussed >>>>>>>>>>>>>>>> at the community >>>>>>>>>>>>>>>> > sync was to introduce a single generic capability >>>>>>>>>>>>>>>> header rather than >>>>>>>>>>>>>>>> > per-feature negotiations; this thread is to gather >>>>>>>>>>>>>>>> broader input on that >>>>>>>>>>>>>>>> > proposal. >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > *Proposal* >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > Send X-Iceberg-Client-Capabilities on every catalog >>>>>>>>>>>>>>>> request: >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > ex: X-Iceberg-Client-Capabilities: >>>>>>>>>>>>>>>> > vended-credentials,remote-signing,scan-planning >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > The Java SDK adds it via HTTPClient.baseHeaders — the >>>>>>>>>>>>>>>> same path used for >>>>>>>>>>>>>>>> > X-Client-Version and X-Client-Git-Commit-Short today. >>>>>>>>>>>>>>>> Other client >>>>>>>>>>>>>>>> > implementations (PyIceberg, Rust, Go, etc.) can mirror >>>>>>>>>>>>>>>> the same enum. >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > Initial capability set for Java SDK: >>>>>>>>>>>>>>>> > - vended-credentials >>>>>>>>>>>>>>>> > - remote-signing >>>>>>>>>>>>>>>> > - scan-planning >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > read-restrictions will be added once PR #13879 lands. >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > * Design notes worth feedback* >>>>>>>>>>>>>>>> > 1. Forward-compat hint, not security. The header is >>>>>>>>>>>>>>>> informational — >>>>>>>>>>>>>>>> > clients >>>>>>>>>>>>>>>> > can trivially spoof its value, so servers MUST NOT >>>>>>>>>>>>>>>> base trust or >>>>>>>>>>>>>>>> > authorization decisions on it. Trust establishment >>>>>>>>>>>>>>>> (mTLS, OAuth, etc.) >>>>>>>>>>>>>>>> > is out of scope. The spec parameter description >>>>>>>>>>>>>>>> states this explicitly. >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > 2. enum: constraint on the schema. Mirrors >>>>>>>>>>>>>>>> X-Iceberg-Access-Delegation. >>>>>>>>>>>>>>>> > Adding >>>>>>>>>>>>>>>> > a new capability is a one-line spec edit; generated >>>>>>>>>>>>>>>> clients can >>>>>>>>>>>>>>>> > validate >>>>>>>>>>>>>>>> > values; servers get a machine-readable closed list >>>>>>>>>>>>>>>> of recognized >>>>>>>>>>>>>>>> > values. >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > 3. Skipped for /v1/oauth/tokens. OAuth token endpoints >>>>>>>>>>>>>>>> may be served by >>>>>>>>>>>>>>>> > external IdPs (Okta, Auth0, etc.) where >>>>>>>>>>>>>>>> Iceberg-specific headers are >>>>>>>>>>>>>>>> > noise. Mirrors how X-Iceberg-Access-Delegation is >>>>>>>>>>>>>>>> handled. >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > *Links* >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > - PR: https://github.com/apache/iceberg/pull/16394 >>>>>>>>>>>>>>>> > - May 12 sync recording: https://youtu.be/b9p6mI-k-0I >>>>>>>>>>>>>>>> > - Related: PR #13879 < >>>>>>>>>>>>>>>> https://github.com/apache/iceberg/pull/13879> >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > Thanks, >>>>>>>>>>>>>>>> > Prashant >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>
