Hi all, Notes from the Aug 18 sync
Recording: https://www.youtube.com/watch?v=hYvjIcg8OsU 1. *Nested fields: forbid overlap?* --------------------------------- Sung presented industry prior art and the option space. Proposal: the server never emits policies on both an ancestor and a descendant in the same ancestry. Only Redshift supports policies on both, via integer priorities; BigQuery is leaf-only, Unity Catalog and Snowflake struct-column-only, and Trino, Hive, Impala and Ranger have no support - so the argument is interoperability, not backward compatibility. The one option that fits everyone is priority order, which would force every policy engine to emit priorities and make IRC take a position on priority semantics. Laurent brought up that conflict resolution belongs in the catalog - this is security and governance, and the catalog should be free to be as strict and prescriptive as possible. Conclusion: room seems inclined to forbid overlap. Does anyone see a case this fails to represent? 2.* No capability negotiation?* ----------------------------- Sung raised this as an open item. Fail-closed means adding any policy type later breaks older clients - requests that succeed start failing. Laurent brought up that this is a feature, not a problem: if a client cannot enforce a stronger masking policy, falling back is the wrong behaviour. Laurent and Russell brought up that if clients advertise versions and two representations of the same policy exist, a client could opt into the weaker one - TLS downgrade attacks being the precedent - so capability negotiation should not be allowed. That leaves a single representation, with the fix being an actionable error rather than a fallback. Consequence to accept explicitly: upgrade clients before servers. 3. *Implementability: POC deep dive* ---------------------------------- We deep dived and walked through the code for all three end-to-end POCs - they work, which is the main evidence the shape is right: Iceberg Generics https://github.com/apache/iceberg/pull/16131 Apache Spark https://github.com/apache/iceberg/pull/16082 Trino https://docs.google.com/document/d/1TBiHZxZHTRHoc58V7f4HJWZFAJwQlizFT7usDNhtT8s/edit Trino is the strongest signal - its access control SPI takes row filters and column projections directly, with no translation into Trino-specific restrictions needed; Iceberg's own code does the masking. Spark is weakest, having no table-level access control API we made it work with extensions. *Please do check out the POCs - if there's something specific you'd likewalked through, let us know.* 4. *Trusted clients: trust, not proof* ------------------------------------ The server needs both the subject whose restrictions are evaluated and the acting client. Laurent brought up that we are conflating *trust with proof*: proof is very hard and very expensive, and arguably unachievable - users injecting UDFs onto the Java classpath can hijack enforcement even with mTLS and confidential compute. So the best we can achieve is trust, granted in proportion to the environment, with published guidance - similar to what the OAuth working group does - on what a good client and implementer should and should not do. Built up incrementally rather than gating v1 on a complete system. open: whether client-side AuthManager supports a token exchange carrying both actor and subject identities. *mTLS works today*; on-behalf-of is not SDK-ready. Pointers to precedent welcome. *Review*: Spec https://github.com/apache/iceberg/pull/13879 API for Actions https://github.com/apache/iceberg/pull/16198 Thanks, Prashant On Tue, Jun 9, 2026 at 2:47 PM Prashant Singh <[email protected]> wrote: > Thank you all for joining the sync today (06/09) > > Summarizing the sync for folks who couldn't attend. (AI assisted) > recording : https://youtu.be/GUhCr-PnSo8 > > Where the room landed: > > 1. *Capabilities header* - leaning drop. It was effectively a versioning > workaround, and as a security mechanism it's spoofable. Trust must be > established out-of-band (mTLS / OAuth / on-behalf-of), not via a client-set > header. > 2. *V1 carries read-restrictions now, with conditional wording*. Adjusted > the spec - please have a look. Field stays optional. Let implementations > move without waiting on V2. > 3. *V2 endpoint wording*. V2 would tighten this to an unconditional MUST > — opting into V2 = committing to parse read-restrictions and fail-closed on > anything unknown. > 4. Two contracts, kept separate: > - Correctness contract (in spec): parse + fail-on-unknown. > - Trust contract (out of spec): admin/catalog decides which clients are > non-malicious and wired up to enforce. Both are required; neither replaces > the other. > 5. *No V2 capability header for now*. A V2-side header to signal "I can > enforce" was discussed for fail-fast, but the case felt narrow. Open to > revisiting. > > Open items rolling into the broader V2 conversation: > - Optional fail-fast capability header. > - V4 table support (null vs empty snapshot list). > > *Call for review:* > - Ryan's expressions-spec PR (read-restrictions has a dependency on it): > https://github.com/apache/iceberg/pull/16652 > - Read restrictions PR — updated the wording ....: > https://github.com/apache/iceberg/pull/13879 > > Best, > Prashant Singh > > On Wed, May 27, 2026 at 11:57 PM Prashant Singh <[email protected]> > wrote: > >> Hi all, >> Quick summary from yesterday's community sync for those who couldn't >> attend. Recording: https://youtu.be/-KEesN1udyY >> 1. *Client Capabilities Header (PR #16394) >> <https://github.com/apache/iceberg/pull/16394>* >> Re-discussed the generic vs. per-feature header debate from the 05/12 >> sync. Rough consensus leaned toward a single >> generic X-Iceberg-Client-Capabilities header, with these clarifications: >> - *Advisory, not authoritative*. Servers MUST NOT use the header for >> trust or authorization decisions. Trust is >> established out-of-band (mTLS / OAuth / engine identity). >> - *No versioning at the header level yet*. The fail-closed contract >> ("client MUST fail on unrecognized payload >> contents") handles forward compatibility. New incompatible behavior >> would be a new capability token, not a version >> suffix. >> - *No change to existing per-request directives*. >> X-Iceberg-Access-Delegation stays as-is for vended-credentials vs. >> remote-signing selection. Capabilities and per-request preferences are >> kept as separate mechanisms. >> >> Dan was absent for the final stretch, following up separately on the >> list before revising the PR. >> >> 2. *Expression Spec Enhancement* >> ID-based field references (needed for unbounded row filter expressions >> in read restrictions) will move forward. Spec >> write-up to follow. In the meantime, the read-restrictions spec PR will >> reference the planned change. >> >> 3. *Read Restrictions Spec - Call for Review* >> The spec PR (#13879) <https://github.com/apache/iceberg/pull/13879> is >> ready for another review pass. It will be updated with the expression spec >> reference once that >> write-up lands, but the rest of the content is in shape for review >> now. Would appreciate eyes on it. >> >> 4. *API for Action functions (PR#16198 >> <https://github.com/apache/iceberg/pull/16198>) - Call for Review* >> Generic functions module + actions wrapper is in the PR with end-to-end >> plumbing. >> >> Thanks, >> Prashant >> >> On Mon, May 18, 2026 at 12:01 PM Prashant Singh <[email protected]> >> wrote: >> >>> Hi all, >>> >>> Sharing a summary of the Iceberg Read Restrictions sync on May 12, >>> 2026, for >>> folks who couldn't attend. (As always, syncs are for discussion only.) >>> Recording: https://youtu.be/b9p6mI-k-0I >>> >>> Topics discussed >>> >>> 1. NULL handling for mask-to-default >>> >>> Question: should mask-to-default preserve NULL inputs (NULL → NULL) or >>> replace >>> them with the type-specific default (NULL → 0 / "" / epoch / etc.)? The >>> direction in the room was to NOT preserve NULL - preserving leaks the >>> existence of NULL, which can itself be sensitive information. The other >>> actions (replace-with-null, mask-alphanum, show-first-4 / show-last-4, >>> sha-256 variants, truncate-to-year / truncate-to-month) keep their >>> natural >>> NULL-preserving behavior; mask-to-default is the exception. This is >>> reflected in the most recent push to PR #13879. >>> >>> 2. Older clients without read-restriction support >>> >>> How should the spec handle clients that don't understand the >>> read-restrictions field returned by loadTable? Direction: introduce a >>> generic >>> client-capability header (X-Iceberg-Client-Capabilities) as a >>> forward-compat >>> signal, separate from per-request signals like >>> X-Iceberg-Access-Delegation. >>> Trust establishment between client and catalog stays out of scope - >>> operator/catalog-implementation concern, not spec. >>> >>> A follow-up PR (#16394) has been raised to add the header and the >>> parameter >>> component; a separate [DISCUSS] thread is being raised in parallel. >>> >>> 3. Identity propagation (Trino, Spark, etc.) >>> >>> Surfaced briefly. Folks acknowledged identity propagation across >>> multi-tenant >>> query engines is a real problem, but it's orthogonal to the spec - it's >>> a catalog / auth-manager concern. Not in scope for #13879. >>> >>> 4. Actions API placement >>> >>> Continued discussion on where action definitions live in the Java API. >>> Direction: ship as utility functions in the api module, mirroring the >>> existing Transform pattern. Don't expose a top-level "Action" type in >>> public Java API — keep "Action" as a REST-spec construct only. >>> >>> >>> Follow-ups >>> >>> - PR #13879 (FGAC read restrictions): incorporated the discussion >>> above and >>> continue iterating on reviewer feedback as they come (please take >>> another look). >>> - PR #16394 (X-Iceberg-Client-Capabilities header): raised, [DISCUSS] >>> thread incoming. >>> - Next sync: bi-weekly cadence - see the sync notes doc for date. >>> >>> Links >>> >>> - Recording: https://youtu.be/b9p6mI-k-0I >>> - Sync notes doc: >>> https://docs.google.com/document/d/1iGNydKY7XT1N5Nz056vDPM0P8v0MFymGqNtOlUGUp-c/edit >>> - PR #13879: https://github.com/apache/iceberg/pull/13879 >>> - PR #16394: https://github.com/apache/iceberg/pull/16394 >>> - DISCUSS: >>> https://lists.apache.org/thread/xlqx6k7g625p38bxxy141wt02d00w2h4 >>> >>> Thanks to everyone who joined. >>> >>> Prashant >>> >>> On Mon, Apr 27, 2026 at 9:05 AM Prashant Singh <[email protected]> >>> wrote: >>> >>>> Thank you all for joining the syncs so far! >>>> >>>> After much discussion and debate, we've narrowed things down to a >>>> final list of 9 predefined actions. >>>> >>>> Spec update: I updated the spec PR [1] last week - bumping it here as >>>> well. Please take a look when you get a chance! >>>> >>>> POC progress: I've been prototyping with both a SQL client and a >>>> NoSQL client. The Apache Spark integration [2] fits cleanly. For NoSQL, I'm >>>> working on the Iceberg Generics reader [3] and py-iceberg in parallel. >>>> >>>> Upcoming sync - *04/28* agenda: >>>> 1. Default mask values per type (I've added some initial proposals to >>>> kick off the conversation) >>>> 2. How and where to put Actions in the iceberg java, where to keep it >>>> to make engine integration seamless and reusable (ofc engines are free to >>>> implement their own) >>>> >>>> Past sync notes and recordings are available here [4]. >>>> >>>> Looking forward to seeing everyone at the next sync. Thank you for >>>> all your valuable feedback! >>>> >>>> Best, >>>> Prashant Singh >>>> >>>> [1] https://github.com/apache/iceberg/pull/13879 >>>> [2] https://github.com/apache/iceberg/pull/16082 >>>> [3] https://github.com/apache/iceberg/pull/16131 >>>> [4] >>>> https://docs.google.com/document/d/1iGNydKY7XT1N5Nz056vDPM0P8v0MFymGqNtOlUGUp-c/edit?tab=t.0#heading=h.tevndn85fps >>>> >>>> On Mon, Mar 23, 2026 at 7:22 PM Prashant Singh < >>>> [email protected]> wrote: >>>> >>>>> Hi all, >>>>> >>>>> Here is the summary for *Iceberg Read Restrictions Sync (03/17)*. >>>>> >>>>> Recording: https://www.youtube.com/watch?v=LObBU3r_GXg >>>>> Design Doc: >>>>> https://docs.google.com/document/d/1D0RcjmiYk0mKtCGak_MyG2dpyj6u19HQ/edit >>>>> >>>>> *Execution Order (agreed)* >>>>> 1. Authorization predicates and row filters on unmasked data >>>>> 2. Column masks applied >>>>> 3. User query filters on masked data >>>>> >>>>> This prevents point attacks where users craft filters to deduce >>>>> masked values. If user filters get pushed down before masking, that's the >>>>> engine's >>>>> responsibility to handle correctly and not let it be open to >>>>> exploitation. >>>>> >>>>> *Masking Functions* >>>>> - Mask to Default — type-specific constant values; preserves schema >>>>> shape for downstream BI tools. Essential for non-nullable columns where >>>>> nulls would break >>>>> joins or engine operations. >>>>> - Replace with Null — kept as a separate option for optional >>>>> columns. Serves a different policy intent than mask-to-default. >>>>> - Alphanumeric Masking — preserves punctuation, redacts >>>>> letters/numbers with Xs. >>>>> - Show First/Last Four — partial visibility for identifiers like >>>>> SSNs. For short strings (<4 chars), the team favored a "dumb mask, smarter >>>>> admin" approach - >>>>> admins should pick appropriate masks rather than building complex >>>>> padding/hashing into the function itself. >>>>> - Date Truncation — truncate to year or month (day/month replaced >>>>> with 01). Standard and undisputed. >>>>> - SHA-256 Hashing — two approaches discussed, both are needed: >>>>> - Query-local (random salt): allows joins within a single query >>>>> but not across sessions. >>>>> - Global stable hash: consistent across sessions for semantic >>>>> layering. >>>>> - We plan to continue discussing this in upcoming syncs. >>>>> >>>>> *Action Items* >>>>> - I will research how Apache Ranger and Oracle handle short-string >>>>> masking before finalizing the spec >>>>> - Check with the BigQuery team on why some of their masking >>>>> behaviors are the way they are (Thanks Talat) >>>>> >>>>> Looking forward to seeing you all in the next sync! >>>>> >>>>> Best, >>>>> Prashant Singh >>>>> >>>>> On Fri, Feb 6, 2026 at 5:35 PM Prashant Singh < >>>>> [email protected]> wrote: >>>>> >>>>>> Thank you everyone for joining the call ! >>>>>> Please find the recording attached [1] >>>>>> On a high level we discussed the following : >>>>>> - *Deny list vs allow list *: >>>>>> what does the client assume if a given column is not part of the >>>>>> required column projection, is it allowed to see that column or not. >>>>>> The consensus seemed to be having *DENY* as a representation, >>>>>> considering the allowlist can be huge for a very wide table. This does >>>>>> not >>>>>> dictate what catalog should be stored while defining its policy, some >>>>>> catalogs have both ALLOW and DENY. >>>>>> Essentially what this DENY list means is what a client *should* >>>>>> expect when consuming Policy evaluation results. >>>>>> Note: *DENY* is generally not recommended since it can cause issues >>>>>> specially lets say a column being added and user getting access to it >>>>>> automatically but in this case since the policy evaluation results are >>>>>> coupled with the loadTable request. >>>>>> so we compute the *DENY* list considering the latest schema that was >>>>>> present at the time when it was loaded. Any new column being added to >>>>>> schema will create a new iceberg schema and clients will not have access >>>>>> to >>>>>> it. >>>>>> I will update the PR soon with this recommendation (request you all >>>>>> to please participate) >>>>>> >>>>>> - *Why Policy Evaluation over Policy Exchange* : we discussed this >>>>>> for a bit and touched why community has been considering this approach >>>>>> mostly due to multitude of policy definition / dialects out there and >>>>>> this >>>>>> is equivalent to vended creds which done based on the grants the users >>>>>> has >>>>>> and >>>>>> defines clear instructions in a portable way to be enforced cross >>>>>> engine >>>>>> >>>>>> - *Predefined masks over dynamic mask* : The spec is trying to have >>>>>> some set of predefined actions mostly inspired by Apache Ranger and there >>>>>> was a discussion / debate around it and there seemed to support for >>>>>> having >>>>>> both rather than choosing one of them specially for masks such as >>>>>> nullify / >>>>>> hash etc. >>>>>> >>>>>> - *Expression Expansion* : Iceberg expression to be more than >>>>>> predicates and its expansion to have UDF references (Iceberg UDF spec got >>>>>> ratified recently), Ryan said he will be taking a look into it soon >>>>>> (thank >>>>>> you so much !), we debated more on dialects etc additionally from UDF pov >>>>>> >>>>>> We plan to keep this discussion going. I see some new feedback on the >>>>>> spec PR [2] will address them and have them added to be discussed more ! >>>>>> >>>>>> [1] https://www.youtube.com/watch?v=_wKszzNtP48 >>>>>> [2] >>>>>> https://github.com/apache/iceberg/pull/13879#discussion_r2760180338 >>>>>> >>>>>> Best, >>>>>> Prashant Singh >>>>>> >>>>>> On Mon, Feb 2, 2026 at 4:47 PM Prashant Singh < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Bumping the thread ^^ >>>>>>> >>>>>>> Looking forward to seeing you all tomorrow >>>>>>> Meeting details: Tuesday, Feb 3⋅9:00 – 10:00am Pacific (recurring >>>>>>> biweekly): https://meet.google.com/gwy-jxos-jif >>>>>>> >>>>>>> I proactively added some comments in the agenda from the spec PR : >>>>>>> https://github.com/apache/iceberg/pull/13879 >>>>>>> >>>>>>> Best, >>>>>>> Prashant Singh >>>>>>> >>>>>>> On Tue, Jan 20, 2026 at 1:58 PM Prashant Singh < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> Iceberg rest catalog returning policy evaluation results for fine >>>>>>>> grained access control enforcement, has been discussed a couple of >>>>>>>> times in >>>>>>>> the past as well as recently in the community, we pretty much have a >>>>>>>> broader agreement on what we wanna do at a higher level but there are >>>>>>>> still >>>>>>>> some open questions and details to hash out details for the spec to get >>>>>>>> ratified [1]. >>>>>>>> >>>>>>>> I wanted to propose a dedicate sync for discussing these and >>>>>>>> closing them, the time slot, we got was (Thanks Steven) : >>>>>>>> >>>>>>>> *Biweekly starting from Feb 3 (9:00 am - 10:00 am PST),* you can >>>>>>>> see the same in your dev event calendar if you subscribe to "Iceberg >>>>>>>> Dev >>>>>>>> Events". >>>>>>>> >>>>>>>> Please do join, we will keep the sync recorded and capture notes on >>>>>>>> the doc [2] for this sync. >>>>>>>> >>>>>>>> [1] https://github.com/apache/iceberg/pull/13879 >>>>>>>> [2] >>>>>>>> https://docs.google.com/document/d/1iGNydKY7XT1N5Nz056vDPM0P8v0MFymGqNtOlUGUp-c/edit?tab=t.0#heading=h.tevndn85fps >>>>>>>> >>>>>>>> Best, >>>>>>>> Prashant Singh >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>
