I had a chance to catch up on the discussion from Tuesday and I think the justification Sung laid out was good.
My original concern was the motivation for not addressing the conflict which centered primarily on: 1) We lack examples (other than Redshift), so we don't know what position to take 2) It's acceptable because it's backwards compatible, so we can fix it later #2 isn't true, and #1 felt like we were just leaving a gap in the proposal that could be addressed by taking a more opinionated stance. Based on my understanding, the expectation is that the catalog should either deconflict (by supplying appropriate substitutions/handle the precedence conflicts) or fail. (I would also have been fine defining an order in which they are applied, but I understand the motivation for keeping the client simple). I'm fine with putting the complexity on the catalog and disallowing the overlap. -Dan On Mon, Aug 10, 2026 at 1:16 PM Russell Spitzer <[email protected]> wrote: > I actually think we already have precedent for how to handle situations > like this in the spec (TLDR; (1 - Disallow Ambiguous States)) from our > earlier discussions with default values. > > For those who remember <https://github.com/apache/iceberg/pull/12841>, we > spent a lot of time discussing how parent and child defaults should > interact when both could supply a value for the same leaf. We discussed all > the various options almost *identically *to what we are discussing now. > After much back and forth we just decided to disallow the ambiguous > configuration. Struct defaults are only null or {}, and field defaults live > on the fields themselves. Simple, with no extra rules that a client can > mess up. > > I think we are in basically the same situation here. Overlapping > projections on a nested container and one of its descendants don't have an > obvious resolution. Similar to the default case, we could try to establish > client rules that every client must implement and could potentially > implement incorrectly, or we could simply declare that state is invalid. > > So I am +1 on Option A / Nevin’s contract (1): catalogs MUST NOT emit > overlapping nested projections, and readers MUST fail closed if they > receive them. Similar to a table with multiple overlapping > initial-defaults, the reader should simply throw an error and state that > the input is not valid. > > > On Fri, Aug 7, 2026 at 3:10 PM Sung Yun <[email protected]> wrote: > >> > While changing this requirement in the future is structurally >> compatible, it is not behaviorally compatible. >> >> Dan, that is a fair point, and I withdraw my claim that relaxing the >> constraint later would be entirely backward-compatible. It would not be, >> given the data access regression you described. >> >> There is a real behavioral consequence whenever a new capability is >> introduced to ReadRestrictions, and I think the community would benefit >> from working out what the cost of introducing new capabilities is, beyond >> the immediate scope of nested policy handling. I would rather >> compartmentalize this discussion to its own thread than litigate the >> question on nested policy conflict resolution based on an uncalibrated >> weighting of that cost. I will raise it at the 8/18 sync. >> >> Setting that aside, Nevin is right that this is partly a question of >> which tradeoffs we value. I think Option A holds up well on >> interoperability with existing policy engines. I put together a survey of >> how current systems handle nested policies to help with that analysis, and >> I am happy to walk through it together during the 8/18 sync [1]. >> >> Here is a short summary. Every surveyed system resolves to a single >> policy per field value, and no system composes more than one masking >> transforms over one value. Most make an ancestor-descendant overlap >> un-authorable, either by excluding the container from masking or by >> permitting exactly one attachment point per column. Redshift is the only >> system that permits both to be attached, and it requires >> administrator-authored priorities and resolves to a single winner before >> the reader is involved. >> >> Two additional properties seem worth weighing in the discussion. First, >> whether moving conflict resolution from the policy engine to the client is >> valuable to the community, and whether a defined semantic in Iceberg can do >> that effectively. Second, whether a composition rule, where transforms >> apply on top of each other, is a capability we want to introduce without >> industry precedent. >> >> As Nevin noted, we have not dug into what those semantics could be. I >> hoped laying them out would help move the discussion forward. >> >> Sung >> >> [1] https://s.apache.org/0w1mu >> >> On 2026/08/07 00:00:55 Nevin Zheng wrote: >> > Hi all, >> > >> > Thanks everyone for the thoughtful discussion. I wanted to offer a >> > slightly different framing that may help us align on the decision >> > we're making. >> > >> > I see two dimensions: >> > >> > - Catalog: MUST NOT send overlaps vs. MAY send overlaps >> > - Reader: fail closed vs. apply defined semantics >> > >> > That gives us four possible contracts: >> > >> > (1) MUST NOT + fail closed — strict contract >> > (2) MUST NOT + defined semantics — defensive fallback >> > (3) MAY + fail closed — partial interoperability >> > (4) MAY + defined semantics — supported semantics >> > >> > My read is that the current proposal is (1). I would place Dan's >> > position as a version of (4), with the concrete semantics still to be >> > defined. >> > >> > At this level, I think this is partly a values/tradeoff decision >> > rather than a question with one technically correct answer. We can >> > value simplicity and fail-closed behavior, or place more value on >> > expressiveness and interoperability. Until we align on those >> > priorities, or find a compromise, I think we may continue disagreeing >> > on the lower-level mechanics. >> > >> > There are of course finer questions around precedence/composition, >> > security, compatibility, implementation complexity, and extensibility. >> > I think choosing the contract at this level first could help clarify >> > those discussions. >> > >> > Another useful lens may be the critical user journey: when overlapping >> > projections occur, who bears the cost? With (1), the query fails >> > safely and the operator/end user has to resolve the configuration. >> > With (4), the query can continue, but Iceberg readers take on the >> > complexity of correctly interpreting the overlap. >> > >> > My preference is (1). Catalogs should produce unambiguous >> > restrictions, and readers should fail closed with a clear error when >> > they don't. Catalogs will inevitably have bugs or misconfigurations; I >> > prefer a loud, safe, diagnosable failure over requiring every reader >> > to interpret ambiguous policy. >> > >> > This also gives us a safe failure mode if we expand the semantics >> > later: older implementations may fail, but they fail closed. >> > >> > That said, I'm comfortable with another coherent contract if the >> > community prefers its tradeoffs and we specify and implement it well. >> > >> > I hope this framing is useful. Please let me know if I've >> > misrepresented the design space or anyone's position. >> > >> > Nevin >> > >> > On Thu, 6 Aug 2026 16:17:39 -0700, Steven Wu <[email protected]> >> wrote: >> > > > This is true regardless. The client is not involved in resolving >> anything. >> > > >> > > I am only discussing main question of this thread: failing the query >> vs resolving precedence rules at client/reader-side for nested fields with >> conflicting policies? >> > > >> > > If a projection targets a nested-typed field (struct, list, or map), >> other projections in the same ReadRestrictions must not target any nested >> field-id (struct subfields, list elements, or >> > > >> > > map keys/values) at any depth. This specification does not define how >> such actions combine. >> > > >> > > A reader that receives such a response must fail the query. >> > > >> > > > Relaxing the requirement is not backward compatible for clients. >> > > >> > > If we relax this in the future, here is how we can do so compatibly. >> > > >> > > A reader that receives such a response must fail the query. >> > > >> > > --> >> > > >> > > A reader that receives such a response may fail the query or may >> apply the following precedence rule to deconflict. >> > > >> > > > we're taking a shortcut by not implementing full schema support for >> this feature >> > > >> > > What does the "full schema support" mean in this context of >> conflicting policies for nested fields? >> > > >> > > Thanks, >> > > Steven >> > > >> > > On Thu, Aug 6, 2026 at 2:43 PM Daniel Weeks <[email protected]> >> wrote: >> > > >> > > Current spec requires clients to fail if the catalog server returns >> conflicting policies. If we relax this requirement in the future with >> precedence rules for client-side de-conflicting, older clients would still >> fail. I don't see it as "an unpredictable behavior change" for older >> clients. I would also the relaxation would probably use the wording of MAY >> (not MUST) applying these precedence rules to deconflict. >> > > >> > > Relaxing the requirement is not backward compatible for clients. That >> would be equivalent to saying that we can make a required field optional >> because the clients need to validate required fields so it's ok if they >> break. Backward compatibility is about not breaking clients, not making it >> ok to break clients. >> > > >> > > I heard comments that this rule is very easy for the policy engine to >> enforce . . . >> > > >> > > Handling nested fields doesn't change how hard it is for a policy >> engine to enforce. They can always fallback to not allowing it (they would >> already be required to detect these cases). >> > > >> > > If we ask clients to resolve conflicting policies >> > > >> > > This makes it sound like we're shifting the burden to the client, but >> that's not the case. We are just defining an order in which they are >> applied. Policy engines are required to resolve conflicts. The client just >> applies the restrictions in a well defined order. >> > > >> > > I would let the policy engine either enforce that >> conflicting/ambiguous policies are not accepted or perform the >> deconflicting/precedence rules before returning to the client. >> > > >> > > This is true regardless. The client is not involved in resolving >> anything. >> > > >> > > The issue here is that we're taking a shortcut by not implementing >> full schema support for this feature and presenting it as a reasonable path >> based on a misconception that it's backward compatible. >> > > >> > > -Dan >> > > >> > > On Thu, Aug 6, 2026 at 11:44 AM Steven Wu <[email protected]> >> wrote: >> > > >> > > > Changing that in the future would break older clients that validate >> and result in an unpredictable changes in behior of clients that assume >> correctness from the catalog. >> > > >> > > Current spec requires clients to fail if the catalog server returns >> conflicting policies. If we relax this requirement in the future with >> precedence rules for client-side de-conflicting, older clients would still >> fail. I don't see it as "an unpredictable behavior change" for older >> clients. I would also the relaxation would probably use the wording of MAY >> (not MUST) applying these precedence rules to deconflict. >> > > >> > > I agree with Prashanth and Sung that it is hard for Iceberg to force >> a specific behavior considering the divergent behaviors in the industry. >> > > >> > > As I mentioned during the sync, I also see it the same as the other >> requirement we put on the catalog server. I heard comments that this rule >> is very easy for the policy engine to enforce. If we ask clients to resolve >> conflicting policies for nested fields with precedence rules, I can argue >> the same thing here for this rule. Can't we also ask clients to define >> resolution precedence with the projection policies? >> > > >> > > 3. A server must not return more than one projection for the same >> field-id >> > > in required-column-projections. If a duplicate field-id appears, the >> reader >> > > must fail the query. >> > > >> > > Since IRC doesn't define policy management APIs and behaviors, I >> would let the policy engine either enforce that conflicting/ambiguous >> policies are not accepted or perform the deconflicting/precedence rules >> before returning to the client. Hence, i agree with the current spec >> writing that clients must fail in the case of conflicting policies with >> nested fields. >> > > >> > > On Thu, Aug 6, 2026 at 9:40 AM Daniel Weeks <[email protected]> >> wrote: >> > > >> > > I'd like to clarify the point on backward compatibility claims here >> first as it indicates a dangerous misunderstanding of compatibility in the >> REST protocol. The statement that "Forbidding is reversible; precedence >> isn't. Relaxing a MUST NOT later is backward-compatible" is incorrect, and >> we need to be very careful regarding backward compatibility. It also >> creates a false sense of security by deferring this, as it implies the >> issue can be addressed later without consequence. >> > > >> > > While changing this requirement in the future is structurally >> compatible, it is not behaviorally compatible. Given the current spec >> wording, a responsible client should validate that they do not receive a >> invalid set of redactions from the catalog as this is prohibited by the >> spec and there's no guidance on how the client should apply them. Changing >> that in the future would break older clients that validate and result in an >> unpredictable changes in behior of clients that assume correctness from the >> catalog. This is neither safe nor backward compatible. >> > > >> > > I think addressing this now is well within scope and will prevent >> unnecessary future protocol evolution. Regardless of what type of >> precedence we define, we already know that some policies (e.g. Redshift) >> that will express more than we will likely include in the protocol. It's >> also likely that different policy systems will have conflicting views on >> this matter. Since we're already taking an opinionated stance and levying >> requirements on the catalog implementation to align their policy with the >> protocol, this isn't any more complicated than prohibiting conflicting >> policy. >> > > >> > > Read restrictions have a number of unexpected cliffs regarding how >> policy changes affect what clients can consume (e.g. ABAC policies that >> update from simple redaction to more contextual redaction) and this is just >> one more scenario that limits interoperability. >> > > >> > > The backward compatibility issue further highlights why we need a >> solid reference implementation as I think it would make the problem clear. >> I don't want to repeat the challenges we had with scan/plan where we >> released the protocol with issues that we, fortuntately, were able to >> address as the reference implementation was built. >> > > >> > > -Dan >> > > >> > > On Thu, Aug 6, 2026 at 7:56 AM Sung Yun <[email protected]> wrote: >> > > >> > > Thanks Prashant for doing such extensive research on this topic. >> > > >> > > I believe the industry divergence on how nested type policies are >> handled is a good indicator that we should hold off on forcing a standard >> for nested conflict resolution right now. Defining a standard is a highly >> sensitive design decision that deserves its own focused, long-term >> community discussion rather than being rushed. >> > > >> > > During our syncs, we’ve discussed several potential evaluation >> models, each with distinct trade-offs: >> > > >> > > - Outermost wins (single policy is enforced) >> > > >> > > - Innermost wins (single policy is enforced) >> > > >> > > - Innermost evaluated first (pipelined and layered sequentially) >> > > >> > > - Outermost evaluated first (pipelined and layered sequentially) >> > > >> > > These models carry different security implications, and hence I'm in >> favor of choosing option A (forbidding overlapping nested policies and >> failing-closed on the client) to not rush this discussion and unblock >> ReadRestrictions today. >> > > >> > > If the community later decides to define a precedence standard, >> relaxing this rule is entirely backward-compatible. On the client/reader >> side, we would simply be introducing a new handling capability for what >> used to be an invalid response. On the server/catalog side, the timeline >> remains entirely in the catalog administrator's control because the new >> behavior only triggers once they choose to update their catalog >> implementations to emit multiple nested policies. >> > > >> > > One final observation on the industry examples: >> > > >> > > Even in Redshift, which is the only example where multiple policies >> can technically be registered on a nested path, the engine treats >> overlapping policies as a conflict. It resolves this conflict by enforcing >> only the single, highest-priority policy at query time. In practice, this >> "highest-priority wins" model is highly consistent with Option A. It >> already assumes that the system must ultimately resolve the conflict down >> to a single, clean policy rather than attempting to chain and layer >> multiple policies sequentially. I believe that this can be the >> responsibility of the policy engine, rather than becoming a preemptive >> standard in the ReadRestrictions spec. >> > > >> > > All in all, I strongly support keeping the current restrictive >> instruction in the spec (Option A) for the initial release and updating the >> words with normative RFC 2119 terminology (using MUST NOT and MUST fail) to >> clearly define the behavior on the client and server side. [1] >> > > >> > > Sung >> > > >> > > [1] >> https://github.com/apache/iceberg/pull/13879#discussion_r3716145795 >> > > >> > > On 2026/08/04 23:25:40 Prashant Singh wrote: >> > > >> > > > Hi all, >> > > >> > > > >> > > >> > > > One open thread on the Read Restrictions spec PR [1] is the last >> item I'd >> > > >> > > > like to settle before calling a vote [2]. The last sync leaned >> towards the >> > > >> > > > restrictive option, but there was an ask by Dan that the >> alternative be >> > > >> > > > explored properly first, so I'm bringing it here. >> > > >> > > > >> > > >> > > > The question >> > > >> > > > >> > > >> > > > required-column-projections binds an action to a field-id, and a >> nested >> > > >> > > > type has a field-id for the container and separate ones beneath it >> (struct >> > > >> > > > subfields; list element; map key and value). >> > > >> > > > >> > > >> > > > Concretely, with 2: address struct<3: street string, 4: city >> string>: >> > > >> > > > >> > > >> > > > - field-id: 2 action: mask-to-fixed-value >> > > >> > > > >> > > >> > > > - field-id: 4 action: replace-with-null >> > > >> > > > >> > > >> > > > mask-to-fixed-value overwrites everything beneath the container, so >> it >> > > >> > > > yields "XXXXXXXX" for each string leaf. Outer-most-wins therefore >> > > >> > > > returns {"street": >> > > >> > > > "XXXXXXXX", "city": "XXXXXXXX"}; inner-most-wins returns {"street": >> null, >> > > >> > > > "city": "XXXXXXXX"}, though the catalog asked for the whole struct >> to be >> > > >> > > > masked. >> > > >> > > > >> > > >> > > > if a catalog returns two projections (one on the outer container >> and one on >> > > >> > > > a field-id below it), how should clients handle it? >> > > >> > > > >> > > >> > > > (A) forbid the overlap; the existing fail-closed rule then makes a >> reader >> > > >> > > > >> > > >> > > > that receives one fail the query. The spec PR currently says: >> > > >> > > > >> > > >> > > > If a projection targets a nested-typed field (struct, list, or >> map), other >> > > >> > > > projections in the same ReadRestrictions must not target any nested >> > > >> > > > field-id (struct subfields, list elements, or >> > > >> > > > >> > > >> > > > map keys/values) at any depth. This specification does not define >> how such >> > > >> > > > actions combine. >> > > >> > > > >> > > >> > > > A reader that receives such a response must fail the query. >> > > >> > > > >> > > >> > > > (B) define precedence (e.g. outer-most wins) and allow it >> > > >> > > > >> > > >> > > > Why current spec PR choose option A >> > > >> > > > >> > > >> > > > No semantics to borrow. There are different behaviors in the current >> > > >> > > > industry practice >> > > >> > > > >> > > >> > > > - >> > > >> > > > >> > > >> > > > BigQuery: >> > > >> > > > - >> > > >> > > > >> > > >> > > > "Policy tags can't be applied to columns that use the STRUCT data >> > > >> > > > type" [3]; >> > > >> > > > - >> > > >> > > > >> > > >> > > > Redshift both experience based on data-types: >> > > >> > > > - >> > > >> > > > >> > > >> > > > "You can only apply masking policies to scalar values on the SUPER >> > > >> > > > path" [4] rejects the overlap. >> > > >> > > > - >> > > >> > > > >> > > >> > > > Redshift is the only one that lets both be expressed, and it calls >> > > >> > > > the pair a conflict, resolved by an admin-supplied priority [5] >> > > >> > > > - >> > > >> > > > >> > > >> > > > Platforms like Snowflake, Oracle, DB2, Vertica, Trino, Hive, Impala >> and >> > > >> > > > Ranger define no parent-vs-nested resolution at all, and where the >> question >> > > >> > > > has been raised it has stayed open: RANGER-3525 since 2021 [6], >> HIVE-22823 >> > > >> > > > [7], trino#16964 [8]. >> > > >> > > > - >> > > >> > > > >> > > >> > > > SQL:2016 defines nothing - <privilege column list> is a flat column >> name >> > > >> > > > list. >> > > >> > > > >> > > >> > > > >> > > >> > > > >> > > >> > > > Forbidding is reversible; precedence isn't. Relaxing a MUST NOT >> later is >> > > >> > > > backward-compatible: invalid responses become valid, and no reader >> changes >> > > >> > > > what it returns for a response that was already legal. If we define >> > > >> > > > outer-most-wins now and later want the inner action to contribute, >> we've >> > > >> > > > silently changed the values a compliant reader returns for the same >> > > >> > > > response. >> > > >> > > > >> > > >> > > > On the future-actions concern: I don't think the prohibition limits >> any >> > > >> > > > action. A future action can target a container, or a nested field - >> just >> > > >> > > > not both in one response, which is the only combination with no >> defined >> > > >> > > > meaning today. >> > > >> > > > >> > > >> > > > Asks >> > > >> > > > >> > > >> > > > 1. Opinions on option A (forbid) vs B (define precedence) >> > > >> > > > >> > > >> > > > 2. If (B), is outer-most-wins the rule — and are we comfortable that >> > > >> > > > >> > > >> > > > nothing in the current action set can show it’s the right choice? >> > > >> > > > >> > > >> > > > I am looking forward to hear what other people think and if they >> have >> > > >> > > > objection to the current proposed spec >> > > >> > > > >> > > >> > > > We time-boxed this to the next sync (Aug 18) in the last one. If >> there's no >> > > >> > > > further input by then, we will keep the current restrictive wording >> and say >> > > >> > > > so in this thread rather than treat silence as agreement - the >> precedence >> > > >> > > > question should be closed explicitly, not by default. And if a use >> case >> > > >> > > > turns up after we ship, Option A lets us relax the rule >> compatibility, >> > > >> > > > which is most of why the PR takes it. Notes from sync [10]. >> > > >> > > > >> > > >> > > > Thanks, >> > > >> > > > >> > > >> > > > Prashant >> > > >> > > > >> > > >> > > > References >> > > >> > > > >> > > >> > > > [1] Spec PR: https://github.com/apache/iceberg/pull/13879 >> > > >> > > > >> > > >> > > > [2] Review thread: >> > > >> > > > https://github.com/apache/iceberg/pull/13879#discussion_r3211673048 >> > > >> > > > >> > > >> > > > [3] BigQuery, column data masking: >> > > >> > > > https://cloud.google.com/bigquery/docs/column-data-masking-intro >> > > >> > > > >> > > >> > > > [4] Redshift, DDM with SUPER paths: >> > > >> > > > https://docs.aws.amazon.com/redshift/latest/dg/t_ddm-super.html >> > > >> > > > >> > > >> > > > [5] Redshift, DDM policy hierarchy: >> > > >> > > > https://docs.aws.amazon.com/redshift/latest/dg/t_ddm-hierarchy.html >> > > >> > > > >> > > >> > > > [6] RANGER-3525: https://issues.apache.org/jira/browse/RANGER-3525 >> > > >> > > > >> > > >> > > > [7] HIVE-22823: https://issues.apache.org/jira/browse/HIVE-22823 >> > > >> > > > >> > > >> > > > [8] trinodb/trino#16964: >> https://github.com/trinodb/trino/issues/16964 >> > > >> > > > >> > > >> > > > [9] IMPALA-15130: >> https://issues.apache.org/jira/browse/IMPALA-15130 >> > > >> > > > >> > > >> > > > [10] Note from ReadRestrictions syncs: >> > > >> > > > >> https://docs.google.com/document/d/1iGNydKY7XT1N5Nz056vDPM0P8v0MFymGqNtOlUGUp-c/edit?tab=t.0#heading=h.h3v6cqyijrx >> > > >> > > > >> > >> >
