jerryshao commented on code in PR #13546: URL: https://github.com/apache/gravitino/pull/13546#discussion_r4143728238
########## design-docs/tag-based-read-restrictions-for-iceberg-rest.md: ########## @@ -0,0 +1,562 @@ +<!-- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Design: Tag-Based Read Restrictions for Iceberg REST + +## Background + +Gravitino can associate governance policies with tags and resolve those policies for metadata +objects. Existing authorization decides whether a subject may read a table, but it cannot restrict +which rows or column values are visible after access is granted. + +The Iceberg REST specification defines `read-restrictions` in a load-table response. A conforming +reader applies a required row filter and required column projections before returning data. This +provides a standard enforcement boundary for portable restrictions. + +This design adds tag-based row-filter and column-mask policies and resolves them into Iceberg REST +`read-restrictions`. It is disabled by default and requires an explicit client capability +declaration while reader support is maturing. + +## Goals + +1. Define typed row-filter and column-mask policy content. +2. Select policies through the policy-on-tag model and effective tags. +3. Support small, deterministic expression syntax for row predicates and mask selection. +4. Bind authored expressions to the authenticated subject and an Iceberg table schema. +5. Return only closed, typed Iceberg expressions and standard Iceberg mask actions. +6. Fail closed when an applicable restriction cannot be resolved or enforced. +7. Provide an end-to-end path that can later move to official Iceberg runtime types without + changing policy content. +8. Reserve a fail-closed extension model for UDF references. + +## Non-Goals + +1. Replacing table-level authorization or granting access through a read-restriction policy. +2. Executing arbitrary expressions or UDFs through the first Iceberg REST implementation. +3. Supporting nested-field masks, roles, identity attributes, nested groups, or general attribute + expressions in the first version. +4. Defining a new direct policy-to-metadata-object association model. +5. Guaranteeing that a reader without Iceberg read-restriction support can safely read governed + tables. + +## Architecture + +The resolution path is: + +```text +Policy and tag administration + -> effective tags for a table and its columns + -> effective row-filter and column-mask policies + -> subject and schema binding + -> canonical Iceberg read restrictions + -> loadTable response + -> trusted Iceberg reader enforcement +``` + +Authorization runs before restriction resolution. A restriction only reduces data visible through +an already-authorized read. It never changes an authorization deny into an allow. + +The first implementation runs only where the Iceberg REST service has the trusted end user in its +request context and can resolve Gravitino policies, tags, users, and groups directly. + +## Policy Model + +This design introduces two built-in policy types: + +| Policy type | Evaluation target | Effect | +| --- | --- | --- | +| `system_row_filter` | Table | Retains only rows matching one resolved predicate. | +| `system_column_mask` | Top-level column | Replaces visible values using one Iceberg mask action. | + +Policies are associated with tags. The existing policy-on-tag resolver selects enabled policies +from effective tags. Row-filter resolution consumes effective policies for a table. Column-mask +resolution consumes effective policies for each top-level column. Direct policy +associations are not used. + +First-version policy content stores one expression. It does not store parser names, +action-vocabulary names, resolved subjects, group membership snapshots, table schemas, field IDs, +or serialized load-table responses. The built-in policy type determines how the expression is +parsed. + +Policy selection and policy effect are separate. Tags and policy-on-tag selectors decide whether a +policy is applicable; a row-filter or column-mask definition states what the selected policy does. +The effect content has no `rules` list and no `when` field. Conditional filter results are written +inside one restricted Rego expression. This follows the Databricks ABAC pattern in which a single +row-filter UDF can use conditional logic to return its Boolean result, while policy applicability +and function input binding remain separate. A future principal-aware selector belongs to the +selection model, not inside a restriction definition. + +### Row-filter content + +The content of a row-filter policy contains exactly one `expression`. + +```json +{ + "name": "restrict_orders", + "comment": "Auditors see US orders; other users see their own orders", + "policyType": "system_row_filter", + "enabled": false, + "content": { + "expression": "filter := col(\"region\") == \"US\" if is_group_member(\"auditors\") else := col(\"owner\") == session_user()" + } +} +``` + +The expression is one complete rule whose result is the row predicate. In the example, the Rego +assignment means “if the subject is an auditor, then use the region predicate; otherwise use the +owner predicate.” It does not select another policy rule. The expression may use trusted request +context and row values in either conditions or results. The first policy-on-tag selector version +does not select by principal, so subject-dependent filtering can remain inside this one expression. + +### Column-mask content + +The content of a column-mask policy also contains exactly one `expression`. Its result is an +Iceberg mask action name. + +```json +{ + "name": "mask_phone_number", + "comment": "Auditors see the final four characters; other users see null", + "policyType": "system_column_mask", + "enabled": false, + "content": { + "expression": "mask := action(\"show-last-4\") if is_group_member(\"auditors\") else := action(\"replace-with-null\")" + } +} +``` + +The rule result is an explicit `action("name")` value, and each condition must use the context-only +expression subset. A bare string is not a mask action and is invalid. A condition cannot contain +`col(...)` because an Iceberg projection selects one action for the complete column, not a different +action per row. Conditional results are resolved before the response is serialized. If more than +one selected policy resolves to a different mask for the same field, resolution fails as a +conflict. + +### Future function reference + +A future UDF-backed definition uses a stable function reference rather than inline implementation +source. It replaces `expression`; exactly one of `expression` and `function` can be present. + +```json +{ + "function": { + "reference": "governance.filters.filter_by_region@v3", + "arguments": [ + { "column": "region" }, + { "literal": { "type": "string", "value": "EMEA" } } + ] + } +} +``` + +The shape is modeled after Databricks ABAC's row-filter UDF and argument binding. `reference` +identifies an immutable function revision. Each argument is explicitly a column or a typed literal; +future schemas can add context and tagged-column bindings without changing existing expression +content. + +Before enabling this form, a separate design must define function resolution authority, execution +privileges, determinism, null behavior, and enforcement capabilities. A row-filter function must +return Boolean. A column-mask function must return the exact logical type required for the masked +field. Function arguments and results do not use implicit conversion: every bound argument must +exactly match the declared function signature. A missing, changed, type-mismatched, or unsupported +function fails closed and never falls back to an unrestricted read. + +The first Iceberg REST implementation rejects `function`. Future support may enable it only when +the resolver can compile the function to a closed standard Iceberg restriction or when a separately +specified enforcement path declares native function support. A raw function reference never enters +an Iceberg `read-restrictions` response. + +## Restricted Rego Expressions + +Both built-in policy types use the restricted Rego subset defined below. Its version is part of the +policy content schema rather than a field repeated in every policy. The subset supports only one +complete rule named `filter` or `mask`; it is not an arbitrary Rego module. A row-filter policy +requires `filter`, whose result and conditions must be Boolean. A column-mask policy requires +`mask`, whose result must be an explicit action value and whose condition must be Boolean and +context-only. + +The grammar is: + +```text +program := filterRule | maskRule +filterRule := unconditionalFilter | conditionalFilter +unconditionalFilter := "filter" ":=" expr +conditionalFilter := "filter" ":=" expr "if" expr filterElse* filterFallback +filterElse := "else" ":=" expr "if" expr +filterFallback := "else" ":=" expr +maskRule := unconditionalMask | conditionalMask +unconditionalMask := "mask" ":=" maskAction +conditionalMask := "mask" ":=" maskAction "if" contextExpr maskElse* maskFallback +maskElse := "else" ":=" maskAction "if" contextExpr +maskFallback := "else" ":=" maskAction +maskAction := "action" "(" string ")" +contextExpr := expr +expr := orExpr +orExpr := andExpr ("or" andExpr)* +andExpr := notExpr ("and" notExpr)* +notExpr := "not" notExpr | compareExpr +compareExpr := primary (("==" | "!=" | "<" | "<=" | ">" | ">=" | "in") primary)? +primary := colRef | sessionUser | groupMember | literal | array | "(" expr ")" +colRef := "col" "(" string ")" +sessionUser := "session_user" "(" ")" +groupMember := "is_group_member" "(" string ")" +literal := string | number | boolean | null +array := "[" literal ("," literal)* "]" +boolean := "true" | "false" +null := "null" +number := "-"? ("0" | nonZeroDigit digit*) ("." digit+)? +digit := "0" | nonZeroDigit +nonZeroDigit := "1" | "2" | "3" | "4" | "5" | "6" | "7" | "8" | "9" +``` + +`filter := value-a if condition else := value-b` and +`mask := action("action-a") if condition else := action("action-b")` have the semantic reading “if +condition, then value-a, otherwise value-b.” Conditional branches are evaluated from left to right +and the first true condition selects its value. A conditional rule requires an unconditional final +`else`, so a selected policy never becomes undefined. Unconditional forms omit `if` and `else`. + +Strings use JSON double-quoted syntax. Packages, imports, additional rules, variables, rule bodies +in braces, comments, exponent notation, leading `+`, leading zeroes, trailing decimal points, +chained comparisons, and arbitrary functions are invalid. A row-filter root must be Boolean, and a +column-mask root must be an explicit `action(...)` value. + +### Keywords, identifiers, and escaping + +The restricted syntax reserves the lowercase keywords `filter`, `mask`, `if`, `else`, `and`, `or`, +`not`, `in`, `true`, `false`, and `null`. `:=` is the rule-result assignment operator; `then` is not +a literal token in Rego syntax because the result precedes `if`. The reserved built-in function +identifiers are `col`, `session_user`, and `is_group_member`. They are case-sensitive and are +recognized only as complete tokens. `action` is the reserved mask-action constructor. For example, +`notebook` is not `not` followed by an identifier. Bare identifiers are not part of the restricted +subset, so an unknown word is always invalid rather than an implicit column reference or function +call. + +`filter` and `mask` are Gravitino restricted-syntax keywords, not standard Rego keywords. `filter` +is valid only as the row-filter rule head, and `mask` is valid only as the column-mask rule head. Review Comment: Filter and mask as reserved keywords may easily conflict with the column name, action function, and others. How do we avoid this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
