Huaxin Gao created SPARK-59726:
----------------------------------
Summary: SPIP: Pluggable Row-Filter and Column-Mask Enforcement in
Catalyst
Key: SPARK-59726
URL: https://issues.apache.org/jira/browse/SPARK-59726
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 5.0.0
Reporter: Huaxin Gao
External catalogs and governance systems need to enforce fine-grained access
control, row-level filtering and column masking, on data read through Spark.
Today Spark has no supported API for a data source or catalog to attach such
policies to a scan and have them enforced correctly in the query plan.
Integrators must inject Catalyst rules through SparkSessionExtensions and use
package-private Catalyst internals, which is brittle across versions and
unsafe: a masking projection added naively can be removed or collapsed by the
optimizer, silently returning unmasked data.
This SPIP proposes a stable, public, optimizer-safe DataSource V2 API for a
table to declare a read access policy (readable columns, a row filter, and a
set of column masks) that Spark enforces with fail-closed semantics: the row
filter evaluates on original values before masking, the masks cannot be
optimized away, and anything Spark cannot resolve fails the read rather than
returning unprotected data.
SPIP document:
https://docs.google.com/document/d/1hYEHORjHUnFzSoBY2DeOPILVJyu6iWXBusYwpF9FQJM/edit?tab=t.0
Discussion thread: <to be added >
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
