Ok this vote fails, I appreciate the clear asks and I'll circle back with two updated docs next week. For the protocol for parallel reads my thought is as close to ADBC parallel reads is the way to go, I'll clarify that a bit too.
Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com/ <https://www.fighthealthinsurance.com/?q=hk_email> Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> YouTube Live Streams: https://www.youtube.com/user/holdenkarau Pronouns: she/her On Fri, Nov 20, 2026 at 2:23 PM Dongjoon Hyun via dev <[email protected]> wrote: > -1 (binding) for now. > > Thank you for driving this, Holden. I support the goal. An OSS way to keep > user code away from raw data is the missing half of SPARK-59726, and Spark > Connect is a natural place for it. My -1 has two reasons. > > First, the SPIP proposes a security boundary without defining it yet. > SPARK-59726 v2 leaves isolating user code out of scope and names Spark > Connect as the way to do it (G4, "Limit"), so this SPIP is where that > boundary has to be written down. > > Second, this vote combines two separate proposals: a Spark Connect data > source with a trusted server mode for FGAC, and parallel reads from > executors, a Spark Connect protocol change that "can benefit Spark Connect > users in general". I'd like to vote on them separately. > > I'll change my vote once group 1 is in the doc and parallel reads either > move to their own SPIP or come with their protocol (groups 2 and 3). > > ### 1. Trust boundary (should be addressed before a vote) > > - **What a client may send.** Today a Spark Connect client is not "just a > normal user" (comment [e]). It can upload JARs and class files > (`AddArtifacts`), send Scala UDFs as Java-serialized closures and Python > UDFs/UDTFs/data sources, run shell commands with `SELECT TRANSFORM(...) > USING`, call static Java methods such as `System.getenv` with `reflect` > (`spark.sql.reflect.allowList` is empty by default), read files by path > (`spark.read.load(path)`, ``SELECT * FROM parquet.`path` ``, `CREATE > TEMPORARY VIEW ... USING`), and set `spark.sql.catalog.<name>` at runtime > to open another catalog over the same storage. Each of these runs with the > trusted cluster's credentials. Please list the allowed RPCs, plan nodes, > functions and configs (default-deny), checked on the analyzed plan rather > than on proto messages, since `SQL`, `expression_string` and > `unresolved_function` can resolve to anything. `PipelinesHandler` (SQL > command allowlist) and `MLHandler` (class allowlist) are precedents. > - **What the server sends back.** After each query, Spark Connect returns > the SQL metrics of every plan node (`MetricGenerator`), including a scan's > row count before the policy filter. `AnalyzePlan` returns `explain`, > `tree_string` and `input_files`, which is the same data as `_file`. Please > make sure SPARK-59726 G3 covers these responses, not only the UI and the > event log. > - **Dependency on SPARK-59726.** Pushed-down operations run in the trusted > optimizer next to the policy, so the boundary holds only if G1 (leakproof > barrier) is enforced there. Please state the dependency and define "safe > pushdown" as G1's leakproof. "The majority of the operations supported in > DSv2 are already safe" does not hold under ANSI mode: division, casts and > overflow can fail on hidden rows, and the optimizer moves and combines > filters freely (we fixed seven such cases this week alone, e.g. > SPARK-62091). > - **Project policy.** Once Spark Connect is a security boundary, an > allowlist bypass is a vulnerability for security@, and every new > function, plan node, RPC and config needs a classification. Please say this > explicitly. > - **Tests.** Please add a negative test for each path above to Q8. > > ### 2. Two proposals in one vote > > - **Different dependencies and stakeholders.** The data source and the > trusted mode build on SPARK-59726, while parallel reads build on the ADBC > SPIP (SPARK-60255), which is still under discussion. Parallel reads also > change what every Spark Connect client (Python, Scala, Go, Swift, Rust) and > the Spark Connect Gateway see. With one vote, a voter cannot support one > part and object to the other. > - **The scope of parallel reads is unclear.** The title and the vote email > include them, but Q7 and Q8 have no milestone for them, Q6 says external > storage may replace them, and the Future section lists their use by regular > clients as future work. The DISCUSS thread on dev@ was titled "Spark > Connect Data Source for Spark FGAC", and its emails did not mention them. > - **Suggestion.** Keep this SPIP to the data source and the trusted mode, > moving data through the trusted driver or external storage first, and > propose parallel reads as a separate SPIP with Appendix A (protocol and > compatibility). > > ### 3. Parallel reads > > - **Protocol.** "ADBC as a basis" leaves the wire protocol open. Spark > Connect is not Flight SQL, and Spark has no Arrow Flight dependency today. > Please specify it, for example a result mode that returns endpoints and > tickets, which a client opts into through `RequestOption` like > `ResultChunkingOptions`, so existing Python/Scala/Go/Swift/Rust clients are > unaffected. > - **Serving from executors.** Task retries, speculation and stage retries > re-read a partition, so results must stay until the reading job ends, and > dynamic allocation must not remove an executor that holds unread results. > Please define executor loss, cancellation and cleanup. > - **Security and networking.** Each trusted executor gets a new port that > the untrusted cluster can reach. It needs TLS and short-lived tickets > scoped to one operation, and it bypasses the Spark Connect Gateway. On > Kubernetes, executor pods have no Service, and exposing them must not > expose the driver RPC and block manager ports (`spark.authenticate` plus a > network policy). > - **The non-parallel path** collects every partition on the trusted driver > and is limited by `spark.driver.maxResultSize` (1g by default). Please > state this limit. > > ### 4. Design and alternatives > > - **TLS and identity.** The Spark Connect server cannot serve TLS itself; > it listens in plaintext. The Python and Scala clients always use TLS when > they send a token to a host other than `localhost`, so "SSL + shared token" > needs a TLS-terminating proxy in front of the trusted server. In addition, > `spark.connect.authenticate.token` is an internal config, and > `UserContext.user_id` is declared by the client. With one shared token per > cluster, each principal needs its own trusted cluster. The Spark Connect > Gateway already verifies JWT/OIDC and replaces `user_id` with the verified > identity, so it may be the better base for identity. > - **Version skew.** Without capability discovery, a client cannot know > which pushdowns an older trusted server accepts, and a rejection fails the > query. Please define the fallback. > - **DSv2 pushdown.** Join pushdown exists since 4.1.0 (SPARK-52187), and > SPARK-60765, which you shepherd, adds set operations and window functions. > Please name it in the doc. > - **JDBC.** The Thrift server is no longer needed: 4.1.0 ships a Spark > Connect JDBC driver (`jdbc:sc://`, SPARK-53914). I tried the JDBC source > and `JDBCTableCatalog` against a 4.3.0 Connect server, and both fail at > `SparkConnectConnection.prepareStatement`, which is not implemented. With > that and a dialect (4.3.0 can map `jdbc:sc:` to the built-in `databricks` > dialect), the non-parallel path may need no new protocol. Worth comparing > in Alternatives. > > Thanks, > Dongjoon. > > > > On 2026/11/20 18:19:21 huaxin gao wrote: > > +1 > > > > On Fri, Nov 20, 2026 at 9:50 AM Holden Karau <[email protected]> > wrote: > > > > > Hi Spark Community, > > > > > > I'm starting a vote on updating Spark to support reading from Spark > > > Connect, including parallel Spark Connect Reads. The original > motivation is > > > to support fine grained access controls, but the parallel Spark Connect > > > reads can benefit Spark Connect users in general. > > > > > > SPIP doc: > > > > https://docs.google.com/document/d/1c-Gw6HQn2akq7V2LWrvY8JYTo13CuPKHOa8gUT4QR7Y/edit?usp=sharing > > > > > > Only discussion happened in the doc (but the thread > > > https://lists.apache.org/thread.html/2ym2k633sd7kr1zsx86q9ccg6qn9h039 > is > > > at for anyone who wants to verify) > > > > > > The vote will be open for at least 72 hours, but I plan to keep it > open a > > > little longer, at least Wednesday @ 5pm pacific to give people time > given > > > potential US holiday travel. > > > > > > Vote passes with 3 +1s from PMC and no -1s from PMC ( > > > https://spark.apache.org/improvement-proposals.html) > > > > > > Cheers, > > > > > > Holden :) > > > -- > > > Twitter: https://twitter.com/holdenkarau > > > Fight Health Insurance: https://www.fighthealthinsurance.com/ > > > <https://www.fighthealthinsurance.com/?q=hk_email> > > > Books (Learning Spark, High Performance Spark, etc.): > > > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > > > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > > > Pronouns: she/her > > > > > > > --------------------------------------------------------------------- > To unsubscribe e-mail: [email protected] > >
