Yes, DS v2 push down only supports INNER/LEFT/RIGHT joins now. Personally, I'm okay to support other join types. I'm a bit afraid, SEMI / ANTI is rare used and FULL OUTER may is not worth to do if no performance gain. I think alias tolerance in join push-down is a good idea.
At 2026-11-04 07:54:19, "Holden Karau" <[email protected]> wrote: I like this proposal, I think we're at a good point to modernize some of the DSv2 pushdowns. On Fri, Oct 30, 2026 at 3:03 PM Jing Zhan <[email protected]> wrote: Hi all, I'd like to start a discussion on a SPIP to extend DSv2 operator push-down. JIRA: https://issues.apache.org/jira/browse/SPARK-60765 Design: https://docs.google.com/document/d/1ah8u-1vxPvgqpjw0qzRNXz0Oo-X8tEzlK9ngmg7EPXU/edit Builds on SPARK-52187 (DSv2 join push-down). Shepherd: Holden Karau Today DSv2 can push filters, column pruning, aggregates, limit/TopN, sample, and INNER/LEFT/RIGHT joins. Those joins are off unless spark.sql.optimizer.datasourceV2JoinPushdown is set. A query that uses other join types, a set operation, or a window function over the same database still pulls the raw rows into Spark. The proposal adds opt-in interfaces, each of which a source can decline: 1. FULL OUTER / SEMI / ANTI join types 2. Alias tolerance in join push-down 3. Make the join push-down config public, and evaluate enabling it by default 4. Join + aggregate composition 5. Set operations (UNION / INTERSECT / EXCEPT) 6. Window functions JDBC/H2 is the reference implementation. This does not cover cross-source push-down or file-based sources. Please reply to this thread. A [VOTE] will follow once the discussion settles. Thanks, Jing Zhan -- Twitter: https://twitter.com/holdenkarau Fight Health Insurance: https://www.fighthealthinsurance.com Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 YouTube Live Streams: https://www.youtube.com/user/holdenkarau Pronouns: she/her
