Yes, DS v2 push down only supports INNER/LEFT/RIGHT joins now. Personally, I'm 
okay to support other join types. I'm a bit afraid, SEMI / ANTI is rare used 
and FULL OUTER may is not worth to do if no performance gain.
I think alias tolerance in join push-down is a good idea.







At 2026-11-04 07:54:19, "Holden Karau" <[email protected]> wrote:

I like this proposal, I think we're at a good point to modernize some of the 
DSv2 pushdowns.


On Fri, Oct 30, 2026 at 3:03 PM Jing Zhan <[email protected]> wrote:

Hi all,

I'd like to start a discussion on a SPIP to extend DSv2 operator push-down.

JIRA: https://issues.apache.org/jira/browse/SPARK-60765
Design: 
https://docs.google.com/document/d/1ah8u-1vxPvgqpjw0qzRNXz0Oo-X8tEzlK9ngmg7EPXU/edit
Builds on SPARK-52187 (DSv2 join push-down).
Shepherd: Holden Karau

Today DSv2 can push filters, column pruning, aggregates, limit/TopN, sample, 
and INNER/LEFT/RIGHT joins. Those joins are off unless
spark.sql.optimizer.datasourceV2JoinPushdown is set. A query that uses other 
join types, a set operation, or a window function over the same database still 
pulls the raw rows into Spark.

The proposal adds opt-in interfaces, each of which a source can decline:
1. FULL OUTER / SEMI / ANTI join types
2. Alias tolerance in join push-down
3. Make the join push-down config public, and evaluate enabling it by default
4. Join + aggregate composition
5. Set operations (UNION / INTERSECT / EXCEPT)
6. Window functions

JDBC/H2 is the reference implementation. This does not cover cross-source 
push-down or file-based sources.

Please reply to this thread. A [VOTE] will follow once the discussion settles.

Thanks,
Jing Zhan




--

Twitter: https://twitter.com/holdenkarau

Fight Health Insurance: https://www.fighthealthinsurance.com
Books (Learning Spark, High Performance Spark, etc.): https://amzn.to/2MaRAG9 
YouTube Live Streams: https://www.youtube.com/user/holdenkarau
Pronouns: she/her

Reply via email to