Hi Spark devs,

I've recently opened SPARK-62006
<https://issues.apache.org/jira/browse/SPARK-62006>, which preserves output
ordering for full outer joins using sort-merge join.

The motivation is to retain useful ordering guarantees from the sort-merge
join, allowing unnecessary downstream sorting to be avoided.

I'd appreciate feedback on the approach, particularly from anyone familiar
with Spark's sort-merge join implementation and physical operator ordering
guarantees.

I also have another smaller, unrelated open PR which improves allocs on
writing to Parquet:

- PR #59566 <https://github.com/apache/spark/pull/59566>:
[SPARK-60321][SQL] Avoid intermediate copies of byte-array-backed strings
in Parquet writes.

Thanks a lot!

Best,
Saath

Reply via email to