comphead commented on code in PR #7256:
URL: https://github.com/apache/datafusion-comet/pull/7256#discussion_r4459239734
##########
spark/src/main/scala/org/apache/spark/sql/comet/CometWindowExec.scala:
##########
@@ -75,16 +75,58 @@ object CometWindowExec extends
CometOperatorSerde[WindowExec] {
}
}
+ // Spark plans no sort below the window when its input already reports the
order the window
+ // needs. That order is relied on, by declaring it natively or by handing
the rows to Spark's
+ // window, only when mayReportFalseOrder trusts it: Spark's window would
group the rows of a
+ // false order wrongly, while the native one rejects an input whose order
it cannot see.
+ val requiredOrdering = op.requiredChildOrdering.head
+ val trustsInputOrder =
+ SortOrder.orderingSatisfies(op.child.outputOrdering, requiredOrdering) &&
+ !CometHashJoinExec.mayReportFalseInputOrder(op.child)
+
+ // The native window finds partitions and peers by comparing keys byte by
byte, so it would
+ // tell apart collated strings that Spark treats as equal, as
CometWindowGroupLimitExec notes.
+ val collated = (op.partitionSpec ++ op.orderSpec.map(_.child))
+ .filter(key => hasNonDefaultStringCollation(key.dataType))
+ if (collated.nonEmpty && trustsInputOrder) {
Review Comment:
Thanks, agreed. 3ff09398ec restores the order instead of relying on a native
rejection. When `mayReportFalseOrder` doubts the order a window's child
reports, `CometExecRule` sorts the input of a window that stays in Spark.
Spark's window, and its collation, then see the rows in order.
With that in place, a collated key falls back again whatever the input. A
window with `ORDER BY` but no `PARTITION BY` over a doubted order falls back
too, since the native window checks no order there. A partitioned window over a
doubted order stays native. It either rejects the input, or runs correctly when
DataFusion sees the order.
`a window that stays in Spark over a nested loop join sorts the join's rows
again` runs your shape: `dense_rank() OVER (ORDER BY s)` over the nested loop
join on a one-partition cache. It also runs the partitioned collated case from
your previous comment and the same unpartitioned shape with an integer order
key. Each compares the rows with Spark, and checks that the native nested loop
join is in the plan and that every Spark window reads from a sort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
