sunchao commented on code in PR #5533:
URL: https://github.com/apache/datafusion-comet/pull/5533#discussion_r4179061096
##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -1046,6 +1054,159 @@ case class CometExecRule(session: SparkSession)
}
}
+ /** Keep opted-in expressions in Spark's row pipeline where an operator can
skip inputs. */
+ private def preserveEvaluationMasks(plan: SparkPlan): SparkPlan = {
+ if (!CometConf.COMET_EXEC_PRESERVE_EVALUATION_MASKS_ENABLED.get()) return
plan
+
+ def findEvaluationMaskName(expr: Expression): Option[String] = {
+ var name: Option[String] = None
+ expr.exists { child =>
+ name = QueryPlanSerde.evaluationMaskName(child)
+ name.isDefined
+ }
+ name
+ }
+
+ def originalPlan(node: SparkPlan): SparkPlan = node match {
+ case scan: CometScanExec =>
+ scan.wrapped
+ .copy(partitionFilters = scan.partitionFilters, dataFilters =
scan.dataFilters)
+ case comet: CometExec => comet.originalPlan
Review Comment:
Fixed in c6748e259286502da38c8a65ceddc1f25426d742. The local selector now
exposes only its sort expressions and its input-materialization boundary. If
restoration changes its child, it is removed so the enclosing top-K owns
projection and offset exactly once.
The new regression reproduces the reported AQE broadcast-stage failure on
the previous implementation and passes with the fix. It covers AQE and fusion
on/off, repeated rule application, nonzero offset and actual outer-limit
fallback results. All 10 evaluation-mask tests pass on Spark 4.1.3; strict
Spark 3.5 compilation and the focused new runtime regression also pass. Both
runs use the verified native artifact whose native inputs match this branch.
Broader requested CI is still pending.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
