andygrove commented on code in PR #5923:
URL: https://github.com/apache/datafusion-comet/pull/5923#discussion_r4155998248
##########
spark/src/test/scala/org/apache/comet/CometCodegenSuite.scala:
##########
@@ -1143,6 +1143,24 @@ class CometCodegenSuite
checkSparkAnswerAndOperator(df)
}
+ test("a scalar subquery inside the dispatched subtree falls back to Spark") {
+ // The dispatcher serializes the tree at plan time, but a subquery's
`result` is populated by
+ // the surrounding operator's `waitForSubqueries` at execution time. The
deserialized copy on
+ // the executor therefore has no result, and `ScalarSubquery.doGenCode`
used to fail the kernel
+ // compile with "Subquery ... has not finished" -- a runtime failure,
after the plan had already
+ // committed to the kernel. `canHandle` now refuses it, so the operator
falls back cleanly.
+ spark.udf.register("addOne", (i: Int) => i + 1)
+ withTable("t", "t2") {
+ sql("CREATE TABLE t (x INT) USING parquet")
+ sql("INSERT INTO t VALUES (1), (2), (3), (4), (5)")
+ sql("CREATE TABLE t2 (v INT) USING parquet")
+ sql("INSERT INTO t2 VALUES (2), (4)")
+ assertNoCodegen {
+ checkSparkAnswer(sql("SELECT addOne((SELECT max(v) FROM t2)) AS r FROM
t"))
+ }
+ }
+ }
+
test("ScalaUDF composed with reused scalar subquery across projection and
filter") {
Review Comment:
The comment on this existing test now contradicts the fix. It says
`waitForSubqueries` populates each subquery's `result` before the bridge
serializes the expression, that the value travels into the cache key bytes so
different subquery values compile distinct kernels, and that the test exercises
the "subquery-correctness invariant documented on `canHandle`". The new test
and the updated `canHandle` comment show that ordering never held, and that
invariant no longer exists. Could we update this comment to say what the test
covers now? As I read it, that is a subquery under a native parent that is
evaluated natively and never reaches a kernel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
