xhumanoid commented on PR #5923: URL: https://github.com/apache/datafusion-comet/pull/5923#issuecomment-5666815555
@mbutrovich ok, will check thanks for pointing one more place found this bug during investigation of HiveSimpleUDF support will create pr separately, started from Spark 3.5 they have doCodeGen method https://issues.apache.org/jira/browse/SPARK-42051 https://issues.apache.org/jira/browse/SPARK-42052 I used ScalaUDF as a base harness and catch it on some of my tests main different with existing unit test: existing do evaluation of subquery separately and ScalaUDF independent, it's not a part of UDF kernel: ```sql SELECT addOne(x) + (SELECT max(v) FROM t2) AS r FROM t WHERE addOne(x) < (SELECT max(v) FROM t2) * 2 ``` in my case subquery is a part of UDF parameter ```sql SELECT addOne( (SELECT max(v) FROM t2) ) AS r FROM t ``` so during code generation we can't use the value returned from subquery without patch exception from driver ``` Caused by: java.lang.IllegalArgumentException: requirement failed: Subquery subquery#2254, [id=#3678] has not finished at scala.Predef$.require(Predef.scala:337) at org.apache.spark.sql.execution.ScalarSubquery.toLiteral(subquery.scala:108) at org.apache.spark.sql.execution.ScalarSubquery.doGenCode(subquery.scala:104) at org.apache.spark.sql.catalyst.expressions.Expression.$anonfun$genCode$3(Expression.scala:229) at scala.Option.getOrElse(Option.scala:201) at org.apache.spark.sql.catalyst.expressions.Expression.genCode(Expression.scala:224) at org.apache.spark.sql.catalyst.expressions.KnownNotNull.doGenCode(constraintExpressions.scala:48) at org.apache.spark.sql.catalyst.expressions.Expression.$anonfun$genCode$3(Expression.scala:229) at scala.Option.getOrElse(Option.scala:201) at org.apache.spark.sql.catalyst.expressions.Expression.genCode(Expression.scala:224) at org.apache.spark.sql.catalyst.expressions.ScalaUDF.$anonfun$doGenCode$2(ScalaUDF.scala:1118) at scala.collection.immutable.ArraySeq.map(ArraySeq.scala:75) at scala.collection.immutable.ArraySeq.map(ArraySeq.scala:35) at org.apache.spark.sql.catalyst.expressions.ScalaUDF.doGenCode(ScalaUDF.scala:1118) at org.apache.spark.sql.catalyst.expressions.Expression.$anonfun$genCode$3(Expression.scala:229) at scala.Option.getOrElse(Option.scala:201) ``` query itself still reused as expected ``` +- CometProject [r#2255], [if (isnull(Subquery subquery#2254, [id=#3677])) null else addOne(knownnotnull(ReusedSubquery Subquery subquery#2254, [id=#3677])) AS r#2255] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
