sunchao commented on code in PR #6538:
URL: https://github.com/apache/datafusion-comet/pull/6538#discussion_r4165179674
##########
spark/src/main/scala/org/apache/comet/rules/CometExecRule.scala:
##########
@@ -756,6 +762,25 @@ case class CometExecRule(session: SparkSession)
}
}
+ /**
+ * A relation keeps the cache format it was stored in, since
`spark.sql.cache.serializer` is
+ * static, so a plan that runs without Comet's native execution still reads
relations cached in
+ * Comet's format. Spark's `InMemoryTableScanExec` reads that format more
slowly than Spark's
+ * own (https://github.com/apache/datafusion-comet/issues/5485), and nothing
else records a
+ * fallback reason in such a plan, so record one on each scan that does.
+ */
+ private def explainSparkReadsOfCometCache(plan: SparkPlan, cause: String):
Unit =
+ plan.foreach {
+ case scan: InMemoryTableScanExec if
CometExecRule.readsCometCacheFormat(scan) =>
+ withFallbackReason(
Review Comment:
[P2] Explicitly discard the returned scan here, for example with `val _ =
withFallbackReason(...)`. The `foreach` callback expects `Unit`, but
`withFallbackReason` returns its node. Consequently, `./mvnw -B test-compile
-Pspark-3.5 -Pstrict-warnings -DskipTests` fails with `discarded non-Unit
value` instead of compiling. This introduces a build failure in the required
strict Scala CI job.
Evidence: Exact-head CI job
https://github.com/apache/datafusion-comet/actions/runs/36998920133/job/110812901711
reports `CometExecRule.scala:775: discarded non-Unit value`. A local Scala
2.12.18 probe using the extracted helper and the repository’s strict compiler
flags exited 1 with the same diagnostic. Changing only the call to `val _ =
withFallbackReason(...)` made that probe exit 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
