kazuyukitanimura commented on PR #7336:
URL:
https://github.com/apache/datafusion-comet/pull/7336#issuecomment-6784233008
Thanks for turning these around so quickly. I noticed nothing has run on
bfc14c767 yet. CI (fork) says no run was found on your fork for that commit,
and Required Checks here passes without a build. The commit changes
isCometLoaded, which every Comet session goes through, so could you start the
fork run? I'll add run-all-spark-profiles too, so 3.4,
3.5, 4.0 and 4.2 get covered along with your fork's 4.1 run.
Following up on the cluster run. @comphead and I both read the executor side
the same way you did, but none of us has run it. Every test runs in local mode,
where executors share the driver's SparkEnv, so nothing in CI covers this. If
the reasoning were wrong, every Comet shuffle would fail on the executors.
Could you do one run on a standalone or Kubernetes cluster with Spark 4.0, the
plugin, and no spark.shuffle.manager before this merges? sc.parallelize(1 to
100, 10).map(_ =>
org.apache.spark.SparkEnv.get.shuffleManager.getClass.getName).distinct.collect
plus a GROUP BY would be enough.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
