andygrove opened a new issue, #5335: URL: https://github.com/apache/datafusion-comet/issues/5335
## What is the problem the feature request solves? Users evaluating Comet have no low-risk way to find out how much of their workload Comet would actually accelerate. Enabling Comet changes execution, so any assessment carries correctness and performance risk and usually needs a separate, dedicated run. ## Describe the potential solution Add a "plan only" / dry-run mode (e.g. `spark.comet.planOnly.enabled`) where Comet runs its full planning pass — operator/expression conversion and fallback analysis — logs the resulting Comet plan, and then executes the original Spark plan instead. The mechanism can reuse what `spark.comet.exec.transitionRevert.enabled` already does in `RevertNativeForTransitionHeavyStages`: after planning, replace Comet operators with their Spark equivalents. Here it would be applied unconditionally to the whole plan rather than per stage above a transition threshold. This composes with existing reporting (extended explain fallback/coverage info and `spark.comet.metrics.enabled`), so a user could run their production workload unchanged and collect an estimate of Comet coverage. Points to discuss: - Where to surface the Comet plan: driver log, extended explain info, and/or metrics. - Whether to also serialize/create the native plan, which catches native-side planning failures and gives a more accurate estimate at some planning cost. - Behavior under AQE, where planning happens per stage. - Whether this belongs as its own config or as a mode of the existing explain/telemetry features. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
