andygrove opened a new issue, #6904: URL: https://github.com/apache/datafusion-comet/issues/6904
### What is the problem the feature request solves? Platforms that run many Spark jobs, such as job schedulers, notebook services and data catalogs, want to show their users whether a job runs on Comet, which parts of it fall back to Spark and why, and to rank the most common gaps across all their jobs. Comet has this information, but it only produces it as text for people to read: - The fallback log (`spark.comet.explain.fallback.enabled`): WARN lines in the driver log. - The `ExtendedExplainInfo` explain provider: in `EXPLAIN` output, and in the Spark UI from Spark 4.0 (SPARK-47289). - Plan-only mode (`spark.comet.explain.planOnly.enabled`): logs the plan Comet would run, with a coverage summary, under a `[Comet plan-only]` prefix, and then runs the query in Spark. This is the closest thing Comet has to a compatibility check, since turning it on doesn't change how a job runs. The coverage summary is one sentence, from `CometCoverageStats.toString`: ``` Comet accelerated 2 out of 2 eligible operators (100%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 0 native, 2 codegen dispatch. ``` A tool that wants any of this has to collect driver logs or explain output and parse English whose wording changes between releases (#5346 lists five spellings of one fallback reason). Driver logs often aren't kept, and the text has no stable structure for operators, so a tool can't map the results back to the plan or aggregate them reliably. `CometCoverageStats.forPlan` and `ExtendedExplainInfo.getFallbackReasons` can be called today, but they're internal API and return the same strings. ### Describe the potential solution A versioned, documented JSON report for each query, containing: - The report schema version, the Comet and Spark versions, and the SQL execution ID. - The coverage numbers from `CometCoverageStats`: eligible operators, Comet operators, transitions, and the native and codegen-dispatch expressions. - One entry per operator in the final plan: the Spark operator, whether Comet converted it and to what, and its fallback reasons. Each reason has a category, a message and, where one applies, the config that would enable it. The structured reasons proposed in #5346 would be the source. Two ways to get it, both off by default: 1. An API that returns the report for a `DataFrame` or a `SparkPlan`. With plan-only mode, a team could check a workload without running it on Comet. 2. A `SparkListenerEvent` that carries the report, posted once per SQL execution so that it lands in the event log. Tools that read event logs through the History Server could use it after the job has finished, without the driver log. `CometExecutorMemoryUsage` already writes Comet data to the event log this way (#6401). Questions to settle: - Which plan to report under AQE. The final plan at the end of the execution seems the most useful; #6442 covers fallback reasons that are lost from it. - A size limit for very large plans. - What compatibility promise the schema gets under the versioning policy. ### Additional context Related: #144 (show fallback reasons in the Spark UI) and #5346 (structured fallback reasons). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
