comphead opened a new pull request, #7681:
URL: https://github.com/apache/datafusion-comet/pull/7681

   ## Which issue does this PR close?
   
   No issue was filed for this small addition.
   
   ## Rationale for this change
   
   Checking whether Comet ran a query natively currently takes
   `new 
ExtendedExplainInfo().getFallbackReasons(df.queryExecution.executedPlan)` plus 
a separate check
   that Comet is enabled at all. About 20 suites repeat the first part, and 
applications that embed
   Comet need the same check. A plan-only check is also misleading on its own: 
when Comet disables
   itself (for example without off-heap memory or without the Comet shuffle 
manager), the plan has no
   fallback reasons because Comet never tagged it, so "no fallback reasons" 
does not mean "accelerated".
   
   ## What changes are included in this PR?
   
   Two methods on `ExtendedExplainInfo`, the class users already reach through
   `spark.sql.extendedExplainProviders`:
   
   - `getFallbackReasons(df: DataFrame): Seq[String]` returns the fallback 
reasons of the DataFrame's
     executed plan. It delegates to the existing `getFallbackReasons(plan: 
SparkPlan)`.
   - `planAccelerated(df: DataFrame): Boolean` is true when Comet is enabled 
for the DataFrame's session
     and the plan has no fallback reasons. "Enabled" uses 
`CometSparkSessionExtensions.isCometLoaded`,
     the same check the Comet planner rules make, rather than 
`spark.comet.enabled` alone.
   
   Neither method runs the query. For an adaptive query that has not run yet, 
the executed plan is the
   initial plan. The scaladoc and the user guide snippet in 
`understanding-comet-plans.md` say to run
   the query first to get the final plan.
   
   The session conf is read through `df.queryExecution.sparkSession`, because 
on Spark 4.x `DataFrame`
   is the API `Dataset`, which has `queryExecution` but not `sessionState`.
   
   ## How are these changes tested?
   
   A new test in `CometExecSuite`, "getFallbackReasons and planAccelerated for 
a DataFrame", covers:
   
   - A query with a known fallback (sampling with replacement): 
`getFallbackReasons(df)` contains the
     reason, matches the plan overload, and `planAccelerated` is false.
   - A fully native filter and project: no reasons, and `planAccelerated` is 
true.
   - The same query with `spark.comet.enabled=false`: `planAccelerated` is 
false.
   
   `spotless:check` passes locally. The suite has not been run locally yet.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to