sunchao commented on code in PR #6949: URL: https://github.com/apache/datafusion-comet/pull/6949#discussion_r4266146462
########## docs/source/user-guide/latest/kubernetes.md: ########## @@ -61,21 +61,24 @@ REVISION: 1 TEST SUITE: None ``` -Create example Spark application file `spark-pi.yaml` +Create an example Spark SQL application file named `comet-dataframe-pi.yaml`. This application uses +Spark's DataFrame API, so it exercises Comet without requiring an external data source. ```bash apiVersion: sparkoperator.k8s.io/v1beta2 kind: SparkApplication metadata: - name: spark-pi + name: comet-dataframe-pi namespace: default spec: type: Scala mode: cluster image: apache/datafusion-comet imagePullPolicy: IfNotPresent - mainClass: org.apache.spark.examples.SparkPi + mainClass: org.apache.spark.examples.sql.SparkDataFramePi Review Comment: [P2] Enable native range execution for the replacement application. `SparkDataFramePi` starts with `spark.range`, but both `spark.comet.exec.range.enabled` and `spark.comet.convert.range.enabled` default to `false` and the manifest enables neither. Its projection, filter, and partial count therefore stay on Spark. Comet also declines the final count over Spark’s partial aggregate, leaving the whole query on Spark. Users get a Pi result but no native plan, contrary to the new verification instructions. Add `"spark.comet.exec.range.enabled": "true"` and validate native execution. Evidence: A local Spark 4.1.3 probe of the example’s query, using the manifest settings and recompiled exact-head planner/config/operator sources over cached dependencies, printed `RANGE_DEFAULTS=false,false`. Its completed plan contained only Spark `HashAggregate`, `Exchange`, `Project`, `Filter`, and `Range`, with zero `Comet native query plan:` messages. Changing only `spark.comet.exec.range.enabled` to `true` produced `CometRange`, `CometProject`, `CometFilter`, `CometHashAggregate`, native shuffle, and 11 native-plan messages. The defaults are in `CometConf.scala:221-228` and `355-363`. The exclusion of Spark-partial/native-final COUNT is documented in `CometAggregateExpressionSerde.scala:85-91`. ########## docs/source/user-guide/latest/kubernetes.md: ########## @@ -110,41 +114,45 @@ Refer to [Comet builds](#comet-docker-images) Run Apache Spark application with Comet enabled ```bash -kubectl apply -f spark-pi.yaml -sparkapplication.sparkoperator.k8s.io/spark-pi created +kubectl apply -f comet-dataframe-pi.yaml +sparkapplication.sparkoperator.k8s.io/comet-dataframe-pi created ``` Check application status ```bash -kubectl get sparkapp spark-pi +kubectl get sparkapp comet-dataframe-pi -NAME STATUS ATTEMPTS START FINISH AGE -spark-pi RUNNING 1 2025-03-18T21:19:48Z <no value> 65s +NAME STATUS ATTEMPTS START FINISH AGE +comet-dataframe-pi COMPLETED 1 ... ... ... ``` To check more runtime details ```bash -kubectl describe sparkapplication spark-pi +kubectl describe sparkapplication comet-dataframe-pi .... Events: Type Reason Age From Message ---- ------ ---- ---- ------- - Normal SparkApplicationSubmitted 8m15s spark-application-controller SparkApplication spark-pi was submitted successfully - Normal SparkDriverRunning 7m18s spark-application-controller Driver spark-pi-driver is running - Normal SparkExecutorPending 7m11s spark-application-controller Executor [spark-pi-68732195ab217303-exec-1] is pending - Normal SparkExecutorRunning 7m10s spark-application-controller Executor [spark-pi-68732195ab217303-exec-1] is running - Normal SparkExecutorCompleted 7m5s spark-application-controller Executor [spark-pi-68732195ab217303-exec-1] completed - Normal SparkDriverCompleted 7m4s spark-application-controller Driver spark-pi-driver completed + Normal SparkApplicationSubmitted 8m15s spark-application-controller SparkApplication comet-dataframe-pi was submitted successfully + Normal SparkDriverRunning 7m18s spark-application-controller Driver comet-dataframe-pi-driver is running + Normal SparkExecutorPending 7m11s spark-application-controller Executor [comet-dataframe-pi-68732195ab217303-exec-1] is pending + Normal SparkExecutorRunning 7m10s spark-application-controller Executor [comet-dataframe-pi-68732195ab217303-exec-1] is running + Normal SparkExecutorCompleted 7m5s spark-application-controller Executor [comet-dataframe-pi-68732195ab217303-exec-1] completed + Normal SparkDriverCompleted 7m4s spark-application-controller Driver comet-dataframe-pi-driver completed ``` Get Driver Logs ```bash -kubectl logs spark-pi-driver +kubectl logs comet-dataframe-pi-driver ``` +The driver log contains the computed approximation (`Pi is roughly ...`). To confirm that Comet +executed the DataFrame query, inspect the executor log for the native plan emitted by Review Comment: [P2] Keep the executor logs available for the documented verification step. Following the guide in order reaches `COMPLETED` before inspecting logs, but Spark defaults `spark.kubernetes.executor.deleteOnTermination` to `true`. On a cluster without centralized log collection, the executor pods and their retrievable logs are therefore gone by this step. Even after enabling native range execution, users cannot obtain the promised execution evidence. Set this option to `false` for the example or document capturing executor logs while the application runs. Evidence: Spark v4.1.3 `SparkDataFramePi.scala:40` calls `spark.stop()` after printing Pi. Kubernetes `Config.scala:695-701` defaults executor deletion to `true`. `KubernetesClusterSchedulerBackend.scala:159-161` consequently invokes `podAllocator.stop(applicationId())`, and `ExecutorPodsAllocator.scala:535-542` deletes the application's executor pods. The manifest supplies no override or log-retention mechanism. This establishes the default cleanup path before post-completion inspection. No live Kubernetes reproduction was performed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
