1cbyc commented on code in PR #6949: URL: https://github.com/apache/datafusion-comet/pull/6949#discussion_r4266414398
########## docs/source/user-guide/latest/kubernetes.md: ########## @@ -61,21 +61,24 @@ REVISION: 1 TEST SUITE: None ``` -Create example Spark application file `spark-pi.yaml` +Create an example Spark SQL application file named `comet-dataframe-pi.yaml`. This application uses +Spark's DataFrame API, so it exercises Comet without requiring an external data source. ```bash apiVersion: sparkoperator.k8s.io/v1beta2 kind: SparkApplication metadata: - name: spark-pi + name: comet-dataframe-pi namespace: default spec: type: Scala mode: cluster image: apache/datafusion-comet imagePullPolicy: IfNotPresent - mainClass: org.apache.spark.examples.SparkPi + mainClass: org.apache.spark.examples.sql.SparkDataFramePi Review Comment: Enabled `spark.comet.exec.range.enabled` in 4a1bb1c so the bundled DataFrame workload enters the native range, projection, filter, and aggregation path. ########## docs/source/user-guide/latest/kubernetes.md: ########## @@ -110,41 +114,45 @@ Refer to [Comet builds](#comet-docker-images) Run Apache Spark application with Comet enabled ```bash -kubectl apply -f spark-pi.yaml -sparkapplication.sparkoperator.k8s.io/spark-pi created +kubectl apply -f comet-dataframe-pi.yaml +sparkapplication.sparkoperator.k8s.io/comet-dataframe-pi created ``` Check application status ```bash -kubectl get sparkapp spark-pi +kubectl get sparkapp comet-dataframe-pi -NAME STATUS ATTEMPTS START FINISH AGE -spark-pi RUNNING 1 2025-03-18T21:19:48Z <no value> 65s +NAME STATUS ATTEMPTS START FINISH AGE +comet-dataframe-pi COMPLETED 1 ... ... ... ``` To check more runtime details ```bash -kubectl describe sparkapplication spark-pi +kubectl describe sparkapplication comet-dataframe-pi .... Events: Type Reason Age From Message ---- ------ ---- ---- ------- - Normal SparkApplicationSubmitted 8m15s spark-application-controller SparkApplication spark-pi was submitted successfully - Normal SparkDriverRunning 7m18s spark-application-controller Driver spark-pi-driver is running - Normal SparkExecutorPending 7m11s spark-application-controller Executor [spark-pi-68732195ab217303-exec-1] is pending - Normal SparkExecutorRunning 7m10s spark-application-controller Executor [spark-pi-68732195ab217303-exec-1] is running - Normal SparkExecutorCompleted 7m5s spark-application-controller Executor [spark-pi-68732195ab217303-exec-1] completed - Normal SparkDriverCompleted 7m4s spark-application-controller Driver spark-pi-driver completed + Normal SparkApplicationSubmitted 8m15s spark-application-controller SparkApplication comet-dataframe-pi was submitted successfully + Normal SparkDriverRunning 7m18s spark-application-controller Driver comet-dataframe-pi-driver is running + Normal SparkExecutorPending 7m11s spark-application-controller Executor [comet-dataframe-pi-68732195ab217303-exec-1] is pending + Normal SparkExecutorRunning 7m10s spark-application-controller Executor [comet-dataframe-pi-68732195ab217303-exec-1] is running + Normal SparkExecutorCompleted 7m5s spark-application-controller Executor [comet-dataframe-pi-68732195ab217303-exec-1] completed + Normal SparkDriverCompleted 7m4s spark-application-controller Driver comet-dataframe-pi-driver completed ``` Get Driver Logs ```bash -kubectl logs spark-pi-driver +kubectl logs comet-dataframe-pi-driver ``` +The driver log contains the computed approximation (`Pi is roughly ...`). To confirm that Comet +executed the DataFrame query, inspect the executor log for the native plan emitted by Review Comment: Set `spark.kubernetes.executor.deleteOnTermination` to `false` in 4a1bb1c, documented why the executor is retained, and added the cleanup command. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
