jerryshao opened a new issue, #13554: URL: https://github.com/apache/gravitino/issues/13554
### Describe the proposal The job system only ships the `local` job executor. It runs every job as a child process of the Gravitino server, and its docs say it is meant for testing. This makes it unsuitable for production: - Jobs compete with the server for CPU and memory, and one host caps how many can run at once. - Job state and output belong to the node that launched the job, so a server restart loses running jobs. - `JobManager` downloads every job resource into the server's local staging directory. An executor that runs jobs elsewhere can't use those paths. - Built-in job templates point at the jobs jar on the server's filesystem. All built-in jobs (Iceberg maintenance and Spark Pi) are Spark jobs. This epic adds a production executor, `spark-k8s`. It runs `SPARK` templates as `SparkApplication` resources of [apache/spark-kubernetes-operator](https://github.com/apache/spark-kubernetes-operator), created through the fabric8 Kubernetes client: - **Stateless and multi-node safe:** all job state lives in Kubernetes, so any server can submit, query and cancel any job. - **Full lifecycle through the existing job REST APIs:** submit, status, cancel and driver output. There are no API changes. - **Resources:** each executor decides whether to fetch job resources. The SPI gains `JobContext` and `submitJob(JobContext, JobTemplate)`, and the default implementation keeps existing executors unchanged. - **Job types:** executors declare the job types they support, so `SHELL` templates on `spark-k8s` are rejected up front. `SHELL` stays local-only, for testing. - **Spark version:** built-in jobs stay on Spark 3.5. The supported operator line is 0.9.x, the last line that supports Spark 3.5, because operator 1.0+ requires Spark 4.0+. - **Kubernetes integration:** the Spark cluster can differ from Gravitino's own cluster, and queueing and gang scheduling are left to YuniKorn or Volcano. A POC runs the built-in Spark Pi job end to end on a local cluster with operator 0.9.0: submit, status, driver output, cancel and failure all work. ### Task list Core - [ ] Add `JobExecutor.supportedJobTypes()` and check it in `JobManager.runJob` - [ ] Move resource fetching into executors: add `JobContext`, `JobResourceFetcher` and `submitJob(JobContext, JobTemplate)` Spark on Kubernetes executor (`plugins/spark-k8s-job-executor`) - [ ] Module skeleton: shaded fabric8 client, configuration, SparkApplication builder, submit/status/cancel - [ ] Driver output, retention/TTL, and environment variables through Secrets - [ ] Run built-in jobs through the `builtinJobsJar` URI - [ ] Concurrency control: operator start timeouts, `maxActiveApplications`, cached per-namespace status LIST - [ ] Cross-cluster connection: `kubeconfig`/`context` or `masterUrl`/`caCertFile`/`tokenFile` - [ ] Register `spark-k8s` in `JobExecutorFactory` and ship the jar in the distribution - [ ] Dockerfile for the `gravitino-spark` image (Spark 3.5) with the built-in jobs jar Docs and verification - [ ] Deployment guide: supported operator and Spark versions, RBAC, image, configuration, cross-cluster, scheduler - [ ] Local end-to-end testing guide (OrbStack or kind) - [ ] Optional K3s-based integration test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
