jerryshao opened a new issue, #13618:
URL: https://github.com/apache/gravitino/issues/13618
### Describe the subtask
To use the `k8s` job executor, users need a Spark image that contains the
built-in jobs, and docs on how to deploy and verify it.
- `gravitino-spark` image:
- Add a Dockerfile based on Spark 3.5, with the jobs jar at
`/opt/gravitino/jobs/gravitino-jobs.jar` and `iceberg-spark-runtime-3.5_2.12`.
- Release the image with each Gravitino version, as the jobs jar in the
image must match the Gravitino server version.
- Deployment guide in `docs/manage-jobs-in-gravitino.md`:
- Requirements: apache/spark-kubernetes-operator 0.9.x and Spark 3.5.
- Setup: RBAC, the image, all the `gravitino.jobExecutor.k8s.*`
configurations, cross-cluster deployment, and queueing with a scheduler such as
YuniKorn.
- Known limitations: stderr is merged into stdout; a job whose driver is
evicted fails; the finished time of a cancelled job can be up to one status
pull interval late; unfinished jobs fail if Gravitino is down longer than the
SparkApplication TTL.
- Local end-to-end guide: run a built-in Iceberg job on a local cluster
(OrbStack or kind), from installing the operator to reading the job output.
- An integration test on K3s with Testcontainers, tagged
`gravitino-docker-test` so it doesn't slow down the default CI.
### Parent issue
#13554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
