jerryshao opened a new issue, #14058: URL: https://github.com/apache/gravitino/issues/14058
### Describe the subtask Follow-ups of the `k8s` job executor added in #13613, #13616 and #13617. **Harden the k8s executor** - Check a terminal driver pod phase (`Succeeded`, `Failed`) before the driver container state, which may still say running after the node is lost. - Clean up the SparkApplication after a create that fails with a 5xx, as after an unreachable API server: it may have been created. - Paginate the namespace listing of the status cache, and don't treat a slow listing as an unreachable API server: fall back to getting the jobs one by one instead. - Delete with a `resourceVersion` precondition where a read decides the deletion (the no-status cleanup and the failed-submit cleanup), and read again on a conflict with bounded retries. An application that has got a status meanwhile follows the normal status path. - Add tests for the submit cleanup, the logs of a deleted application, `isDriverRunning` and the cache eviction. **Sweep orphaned SparkApplications** - A SparkApplication can be left without a job in the store, for example by a submission whose cleanup couldn't reach the API server, or after its job is deleted from the store. - Add `JobExecutor#deleteOrphanedJobs(Set<String> knownJobExecutionIds)`, a no-op by default. `JobManager` calls it periodically with the execution ids of the jobs of all metalakes. - The k8s executor deletes only the SparkApplications of its cluster and name prefix, managed by Gravitino, whose execution id isn't known and that are older than a grace period, which covers a submission whose job isn't stored yet. It lists them across namespaces, and falls back to the configured namespace and the namespaces of the known jobs if that is forbidden. **Job status poll log noise** - `JobManager` logs an ERROR with a full stack trace for every job on every poll while the job executor fails, for example while the API server is unreachable, or for a job of another cluster. Log one summary per poll instead, and the details of each job at DEBUG. **Output size Javadoc** - `JobExecutor#getJobStdout`/`getJobStderr` and `gravitino.job.outputMaxBytes` say `maxBytes` bounds both the read cost and the response size. It bounds the response size, but an executor may not bound the bytes it reads: the k8s executor reads the last lines of the pod log. ### Parent issue #13554 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
