jerryshao opened a new issue, #14014: URL: https://github.com/apache/gravitino/issues/14014
### Version main branch ### Describe what's wrong `JobManager` runs two background tasks with `scheduleAtFixedRate`: staging-directory cleanup (`cleanUpStagingDirs`) and job status polling (`pullAndUpdateJobStatus`). Neither thread factory sets an uncaught-exception handler, and the returned futures are never read. Under the `ScheduledExecutorService` contract, an exception that escapes a fixed-rate task cancels all later runs of that task, and the exception stays in the unread future, so it is never logged. Both tasks make metadata reads outside any try: - `pullAndUpdateJobStatus` calls `MetalakeManager.listInUseMetalakes` and then `listJobs(metalake, ...)` for each metalake. Only the per-job `pullAndUpdateOwnedJobStatus` is guarded. - `cleanUpStagingDirs` makes the same two calls. Its per-job delete loop catches only `OptimisticLockException` and `IOException`. - `listJobs` starts with `checkMetalake`, which throws `MetalakeNotInUseException` / `NoSuchMetalakeException` if the metalake was disabled or dropped after `listInUseMetalakes` returned it. Both `listJobs` and `listInUseMetalakes` wrap a store `IOException` in a `RuntimeException`. After one such failure, no job changes status again. Jobs stay `QUEUED` / `STARTED` / `CANCELLING` after their processes exit, and their templates can't be deleted. If cleanup dies, staging directories pile up until the disk fills or the server restarts. ### Error message and/or stacktrace None. The exception is swallowed by the `ScheduledFuture`. ### How to reproduce 1. Make the entity store fail once during a status poll (e.g. the backing database restarts or fails over). Or disable a metalake between `listInUseMetalakes` and `listJobs` in a poll. 2. Submit a new job and let it finish. 3. The job never leaves `QUEUED` / `STARTED`, and nothing about the stopped task appears in the server log. ### Additional context Possible fix: in both tasks, catch and log failures for each metalake and continue with the next one. Add a top-level catch around each task body so nothing can cancel the schedule. Also catch `RuntimeException` in the per-job delete loop. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
