jerryshao opened a new issue, #14046: URL: https://github.com/apache/gravitino/issues/14046
### Version main branch ### Describe what's wrong The local job executor gets a new identity on every server start. A restarted server therefore no longer owns the jobs it ran before the restart. Those jobs stay queued, started or cancelling until they expire after `gravitino.job.stagingDirKeepTimeInMs`, up to about 7.7 days by default. While they are active, their job template can't be deleted. The job processes are gone, and only expiry ever settles these jobs, so cancel is the only thing an operator can do. That doesn't help either: - Cancelling a job that this server doesn't own only marks it as `CANCELLING`, for its owner to cancel at the owner's next status pull. - The owner no longer exists, so nothing ever moves the job out of `CANCELLING`. - `deleteJobTemplate` keeps failing with `InUseException`. The same applies to a multi-server deployment when one server exits for good. ### Error message and/or stacktrace ``` InUseException: Job template <template> under metalake <metalake> has active jobs associated with it ``` ### How to reproduce 1. Use the local job executor, and register a shell job template that sleeps for a long time. 2. Run a job from it and wait until it is `STARTED`. 3. Restart the Gravitino server. 4. Cancel the job. It moves to `CANCELLING`. 5. Wait any number of status pull intervals. The job is still `CANCELLING`. 6. Delete the job template. It fails with `InUseException`. ### Additional context The local job executor can't resume these jobs, because their state lived in the previous server process. The goal is only to let an operator clear them. Proposed fix: - **When to settle:** when a node-local job owned by another executor is still `CANCELLING` three status pull intervals after the cancel request, the server pulling job statuses marks it as `CANCELLED`. - **Why a live owner isn't affected:** a live owner answers a cancellation within about two intervals, so cross-server cancellation keeps working. - **Concurrent writes:** if several servers settle the same job at once, OCC resolves it. This relies on all servers using the same `gravitino.job.statusPullIntervalInMs` and keeping their clocks in sync. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
