Gangadharayya Hiremath created FLINK-40252:
----------------------------------------------
Summary: Graceful TaskManager termination on Kubernetes
Key: FLINK-40252
URL: https://issues.apache.org/jira/browse/FLINK-40252
Project: Flink
Issue Type: Improvement
Components: Deployment / Kubernetes
Reporter: Gangadharayya Hiremath
{quote}*Motivation*
On Kubernetes, a TaskManager pod is handled identically whether it's lost
voluntarily (node drain, spot/preemptible reclaim, rolling upgrade) or crashes
outright: the ResourceManager only learns it's gone once the heartbeat times
out (50s by default), and {{SIGTERM}} tears the pod down immediately, giving
already-running tasks no chance to make further progress before being torn down
and rescheduled.
*Proposed change*
Add an opt-in {{kubernetes.taskmanager.termination-grace-period}} config option
that gives a TaskManager pod a real, bounded grace window on voluntary eviction:
* A new {{TerminationGracePeriodDecorator}} sets
{{terminationGracePeriodSeconds}} and a {{preStop}} hook on the TaskManager pod
(never overriding a value already present in the user's pod template).
* The {{preStop}} hook sends {{SIGUSR2}} to the main process, routed by
{{TaskManagerRunner}} to a new {{{}TaskExecutor#prepareForTermination(){}}},
which proactively disconnects from the ResourceManager instead of waiting to be
timed out, freeing the slot immediately. The hook then sleeps for most of the
remaining grace period before returning (Kubernetes only sends {{SIGTERM}} once
{{preStop}} returns), so already-running tasks keep making progress in the
meantime.
Separately, this fixes a long-standing {{TODO}} in
{{ResourceManager#closeTaskManagerConnection}} ({{{}// TODO :: suggest failed
task executor to stop itself{}}}): it already calls back into the presumed-dead
TaskExecutor via {{TaskExecutorGateway#disconnectResourceManager}} — but that
RPC tells the TaskExecutor to {_}reconnect{_}, the wrong instruction for a
TaskExecutor the ResourceManager has already unregistered and told every
JobMaster to fail. Adds {{{}TaskExecutorGateway#fenceAndStop{}}}, called here
instead.
*Scope*
Deliberately out of scope for this first pass: equivalent JobManager-side
graceful shutdown (needs its own {{SIGUSR2}} handler first), and a
TaskManager-side proactive self-fencing/isolation timeout for a true network
partition.
*Pull request:* [https://github.com/apache/flink/pull/28729]
{quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
