Rui Fan created FLINK-40081:
-------------------------------
Summary: Enable the spilling backend for checkpointing during
recovery
Key: FLINK-40081
URL: https://issues.apache.org/jira/browse/FLINK-40081
Project: Flink
Issue Type: Sub-task
Components: Runtime / Checkpointing
Reporter: Rui Fan
Assignee: Rui Fan
Swap the memory backend for the disk backend behind
execution.checkpointing.during-recovery.enabled: recovered channel state is
fetched (and rescale-filtered) into spill files, the drainer pushes it into the
recovery-state channels while serving as the recovery-checkpoint trigger, and
checkpoints taken during recovery snapshot the not-yet-drained slice.
Scope:
* Handler factory returns the spilling handlers when the flag is on; the v1
in-memory filtering path is deleted.
* StreamTask flag-on recovery becomes fetch → requestPartitions →
build/install drainer → drain → NO_OP; the transitional in-memory trigger and
conversion-time push are deleted.
* The unbounded heap fallback in RecoveredInputChannel#requestBufferBlocking
is removed — spilling supersedes it, eliminating the OOM risk of the memory
backend.
* RecordFilterContext hardened for minimal and batch environments (spill tmp
directories enforced only when the flag is on).
* IT cases: unaligned checkpoint during recovery, rescale-filter large-record
OOM regression, non-source recovery race smoke test.
The randomized ITCase fleet (which runs the flag in ~50% of runs) must stay
green across the backend swap — the acceptance bar for spilling being a drop-in
replacement of the memory backend.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
