Martijn Visser created FLINK-40076:
--------------------------------------
Summary:
RescaleTimelineITCase.testRecordNonTerminatedRescaleMergingWithNewRecoverableFailureTriggerCause
fails intermittently: snapshot taken before recoverable failover is merged
into in-progress rescale
Key: FLINK-40076
URL: https://issues.apache.org/jira/browse/FLINK-40076
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 2.4.0
Reporter: Martijn Visser
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=76751&view=results
(leg: test_cron_adaptive_scheduler core)
{code}
org.opentest4j.AssertionFailedError:
expected: RECOVERABLE_FAILOVER
but was: UPDATE_REQUIREMENT
at
org.apache.flink.runtime.scheduler.adaptive.timeline.RescaleTimelineITCase.lambda$testRecordNonTerminatedRescaleMergingWithNewRecoverableFailureTriggerCause$17(RescaleTimelineITCase.java:582)
at
org.apache.flink.runtime.scheduler.adaptive.timeline.RescaleTimelineITCase.runAdaptedParameterizedAssertion(RescaleTimelineITCase.java:630)
at
org.apache.flink.runtime.scheduler.adaptive.timeline.RescaleTimelineITCase.testRecordNonTerminatedRescaleMergingWithNewRecoverableFailureTriggerCause(RescaleTimelineITCase.java:574)
{code}
The test fails fast (~0.4s), commit 3196a998cf0. Distinct from the earlier
races fixed in this class (FLINK-39902, FLINK-39903, FLINK-40009, FLINK-40010)
and from the open FLINK-40067, which all target other methods.
Root cause: after {{miniCluster.terminateTaskManager(0)}} the test synchronizes
via {{waitForVertexParallelismReachedAndJobRunning(..., PARALLELISM)}}.
{{CommonTestUtils.waitUntilCondition}} evaluates its predicate before sleeping,
and immediately after the TaskManager loss the job is still observed RUNNING at
PARALLELISM, so the wait returns on stale pre-failover state and provides no
synchronization (after the failover the job runs at lower parallelism, so the
condition can only ever match the stale state). The {{ExecutionGraphInfo}}
snapshot can therefore be taken before
{{AdaptiveScheduler.recordRescaleForJobRestarting}} merges the failover into
the in-progress UPDATE_REQUIREMENT rescale and re-stamps its trigger cause to
RECOVERABLE_FAILOVER. Product behavior is correct; the test synchronization is
wrong.
Proposed fix: before snapshotting, wait until the failover has actually been
merged (rescale history size 2 with latest trigger cause RECOVERABLE_FAILOVER),
gated on the enabled-history parameterization (the disabled-history leg asserts
an empty history and needs no wait). Assertions unchanged. Verified locally
with 25 iterations (50 executions across both parameterizations), all green.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
