Martijn Visser created FLINK-40078:
--------------------------------------
Summary: Lost non-loss-tolerant OperatorEvent to an
already-failing task triggers a failover that masks the original failure cause
Key: FLINK-40078
URL: https://issues.apache.org/jira/browse/FLINK-40078
Project: Flink
Issue Type: Bug
Reporter: Martijn Visser
When an OperatorCoordinator sends an event to a subtask that has just failed
for an application reason, the delivery fails with {{TaskNotRunningException}}
({{Task.deliverOperatorEvent}}). Because the event is not loss-tolerant,
{{SubtaskGatewayImpl.sendEvent}} escalates this into
{code}
org.apache.flink.util.FlinkException: An OperatorEvent from an
OperatorCoordinator to a task was lost. Triggering task failover to ensure
consistency. Event: '[NoMoreSplitEvent]', ...
{code}
and triggers a task failover. Under {{NoRestartBackoffTimeStrategy}} that
lost-event failover can win the race to the scheduler and become the job's
reported terminal failure cause, completely replacing the original application
error in the cause chain (the coordinator is co-located with the JobMaster, so
its failure path can reach the scheduler faster than the failed task's own
failure RPC).
The consequence is that a user debugging a failed job sees "An OperatorEvent
... was lost. Triggering task failover to ensure consistency" instead of their
actual error. This is general: any coordinator event in flight at the moment a
task fails for an application reason can mask that failure. It was observed
concretely in FLINK-40077, where a {{NoMoreSplitsEvent}} in flight to a task
that intentionally failed on the NOT NULL enforcer replaced the enforcer error
as the reported cause (there it was worked around in the test scaffolding; this
ticket is about the runtime behavior itself).
The escalation mechanism already has an escape hatch: on
{{TaskNotRunningException}}, {{SubtaskGatewayImpl}} returns without failover
when {{evt.isLossTolerant()}} is true. {{NoMoreSplitsEvent}} is currently not
marked loss-tolerant.
Possible directions (needs input from runtime/coordination owners, as this
touches checkpoint/failover consistency semantics):
* Mark {{NoMoreSplitsEvent}} loss-tolerant, since a task that is FAILED or
FINISHED has no further use for it.
* More generally, suppress the redundant lost-event failover when the target
task is already failing/failed for another reason, so the original cause is
preserved.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
