Till Rohrmann created FLINK-12863:
-------------------------------------
Summary: Race condition between slot offerings and
AllocatedSlotReport
Key: FLINK-12863
URL: https://issues.apache.org/jira/browse/FLINK-12863
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.9.0
Reporter: Till Rohrmann
Fix For: 1.9.0
With FLINK-11059 we introduced the {{AllocatedSlotReport}} which is used by the
{{TaskExecutor}} to synchronize its internal view on slot allocations with the
view of the {{JobMaster}}. It seems that there is a race condition between
offering slots and receiving the report because the {{AllocatedSlotReport}} is
sent by the {{HeartbeatManagerSenderImpl}} from a separate thread.
Due to that it can happen that we generate an {{AllocatedSlotReport}} just
before getting new slots offered. Since the report is sent from a different
thread, it can then happen that the response to the slot offerings is sent
earlier than the {{AllocatedSlotReport}}. Consequently, we might receive an
outdated slot report on the {{TaskExecutor}} causing active slots to be
released.
In order to solve the problem I propose to add a fencing token to the
{{AllocatedSlotReport}} which is being updated whenever we offer new slots to
the {{JobMaster}}. When we receive the {{AllocatedSlotReport}} on the
{{TaskExecutor}} we compare the current slot report fencing token with the
received one and only process the report if they are equal. Otherwise we wait
for the next heartbeat to send us an up to date {{AllocatedSlotReport}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
