Pierre Villard created NIP-52:
---------------------------------
Summary: Reconnecting node can revert newer cluster node statuses
received during flow synchronization
Key: NIP-52
URL: https://issues.apache.org/jira/browse/NIP-52
Project: NiFi Improvement Proposal
Issue Type: Bug
Reporter: Pierre Villard
Assignee: Pierre Villard
When a node handles a reconnection request,
*StandardFlowService.handleReconnectionRequest* applies the node connection
statuses from the request twice. The first time is before the flow is loaded,
via loadFromConnectionResponse → initializeNodeConnectionState. The second time
is through ClusterCoordinator.resetNodeStatuses after the flow has been loaded.
Because resetNodeStatuses replaces statuses unconditionally, any status update
the node received while synchronizing the flow is reverted to the older value
from the request.
If a peer became CONNECTED during that window, the reconnecting node keeps
seeing it as CONNECTING. If the node is then offloaded, for example
decommissioned shortly after a rolling restart, its load-balanced queues never
transfer FlowFiles to that peer. The load-balance client only sends to
CONNECTED peers, and heartbeats stop at disconnect, so the view is never
corrected. The offload stays in OFFLOADING indefinitely, with FlowFiles left on
the node, until the node is restarted.
The second application should be replaced by a notification of the current
statuses to cluster topology listeners. Queues created while loading the flow
still align with the current topology, without reverting newer statuses. The
comment on resetNodeStatuses, which describes an update-id comparison it
doesn’t perform, should also be corrected.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)