[
https://issues.apache.org/jira/browse/IGNITE-21523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825539#comment-17825539
]
Roman Puchkovskiy commented on IGNITE-21523:
--------------------------------------------
The problem was caused by the following: there seems to be something like a
race between a server socket closure and an acceptance of an incoming
connection. If a client was opening a connection to the server right when the
server was shutdown, the connection seemed to be opened for the client, but the
server never realized that it accepted that connection (and so it did not close
it during its stop procedure). As a result, the connection was alive from the
point of view of the client, but it was abandoned on the server.
I solved this by adding a send of a 'probe' message from the client to the
server right after the connection gets opened at the client. This allows to
detect such a broken channel and reopen it later. It is a form of a one-time
heartbeat. Later, we should introduce a proper heartbeat mechanism (and remove
that 'probe' sending).
> ItNodeRestartsTest.testRestarts is flaky
> ----------------------------------------
>
> Key: IGNITE-21523
> URL: https://issues.apache.org/jira/browse/IGNITE-21523
> Project: Ignite
> Issue Type: Bug
> Reporter: Roman Puchkovskiy
> Assignee: Roman Puchkovskiy
> Priority: Major
> Labels: ignite-3
> Fix For: 3.0.0-beta2
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> After running the test repeatedly, the following happened: after a restart of
> 2 clusterServices, one of the restarted cluster services (port 3344) and one
> of not restarted cluster services (port 3345) could not 'see' each other due
> to metadata obtaining timeout:
> [2024-02-12T14:16:54,968][WARN ][sc-cluster-3345-1409][MetadataStore]
> [default:inrt_tr_3345:[email protected]:3345][56ad9288-7ce4-4f67-b11c-8e1eaf6d6612]
> Timeout getting GetMetadataResp from 127.0.1.1:3344 within 1000 ms, cause:
> java.util.concurrent.TimeoutException: Did not observe any item or terminal
> signal within 1000ms in 'source(MonoDefer)' (and no fallback has been
> configured)
> [2024-02-12T14:16:54,968][WARN ][sc-cluster-3345-1409][MembershipProtocol]
> [default:inrt_tr_3345:[email protected]:3345][updateMembership][SYNC]
> Skipping to add/update member: \{m:
> default:inrt_tr_3344:[email protected]:3344, s: ALIVE, inc: 0}, due
> to failed fetchMetadata call (cause: java.util.concurrent.TimeoutException:
> Did not observe any item or terminal signal within 1000ms in
> 'source(MonoDefer)' (and no fallback has been configured))
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
