Andrey Aleksandrov created IGNITE-8098:
------------------------------------------
Summary: Getting affinity for topology version earlier than
affinity is calculated because of data race
Key: IGNITE-8098
URL: https://issues.apache.org/jira/browse/IGNITE-8098
Project: Ignite
Issue Type: Bug
Affects Versions: 2.3
Reporter: Andrey Aleksandrov
>From time to time the Ignite cluster with services throws next exception
>during restarting of some nodes:
java.lang.IllegalStateException: Getting affinity for topology version earlier
than affinity is calculated [locNode=TcpDiscoveryNode
[id=c770dbcf-2908-442d-8aa0-bf26a2aecfef, addrs=[10.44.162.169, 127.0.0.1],
sockAddrs=[clrv0000041279.ic.ing.net/10.44.162.169:56500, /127.0.0.1:56500],
discPort=56500, order=11, intOrder=8, lastExchangeTime=1520931375337, loc=true,
ver=2.3.3#20180213-sha1:f446df34, isClient=false], grp=ignite-sys-cache,
topVer=AffinityTopologyVersion [topVer=13, minorTopVer=0],
head=AffinityTopologyVersion [topVer=15, minorTopVer=0],
history=[AffinityTopologyVersion [topVer=11, minorTopVer=0],
AffinityTopologyVersion [topVer=11, minorTopVer=1], AffinityTopologyVersion
[topVer=12, minorTopVer=0], AffinityTopologyVersion [topVer=15, minorTopVer=0]]]
Looks like the reason of this issue is the data race in GridServiceProcessor
class.
How to reproduce:
1)To simulate data race you should update next place in source code:
Class: GridServiceProcessor
Method: @Override public void onEvent(final DiscoveryEvent evt, final
DiscoCache discoCache) {
Place:
....
try {
svcName.set(dep.configuration().getName());
ctx.cache().internalCache(UTILITY_CACHE_NAME).context().affinity().
affinityReadyFuture(topVer).get();
//HERE (between GET and REASSIGN) you should add Thread.sleep(100) for example.
//try {
//Thread.sleep(100);
//}
//catch (InterruptedException e1) {
//e1.printStackTrace();
//}
reassign(dep, topVer);
}
catch (IgniteCheckedException ex) {
if (!(e instanceof ClusterTopologyCheckedException))
LT.error(log, ex, "Failed to do service reassignment (will retry): " +
dep.configuration().getName());
retries.add(dep);
}
...
2)After that you should imitate start/shutdown iterations. For reproducing I
used GridServiceProcessorBatchDeploySelfTest (but timeout on future.get should
be increased to avoid timeout error)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
