[
https://issues.apache.org/jira/browse/IGNITE-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906975#comment-16906975
]
Igor Kamyshnikov commented on IGNITE-7791:
------------------------------------------
Added for history indexing purposes:
This bug is seen for Ignite 2.5 for the following scenario:
1) run ignite server nodes
2) join with a client node
3) kill all server nodes
4) start server nodes again
5) await for the client to reconnect
6) client logs will have a NullPointerException (Failed to reinitialize local
partitions (preloading will be stopped))
7) cache get/put operations executed in non-tx context will hang forever:
{noformat}
Client topology version mismatch, need remap lock request
[reqTopVer=AffinityTopologyVersion [topVer=5, minorTopVer=0],
locTopVer=AffinityTopologyVersion [topVer=5, minorTopVer=1],
req=GridNearLockRequest [topVer=AffinityTopologyVersion [topVer=5,
minorTopVer=0] ...
{noformat}
{noformat}
Starting (re)map for mappings [mappings=[GridNearLockMapping
[node=TcpDiscoveryNode [id=02315613-a512-45af-8983-b06b64ee3dd2,
addrs=[0:0:0:0:0:0:0:1%lo, 10.0.2.15, 10.0.3.23, 127.0.0.1],
sockAddrs=[cas-3/10.0.3.23:47500, /10.0.2.15:47500, /0:0:0:0:0:0:0:1%lo:47500,
/127.0.0.1:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1565685652269, loc=false, ver=2.5.0#20190705-sha1:7d4d79b0,
isClient=false],... futs=[false]]]]]
{noformat}
NPE:
{noformat}
#### INFO 2019-08-13 08:26:20.282 (1565684780282) [exchange-worker-#47] [time]
> Started exchange init [topVer=AffinityTopologyVersion [topVer=4,
minorTopVer=1], crd=false, evt=DISCOVERY_CUSTOM_EVT,
evtNode=941c65d5-3ae5-41d5-98fa-8d300edafc94, customEvt=CacheAffinityCh
angeMessage [id=346351a8c61-b59cbf8f-ca75-46fc-9ebe-a76d424a60af,
topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], exchId=null,
partsMsg=null, exchangeNeeded=true], allowMerge=false]
#### ERROR 2019-08-13 08:26:20.366 (1565684780366) [exchange-worker-#47]
[GridDhtPartitionsExchangeFuture] > Failed to reinitialize local partitions
(preloading will be stopped): GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], discoE
vt=DiscoveryCustomEvent [customMsg=CacheAffinityChangeMessage
[id=346351a8c61-b59cbf8f-ca75-46fc-9ebe-a76d424a60af,
topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], exchId=null,
partsMsg=null, exchangeNeeded=true], affTopVer=AffinityTopologyVersion
[topVer=4, mi
norTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=941c65d5-3ae5-41d5-98fa-8d300edafc94, addrs=[0:0:0:0:0:0:0:1%lo, 10.0.2.15,
10.0.3.22, 127.0.0.1], sockAddrs=[cas-2/10.0.3.22:47500, /10.0.2.15:47500,
/0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500], discPort=
47500, order=1, intOrder=1, lastExchangeTime=1565684762464, loc=false,
ver=2.5.0#20190705-sha1:7d4d79b0, isClient=false], topVer=4, nodeId8=e64e8e54,
msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1565684780279]], nodeId=941c65d5,
evt=DISCOVERY_CUSTOM_EVT]
java.lang.NullPointerException
at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$8.applyx(CacheAffinitySharedManager.java:993)
at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$8.applyx(CacheAffinitySharedManager.java:983)
at
org.apache.ignite.internal.util.lang.IgniteInClosureX.apply(IgniteInClosureX.java:38)
at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.forAllCacheGroups(CacheAffinitySharedManager.java:1118)
at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onChangeAffinityMessage(CacheAffinitySharedManager.java:983)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAffinityChangeRequest(GridDhtPartitionsExchangeFuture.java:1011)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:656)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2419)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2299)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
#### INFO 2019-08-13 08:26:20.368 (1565684780368) [exchange-worker-#47]
[GridDhtPartitionsExchangeFuture] > Finish exchange future
[startVer=AffinityTopologyVersion [topVer=4, minorTopVer=1], resVer=null,
err=java.lang.NullPointerException]
#### ERROR 2019-08-13 08:26:20.457 (1565684780457) [exchange-worker-#47]
[GridCachePartitionExchangeManager] > Failed to wait for completion of
partition map exchange (preloading will not start):
GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryCustomEvent
[customMsg=null, affTopVer=AffinityTopologyVersion [topVer=4, minorTopVer=1],
super=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=941c65d5-3ae5-41d5-98fa-8d300edafc94, addrs=[0:0:0:0:0:0:0:1%lo, 10.0.2.15,
10.0.3.22, 127.0.0.1], sockAddrs=[cas-2/10.0.3.22:47500, /10.0.2.15:47500,
/0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500], discPort=47500, order=1,
intOrder=1, lastExchangeTime=1565684762464, loc=false,
ver=2.5.0#20190705-sha1:7d4d79b0, isClient=false], topVer=4, nodeId8=e64e8e54,
msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1565684780279]],
crd=TcpDiscoveryNode [id=941c65d5-3ae5-41d5-98fa-8d300edafc94,
addrs=[0:0:0:0:0:0:0:1%lo, 10.0.2.15, 10.0.3.22, 127.0.0.1],
sockAddrs=[cas-2/10.0.3.22:47500, /10.0.2.15:47500, /0:0:0:0:0:0:0:1%lo:47500,
/127.0.0.1:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1565684762464, loc=false, ver=2.5.0#20190705-sha1:7d4d79b0,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=4, minorTopVer=1],
discoEvt=DiscoveryCustomEvent [customMsg=null,
affTopVer=AffinityTopologyVersion [topVer=4, minorTopVer=1],
super=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=941c65d5-3ae5-41d5-98fa-8d300edafc94, addrs=[0:0:0:0:0:0:0:1%lo, 10.0.2.15,
10.0.3.22, 127.0.0.1], sockAddrs=[cas-2/10.0.3.22:47500, /10.0.2.15:47500,
/0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500], discPort=47500, order=1,
intOrder=1, lastExchangeTime=1565684762464, loc=false,
ver=2.5.0#20190705-sha1:7d4d79b0, isClient=false], topVer=4, nodeId8=e64e8e54,
msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1565684780279]], nodeId=941c65d5,
evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter
[ignoreInterrupts=false, state=DONE, res=false, hash=859557540], init=false,
lastVer=null, partReleaseFut=null, exchActions=null,
affChangeMsg=CacheAffinityChangeMessage
[id=346351a8c61-b59cbf8f-ca75-46fc-9ebe-a76d424a60af,
topVer=AffinityTopologyVersion [topVer=4, minorTopVer=0], exchId=null,
partsMsg=null, exchangeNeeded=true], initTs=1565684780279,
centralizedAff=false, forceAffReassignment=false, changeGlobalStateE=null,
done=true, state=CLIENT, evtLatch=0,
remaining=[73419b18-8475-499d-852a-0b6ef599cf3c,
2e84a25b-9993-4a0b-8275-030281eba41a, 941c65d5-3ae5-41d5-98fa-8d300edafc94],
super=GridFutureAdapter [ignoreInterrupts=false, state=DONE,
res=java.lang.NullPointerException, hash=1078706880]]
class org.apache.ignite.IgniteCheckedException: null
at
org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7307)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:259)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:207)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:159)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:151)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2433)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2299)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$8.applyx(CacheAffinitySharedManager.java:993)
at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$8.applyx(CacheAffinitySharedManager.java:983)
at
org.apache.ignite.internal.util.lang.IgniteInClosureX.apply(IgniteInClosureX.java:38)
at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.forAllCacheGroups(CacheAffinitySharedManager.java:1118)
at
org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onChangeAffinityMessage(CacheAffinitySharedManager.java:983)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAffinityChangeRequest(GridDhtPartitionsExchangeFuture.java:1011)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:656)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:2419)
... 3 more
{noformat}
Cherry-picking of the commit of the fix proposed in this ticket seems to work
out.
Similar stories on internet:
http://apache-ignite-users.70518.x6.nabble.com/NPE-When-joining-grid-td22326.html
http://mail-archives.apache.org/mod_mbox/ignite-user/201903.mbox/%[email protected]%3e
> Ignite Client Nodes: failed test
> IgniteClientReconnectCacheTest.testReconnectCacheDestroyedAndCreated()
> -------------------------------------------------------------------------------------------------------
>
> Key: IGNITE-7791
> URL: https://issues.apache.org/jira/browse/IGNITE-7791
> Project: Ignite
> Issue Type: Bug
> Reporter: Sergey Chugunov
> Assignee: Maxim Muzafarov
> Priority: Major
> Labels: MakeTeamcityGreenAgain
> Fix For: 2.7
>
> Attachments: IgniteClientReconnectCacheDelayExchangeTest.java
>
>
> Test is flaky, success rate: 32.4%, test history is
> [here|https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&testNameId=-9174769196124217030&tab=testDetails]
> Reproducible locally.
> Test fails on waiting for disco cache content:
> {noformat}
> junit.framework.AssertionFailedError
> at junit.framework.Assert.fail(Assert.java:55)
> at junit.framework.Assert.assertTrue(Assert.java:22)
> at junit.framework.Assert.assertTrue(Assert.java:31)
> at junit.framework.TestCase.assertTrue(TestCase.java:201)
> at
> org.apache.ignite.internal.IgniteClientReconnectCacheTest.checkCacheDiscoveryData(IgniteClientReconnectCacheTest.java:1414)
> at
> org.apache.ignite.internal.IgniteClientReconnectCacheTest.testReconnectCacheDestroyedAndCreated(IgniteClientReconnectCacheTest.java:897)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at junit.framework.TestCase.runTest(TestCase.java:176)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2001)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:133)
> at
> org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:1916)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> This fail may be caused by another exception earlier in the log:
> {noformat}
> [2018-02-22
> 14:32:57,972][ERROR][exchange-worker-#3025%internal.IgniteClientReconnectCacheTest3%][GridDhtPartitionsExchangeFuture]
> Failed to reinitialize local partitions (preloading will be stopped):
> GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=6,
> minorTopVer=1], discoEvt=DiscoveryCustomEvent
> [customMsg=CacheAffinityChangeMessage
> [id=3cea94db161-760b7b18-b776-4ee7-ae5a-64f9494eaa36,
> topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], exchId=null,
> partsMsg=null, exchangeNeeded=true], affTopVer=AffinityTopologyVersion
> [topVer=6, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=1b7f7330-69d8-4bd0-b2e9-717736100000, addrs=[127.0.0.1],
> sockAddrs=[/127.0.0.1:47500], discPort=47500, order=1, intOrder=1,
> lastExchangeTime=1519299177395, loc=false, ver=2.5.0#19700101-sha1:00000000,
> isClient=false], topVer=6, nodeId8=658aeb36, msg=null,
> type=DISCOVERY_CUSTOM_EVT, tstamp=1519299177971]], nodeId=1b7f7330,
> evt=DISCOVERY_CUSTOM_EVT]
> java.lang.AssertionError: 236160867
> at
> org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$CachesInfo.group(CacheAffinitySharedManager.java:2636)
> at
> org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$8.applyx(CacheAffinitySharedManager.java:989)
> at
> org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager$8.applyx(CacheAffinitySharedManager.java:983)
> at
> org.apache.ignite.internal.util.lang.IgniteInClosureX.apply(IgniteInClosureX.java:38)
> at
> org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.forAllCacheGroups(CacheAffinitySharedManager.java:1118)
> at
> org.apache.ignite.internal.processors.cache.CacheAffinitySharedManager.onChangeAffinityMessage(CacheAffinitySharedManager.java:983)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onAffinityChangeRequest(GridDhtPartitionsExchangeFuture.java:992)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:648)
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2337)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
