Ksenia Rybakova created IGNITE-7570:
---------------------------------------
Summary: Client nodes failed with "Failed to process invalid
partitions response" during failover
Key: IGNITE-7570
URL: https://issues.apache.org/jira/browse/IGNITE-7570
Project: Ignite
Issue Type: Bug
Affects Versions: 2.4
Reporter: Ksenia Rybakova
Some client nodes fail with "Failed to process invalid partitions response"
during failover test:
{noformat}
[2018-01-30 16:27:58,610][INFO ][sys-#190][GridDhtPartitionsExchangeFuture]
Received full message, will finish exchange
[node=80ebd2ac-1432-4bfc-bab7-d9dbf56cdeb4, resVer=AffinityTopologyVersion
[topVer=37, minorTopVer=0]]
[2018-01-30 16:27:58,688][INFO ][sys-#190][GridDhtPartitionsExchangeFuture]
Finish exchange future [startVer=AffinityTopologyVersion [topVer=37,
minorTopVer=0], resVer=AffinityTopologyVersion [topVer=37, minorTopVer=0],
err=null]
<16:27:58><benchmark-worker-32><yardstick> The benchmark of random operation
failed.
javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException:
Failed to process invalid partitions response (remote node reported invalid
partitions but remote topology version does not differ from local)
[topVer=AffinityTopologyVersion [topVer=37, minorTopVer=0],
rmtTopVer=AffinityTopologyVersion [topVer=37, minorTopVer=0], part=204,
nodeId=80ebd2ac-1432-4bfc-bab7-d9dbf56cdeb4]
at
org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1294)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1673)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:852)
at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:676)
at
org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.doGet(IgniteCacheRandomOperationBenchmark.java:776)
at
org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.executeRandomOperation(IgniteCacheRandomOperationBenchmark.java:624)
at
org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.executeOutOfTx(IgniteCacheRandomOperationBenchmark.java:602)
at
org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark.test(IgniteCacheRandomOperationBenchmark.java:207)
at org.yardstickframework.impl.BenchmarkRunner$2.run(BenchmarkRunner.java:178)
at java.lang.Thread.run(Thread.java:748)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to process
invalid partitions response (remote node reported invalid partitions but remote
topology version does not differ from local) [topVer=AffinityTopologyVersion
[topVer=37, minorTopVer=0], rmtTopVer=AffinityTopologyVersion [topVer=37,
minorTopVer=0], part=204, nodeId=80ebd2ac-1432-4bfc-bab7-d9dbf56cdeb4]
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.checkError(GridPartitionedSingleGetFuture.java:596)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.onResult(GridPartitionedSingleGetFuture.java:505)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheAdapter.processNearSingleGetResponse(GridDhtCacheAdapter.java:349)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$1400(GridDhtAtomicCache.java:130)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$15.apply(GridDhtAtomicCache.java:422)
at
org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$15.apply(GridDhtAtomicCache.java:417)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1060)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:378)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:304)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:99)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:293)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
at
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:499)
... 1 more
Finishing main test [ts=1517318878863, date=Tue Jan 30 16:27:58 MSK 2018]
ERROR: Shutting down benchmark driver to unexpected exception.
{noformat}
Test config:
CacheRandomOperationBenchmark
- 20 server nodes, 10 client nodes at 10 hosts
- 34 caches with different configs with and without PDS, 3 backups
- preload amount 250
- key range 500K
- operations: PUT PUT_ALL GET GET_ALL INVOKE INVOKE_ALL REMOVE REMOVE_ALL
PUT_IF_ABSENT REPLACE
- 2 of 20 servers are being restarted every 15 minutes
Complete yardstick configs are attached.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
