[
https://issues.apache.org/jira/browse/IGNITE-29052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Evgeny Stanilovsky updated IGNITE-29052:
----------------------------------------
Fix Version/s: 2.19
> Fix flaky IgniteCacheGroupsTest.testRestartsAndCacheCreateDestroy
> -----------------------------------------------------------------
>
> Key: IGNITE-29052
> URL: https://issues.apache.org/jira/browse/IGNITE-29052
> Project: Ignite
> Issue Type: Bug
> Reporter: Dmitry Werner
> Assignee: Dmitry Werner
> Priority: Major
> Labels: MakeTeamcityGreenAgain
> Fix For: 2.19
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Success Rate: 21.5% (Last 107 Runs) 84 failed, 23 successful
> https://ci2.ignite.apache.org/test/7147504629922790672?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E
> Current failure (from CI log _Apache_Ignite_2.x_Tests_Cache_3_43882.log)
> The test starts at 15:19:30 and is interrupted after the hardened 600s
> timeout (getTestTimeout() = 10 * 60_000 overridden by suite/CI watchdog).
> Reported failure:
> [ERROR]
> org.apache.ignite.internal.processors.cache.IgniteCacheGroupsTest.testRestartsAndCacheCreateDestroy
> -- Time elapsed: 601.1 s <<< ERROR!
> java.util.concurrent.TimeoutException: Test has been timed out
> [test=testRestartsAndCacheCreateDestroy, timeout=600000]
> Main thread stack (at dump time, blocked in `.get()`):
>
> o.a.i.IgniteCacheGroupsTest.testRestartsAndCacheCreateDestroy(IgniteCacheGroupsTest.java:3865)
> at
> o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:147)
> at
> o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:184)
> at jdk.internal.misc.Unsafe.park(Native Method)
> Cache destroy thread (`async-runnable-runner-1`) blocked in
> `destroyCache`:
> o.a.i.IgniteCacheGroupsTest$41.run(IgniteCacheGroupsTest.java:3791)
> at o.a.i.i.IgniteKernal.destroyCache(IgniteKernal.java:2374)
> at o.a.i.i.IgniteKernal.destroyCache0(IgniteKernal.java:2384)
> at
> o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:147)
> at
> o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:184)
> at java.util.concurrent.locks.LockSupport.park(Native Method)
> All op threads (`op-thread-3..10`) parked on operation futures, e.g.:
>
> o.a.i.IgniteCacheGroupsTest.cacheOperation(IgniteCacheGroupsTest.java:4120)
> // put
>
> o.a.i.IgniteCacheGroupsTest.cacheOperation(IgniteCacheGroupsTest.java:4155)
> // putAll
>
> o.a.i.IgniteCacheGroupsTest.cacheOperation(IgniteCacheGroupsTest.java:4145)
> // getAll
> at
> o.a.i.i.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:3906)
> at
> o.a.i.i.util.future.GridFutureAdapter.get(GridFutureAdapter.java:147)
> at
> o.a.i.i.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:184)
> at java.util.concurrent.locks.LockSupport.park(Native Method)
> *Root cause of the test failure*
> IgniteCacheGroupsTest#testRestartsAndCacheCreateDestroy fails not due to test
> timeouts or hung transactions, but due to a client-initiated exchange
> deadlock (dynamic cache change) racing against a server node restart that
> rejoins with the same consistentId/port.
> *Mechanism:*
> 1. The client initiates a dynamic cache change (destroy/create c0) — a
> separate distributed exchange (topVer=6.3) started from the client node.
> 2. At the same time the restart thread stops a server node (node 2,
> consistentId=127.0.0.1:47502) and restarts it with the same
> consistentId/port. The rejoin spawns its own chain of exchanges (NODE_LEFT,
> NODE_JOINED, …) which merge with the client's exchange.
> 3. The client waits for the coordinator to confirm exchange 6.3
> ("waiting for coordinator response"), but its partition single-messages are
> never acknowledged (msgsAckedByRmt=0, recovery descriptor connected=false).
> 4. Because the client's exchange is never completed, the cluster-wide
> merge is blocked; startGrid(2) in the restart thread hangs on await exchange,
> and restartFut.get(...) fails with IgniteFutureTimeoutCheckedException at
> IgniteCacheGroupsTest.java:3904.
> Key detail: the restarted node keeps its identity (consistentId/discPort),
> which breaks the client→coordinator recovery session during the rejoin. It's
> a race of "client-initiated dynamic-cache-change ⇄ restart of a node with the
> same consistentId".
> *Proposed fix*
> Synchronize client create/destroy and node restarts with a shared
> ReentrantLock restartLock in the test:
> - restart thread: lock → stopGrid/startGrid → unlock;
> - cache thread: lock → destroyCache/createCache → unlock.
> A node restart therefore executes sequentially and never overlaps an
> in-flight client create/destroy (which is what triggers the deadlock), while
> ordinary cache operations continue running concurrently with restarts. The
> client stays the initiator of create/destroy, so test coverage is preserved.
> A side refinement was made to the group assertion loop to iterate over both
> GROUP1/GROUP2 and tolerate a group being absent on a node (all its caches
> re-created under another group during the iteration).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
