[
https://issues.apache.org/jira/browse/IGNITE-29200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry Werner updated IGNITE-29200:
-----------------------------------
Description:
Success Rate: 61.3% (Last 106 Runs): 41 failed, 65 successful
[https://ci2.ignite.apache.org/test/-5821376133685081776?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E]
Test failed on CI: java.util.concurrent.TimeoutException: Test has been timed
out [test=testConcurrentBuildNewIndexAndRebuildIndexes1, timeout=300000]
*Root cause*
The test needs two build-idx-runner threads at once: a slowed-down index
rebuild (10 ms/row × 100k rows) and a concurrent CREATE INDEX.
The build-index pool defaults to min(4, max(1, CPU / 4))
(IgniteConfiguration.java:178), i.e. 1 thread on low-core CI agents.
The rebuild starts first and occupies the only thread for ~1000 s, the CREATE
INDEX worker stays queued, and the test hangs
waiting for it to start until the 300 s timeout.
*Fix*
Test-only: set buildIndexThreadPoolSize=2 for
testConcurrentBuildNewIndexAndRebuildIndexes0/1 (the cache has a single
partition,
so at most two workers). ...Indexes0 does not hang with 1 thread (create-index
starts first and is released by the test),
but without the pinned pool it runs the two operations sequentially instead of
concurrently as intended.
was:
Success Rate: 61.3% (Last 106 Runs): 41 failed, 65 successful
https://ci2.ignite.apache.org/test/-5821376133685081776?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E
Test failed on CI: java.util.concurrent.TimeoutException: Test has been timed
out [test=testConcurrentBuildNewIndexAndRebuildIndexes1, timeout=300000]
*Root cause*
The test requires two build-idx-runner workers to run concurrently: a
slowed-down forceRebuildIndexes rebuild (10 ms/row over 100k rows) plus a
starting CREATE INDEX whose first processed row releases
slowdownIdxCreateConsumer.startBuildIdxFut.
The build-index thread pool is sized by default as min(4, max(1,
AVAILABLE_PROC_CNT / 4)) (IgniteConfiguration.java:178) and the test never
pinned it. On low-core CPU-limited CI agents this evaluates to 1, so:
1. forceRebuildIndexes starts first and monopolizes the only pool thread for
~1000 s (100k rows × 10 ms).
2. The CREATE INDEX partition worker stays queued in the pool and never starts;
the DDL blocks at GridQueryProcessor.java:2194 (buildIdxFut.get()).
3. The test hangs at ResumeCreateIndexTest.java:252 (startBuildIdxFut.get())
and is killed by the 300 s test timeout. The sleepTime.set(0) speed-up on line
257 is unreachable.
Thread dump at timeout confirms: exactly one build-idx-runner thread in the JVM
executing the rebuild closure, no create-index worker thread.
The sibling testConcurrentBuildNewIndexAndRebuildIndexes0 passes because
create-index starts first and never needs two workers at once — only
...Indexes1's ordering (slow rebuild first) exposes the starvation.
*Fix*
Test-only change (ResumeCreateIndexTest.java): pin the pool the test depends
on: setBuildIndexThreadPoolSize(2) for the node in
testConcurrentBuildNewIndexAndRebuildIndexes1.
Two threads are sufficient — the cache has a single partition
(RendezvousAffinityFunction(false, 1)), i.e. at most two partition workers.
> Fix flaky ResumeCreateIndexTest.testConcurrentBuildNewIndexAndRebuildIndexes1
> -----------------------------------------------------------------------------
>
> Key: IGNITE-29200
> URL: https://issues.apache.org/jira/browse/IGNITE-29200
> Project: Ignite
> Issue Type: Bug
> Reporter: Dmitry Werner
> Assignee: Dmitry Werner
> Priority: Minor
> Labels: MakeTeamcityGreenAgain, ise
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Success Rate: 61.3% (Last 106 Runs): 41 failed, 65 successful
> [https://ci2.ignite.apache.org/test/-5821376133685081776?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E]
> Test failed on CI: java.util.concurrent.TimeoutException: Test has been timed
> out [test=testConcurrentBuildNewIndexAndRebuildIndexes1, timeout=300000]
> *Root cause*
> The test needs two build-idx-runner threads at once: a slowed-down index
> rebuild (10 ms/row × 100k rows) and a concurrent CREATE INDEX.
> The build-index pool defaults to min(4, max(1, CPU / 4))
> (IgniteConfiguration.java:178), i.e. 1 thread on low-core CI agents.
> The rebuild starts first and occupies the only thread for ~1000 s, the CREATE
> INDEX worker stays queued, and the test hangs
> waiting for it to start until the 300 s timeout.
> *Fix*
> Test-only: set buildIndexThreadPoolSize=2 for
> testConcurrentBuildNewIndexAndRebuildIndexes0/1 (the cache has a single
> partition,
> so at most two workers). ...Indexes0 does not hang with 1 thread
> (create-index starts first and is released by the test),
> but without the pinned pool it runs the two operations sequentially instead
> of concurrently as intended.
>
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
