[
https://issues.apache.org/jira/browse/IGNITE-24172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kirill Tkalenko updated IGNITE-24172:
-------------------------------------
Description:
Flaky
*org.apache.ignite.internal.table.distributed.index.IndexMetaStorageRecoveryTest#testMissingDropTableWithUpdateLwmBeforeRestart*
was detected, we need to investigate why it failed, the fail was detected on
the
[TC|https://ci.ignite.apache.org/viewLog.html?buildId=8737030&buildTypeId=ApacheIgnite3xGradle_Test_RunAllTests&fromSakuraUI=true],
it is reproduced locally quite rarely.
At the moment, after restarting the components, the index meta removes from the
local cache but remains in the metastorage. We need to find out why this is
happening.
The problem is that we use a test implementation of
*org.apache.ignite.internal.metastorage.impl.StandaloneMetaStorageManager*,
which internally uses mocked components, in particular
*org.apache.ignite.internal.raft.client.TopologyAwareRaftGroupService*, which
for some reason reordered the execution of raft commands. When adding logging,
it turned out that after restarting the components to restore the index meta,
we send an update command and then a key delete command, but in the metastorage
state machine they were executed in the reverse order, so there is no meta in
the local volatile cache, but there is in the metastorage.
We need to protect ourselves from such a situation by supplementing the
conditions for updating keys in the metastorage.
{noformat}
org.opentest4j.AssertionFailedError: expected: <null> but was: <IndexMeta
[catalogVersion=3, indexId=4, tableId=3, tableVersionOnIndexCreation=1,
indexName=TEST_TABLE_PK, currentStatus=READ_ONLY, statusChanges=UnmodifiableMap
{AVAILABLE=MetaIndexStatusChange [catalogVersion=2,
activationTs=113723603341803522], READ_ONLY=MetaIndexStatusChange
[catalogVersion=3, activationTs=113723603343048704]}]>
at
app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at
app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at app//org.junit.jupiter.api.AssertNull.failNotNull(AssertNull.java:50)
at app//org.junit.jupiter.api.AssertNull.assertNull(AssertNull.java:35)
at app//org.junit.jupiter.api.AssertNull.assertNull(AssertNull.java:30)
at app//org.junit.jupiter.api.Assertions.assertNull(Assertions.java:279)
at
app//org.apache.ignite.internal.table.distributed.index.IndexMetaStorageRecoveryTest.testMissingDropTableWithUpdateLwmBeforeRestart(IndexMetaStorageRecoveryTest.java:458)
at java.base@17.0.6/java.lang.reflect.Method.invoke(Method.java:568)
at java.base@17.0.6/java.util.ArrayList.forEach(ArrayList.java:1511)
at java.base@17.0.6/java.util.ArrayList.forEach(ArrayList.java:1511)
{noformat}
was:
Flaky
*org.apache.ignite.internal.table.distributed.index.IndexMetaStorageRecoveryTest#testMissingDropTableWithUpdateLwmBeforeRestart*
was detected, we need to investigate why it failed, the fail was detected on
the
[TC|https://ci.ignite.apache.org/viewLog.html?buildId=8737030&buildTypeId=ApacheIgnite3xGradle_Test_RunAllTests&fromSakuraUI=true],
it is reproduced locally quite rarely.
At the moment, after restarting the components, the index meta removes from the
local cache but remains in the metastorage. We need to find out why this is
happening.
{noformat}
org.opentest4j.AssertionFailedError: expected: <null> but was: <IndexMeta
[catalogVersion=3, indexId=4, tableId=3, tableVersionOnIndexCreation=1,
indexName=TEST_TABLE_PK, currentStatus=READ_ONLY, statusChanges=UnmodifiableMap
{AVAILABLE=MetaIndexStatusChange [catalogVersion=2,
activationTs=113723603341803522], READ_ONLY=MetaIndexStatusChange
[catalogVersion=3, activationTs=113723603343048704]}]>
at
app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
at
app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
at app//org.junit.jupiter.api.AssertNull.failNotNull(AssertNull.java:50)
at app//org.junit.jupiter.api.AssertNull.assertNull(AssertNull.java:35)
at app//org.junit.jupiter.api.AssertNull.assertNull(AssertNull.java:30)
at app//org.junit.jupiter.api.Assertions.assertNull(Assertions.java:279)
at
app//org.apache.ignite.internal.table.distributed.index.IndexMetaStorageRecoveryTest.testMissingDropTableWithUpdateLwmBeforeRestart(IndexMetaStorageRecoveryTest.java:458)
at java.base@17.0.6/java.lang.reflect.Method.invoke(Method.java:568)
at java.base@17.0.6/java.util.ArrayList.forEach(ArrayList.java:1511)
at java.base@17.0.6/java.util.ArrayList.forEach(ArrayList.java:1511)
{noformat}
> Fix flaky
> IndexMetaStorageRecoveryTest#testMissingDropTableWithUpdateLwmBeforeRestart
> -------------------------------------------------------------------------------------
>
> Key: IGNITE-24172
> URL: https://issues.apache.org/jira/browse/IGNITE-24172
> Project: Ignite
> Issue Type: Bug
> Reporter: Kirill Tkalenko
> Assignee: Kirill Tkalenko
> Priority: Major
> Labels: ignite-3
>
> Flaky
> *org.apache.ignite.internal.table.distributed.index.IndexMetaStorageRecoveryTest#testMissingDropTableWithUpdateLwmBeforeRestart*
> was detected, we need to investigate why it failed, the fail was detected on
> the
> [TC|https://ci.ignite.apache.org/viewLog.html?buildId=8737030&buildTypeId=ApacheIgnite3xGradle_Test_RunAllTests&fromSakuraUI=true],
> it is reproduced locally quite rarely.
> At the moment, after restarting the components, the index meta removes from
> the local cache but remains in the metastorage. We need to find out why this
> is happening.
> The problem is that we use a test implementation of
> *org.apache.ignite.internal.metastorage.impl.StandaloneMetaStorageManager*,
> which internally uses mocked components, in particular
> *org.apache.ignite.internal.raft.client.TopologyAwareRaftGroupService*, which
> for some reason reordered the execution of raft commands. When adding
> logging, it turned out that after restarting the components to restore the
> index meta, we send an update command and then a key delete command, but in
> the metastorage state machine they were executed in the reverse order, so
> there is no meta in the local volatile cache, but there is in the metastorage.
> We need to protect ourselves from such a situation by supplementing the
> conditions for updating keys in the metastorage.
> {noformat}
> org.opentest4j.AssertionFailedError: expected: <null> but was: <IndexMeta
> [catalogVersion=3, indexId=4, tableId=3, tableVersionOnIndexCreation=1,
> indexName=TEST_TABLE_PK, currentStatus=READ_ONLY,
> statusChanges=UnmodifiableMap {AVAILABLE=MetaIndexStatusChange
> [catalogVersion=2, activationTs=113723603341803522],
> READ_ONLY=MetaIndexStatusChange [catalogVersion=3,
> activationTs=113723603343048704]}]>
> at
> app//org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
> at
> app//org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
> at app//org.junit.jupiter.api.AssertNull.failNotNull(AssertNull.java:50)
> at app//org.junit.jupiter.api.AssertNull.assertNull(AssertNull.java:35)
> at app//org.junit.jupiter.api.AssertNull.assertNull(AssertNull.java:30)
> at app//org.junit.jupiter.api.Assertions.assertNull(Assertions.java:279)
> at
> app//org.apache.ignite.internal.table.distributed.index.IndexMetaStorageRecoveryTest.testMissingDropTableWithUpdateLwmBeforeRestart(IndexMetaStorageRecoveryTest.java:458)
> at java.base@17.0.6/java.lang.reflect.Method.invoke(Method.java:568)
> at java.base@17.0.6/java.util.ArrayList.forEach(ArrayList.java:1511)
> at java.base@17.0.6/java.util.ArrayList.forEach(ArrayList.java:1511)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
