[
https://issues.apache.org/jira/browse/IGNITE-29136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitry Werner updated IGNITE-29136:
-----------------------------------
Description:
*Summary:*
`ViewsIntegrationTest.testSchemaCleaning` fails intermittently: SQL view is
still visible in `sys.views` right after the schema cache is destroyed.
https://ci2.ignite.apache.org/test/-4439343006317346895?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E
Stacktrace:
java.lang.AssertionError: Unexpected views count
[grid=integration.ViewsIntegrationTest2, schemaName=CACHE1, viewName=MY_VIEW]
at
org.apache.ignite.internal.processors.query.calcite.integration.ViewsIntegrationTest.assertViewsCount(ViewsIntegrationTest.java:411)
at
org.apache.ignite.internal.processors.query.calcite.integration.ViewsIntegrationTest.testSchemaCleaning(ViewsIntegrationTest.java:189)
Caused by: java.lang.AssertionError: expected:<0> but was:<1>
The failing parameterization is `[2]` = `IndexingQueryEngineConfiguration` (H2
engine); the Calcite parameterization `[1]` passed. The whole test completed in
0.451 s (node uptimes 0.15–0.40 s) on a heavily loaded agent.
*Root cause*
`testSchemaCleaning` asserts that a view created in the cache's SQL schema
disappears from `sys.views` immediately after the cache is destroyed
(ViewsIntegrationTest.java:186-189). However, view cleanup is applied on nodes
**asynchronously** relative to `destroyCache` completion:
1. `client.destroyCache("CACHE1")` finishes the synchronous destroy exchange;
on every node `SchemaManager.onCacheStopped` drops the SQL schema when its
usage count reaches zero (core: `SchemaManager.java:420-466`).
2. Removal of SQL views tied to the dropped schema is *not* part of that
synchronous path: `SchemaManager.java:460` calls
`SchemaSqlViewManager.clearSchemaViews(schemaName)`
(`SchemaSqlViewManager.java:120-137`), which runs **only on the coordinator**
and deletes the `ignite.sql.view.<schema>.*` metastorage keys via
`metastorage.removeAsync(k)` — **the returned future is ignored**
(fire-and-forget).
3. The actual local view removal on every node happens later, via a metastorage
notification round trip: `DistributedMetaStorageImpl.onGridDataReceived` →
`SchemaSqlViewManager.processMetastorageUpdate`
(`SchemaSqlViewManager.java:150-164`) → `SchemaManager.dropView` →
`H2SchemaManager.onViewDropped`.
4. The test queries `sys.views` locally on all grids without waiting, so a node
that has not yet processed the metastorage update still lists `CACHE1.MY_VIEW`
→ `expected:<0> but was:<1>`.
The same family of races is visible in the log as benign
`SchemaManager$CompoundSchemaChangeListener "Failed to notify listener (will
ignore): Failed to create view: MY_VIEW — Table "MY_TABLE" not found"` warnings
during node joins of sibling tests — metastorage view updates replaying ahead
of the local SQL schema state.
*Proposed fix*
Make `assertViewsCount` wait for convergence instead of asserting immediately:
poll all grids with `GridTestUtils.waitForCondition` (timeout 10 s) until every
grid reports the expected count, and fail with a descriptive message (grid,
schema, view, expected count) on timeout. This covers both the drop-side race
(`cnt == 0` after `destroyCache`) and the symmetric create-side race (`cnt ==
1` after `CREATE VIEW`, which is propagated through the metastorage the same
way). No product code changes; the semantic that `destroyCache` does not
guarantee metastorage view cleanup propagation is left untouched.
was:
*Summary:*
`ViewsIntegrationTest.testSchemaCleaning` fails intermittently: SQL view is
still visible in `sys.views` right after the schema cache is destroyed
java.lang.AssertionError: Unexpected views count
[grid=integration.ViewsIntegrationTest2, schemaName=CACHE1, viewName=MY_VIEW]
at
org.apache.ignite.internal.processors.query.calcite.integration.ViewsIntegrationTest.assertViewsCount(ViewsIntegrationTest.java:411)
at
org.apache.ignite.internal.processors.query.calcite.integration.ViewsIntegrationTest.testSchemaCleaning(ViewsIntegrationTest.java:189)
Caused by: java.lang.AssertionError: expected:<0> but was:<1>
The failing parameterization is `[2]` = `IndexingQueryEngineConfiguration` (H2
engine); the Calcite parameterization `[1]` passed. The whole test completed in
0.451 s (node uptimes 0.15–0.40 s) on a heavily loaded agent.
*Root cause*
`testSchemaCleaning` asserts that a view created in the cache's SQL schema
disappears from `sys.views` immediately after the cache is destroyed
(ViewsIntegrationTest.java:186-189). However, view cleanup is applied on nodes
**asynchronously** relative to `destroyCache` completion:
1. `client.destroyCache("CACHE1")` finishes the synchronous destroy exchange;
on every node `SchemaManager.onCacheStopped` drops the SQL schema when its
usage count reaches zero (core: `SchemaManager.java:420-466`).
2. Removal of SQL views tied to the dropped schema is *not* part of that
synchronous path: `SchemaManager.java:460` calls
`SchemaSqlViewManager.clearSchemaViews(schemaName)`
(`SchemaSqlViewManager.java:120-137`), which runs **only on the coordinator**
and deletes the `ignite.sql.view.<schema>.*` metastorage keys via
`metastorage.removeAsync(k)` — **the returned future is ignored**
(fire-and-forget).
3. The actual local view removal on every node happens later, via a metastorage
notification round trip: `DistributedMetaStorageImpl.onGridDataReceived` →
`SchemaSqlViewManager.processMetastorageUpdate`
(`SchemaSqlViewManager.java:150-164`) → `SchemaManager.dropView` →
`H2SchemaManager.onViewDropped`.
4. The test queries `sys.views` locally on all grids without waiting, so a node
that has not yet processed the metastorage update still lists `CACHE1.MY_VIEW`
→ `expected:<0> but was:<1>`.
The same family of races is visible in the log as benign
`SchemaManager$CompoundSchemaChangeListener "Failed to notify listener (will
ignore): Failed to create view: MY_VIEW — Table "MY_TABLE" not found"` warnings
during node joins of sibling tests — metastorage view updates replaying ahead
of the local SQL schema state.
*Proposed fix*
Make `assertViewsCount` wait for convergence instead of asserting immediately:
poll all grids with `GridTestUtils.waitForCondition` (timeout 10 s) until every
grid reports the expected count, and fail with a descriptive message (grid,
schema, view, expected count) on timeout. This covers both the drop-side race
(`cnt == 0` after `destroyCache`) and the symmetric create-side race (`cnt ==
1` after `CREATE VIEW`, which is propagated through the metastorage the same
way). No product code changes; the semantic that `destroyCache` does not
guarantee metastorage view cleanup propagation is left untouched.
> Fix flaky ViewsIntegrationTest.testSchemaCleaning
> -------------------------------------------------
>
> Key: IGNITE-29136
> URL: https://issues.apache.org/jira/browse/IGNITE-29136
> Project: Ignite
> Issue Type: Bug
> Reporter: Dmitry Werner
> Assignee: Dmitry Werner
> Priority: Minor
>
> *Summary:*
> `ViewsIntegrationTest.testSchemaCleaning` fails intermittently: SQL view is
> still visible in `sys.views` right after the schema cache is destroyed.
> https://ci2.ignite.apache.org/test/-4439343006317346895?currentProjectId=IgniteTests24Java8&branch=%3Cdefault%3E
> Stacktrace:
> java.lang.AssertionError: Unexpected views count
> [grid=integration.ViewsIntegrationTest2, schemaName=CACHE1, viewName=MY_VIEW]
> at
> org.apache.ignite.internal.processors.query.calcite.integration.ViewsIntegrationTest.assertViewsCount(ViewsIntegrationTest.java:411)
> at
> org.apache.ignite.internal.processors.query.calcite.integration.ViewsIntegrationTest.testSchemaCleaning(ViewsIntegrationTest.java:189)
> Caused by: java.lang.AssertionError: expected:<0> but was:<1>
> The failing parameterization is `[2]` = `IndexingQueryEngineConfiguration`
> (H2 engine); the Calcite parameterization `[1]` passed. The whole test
> completed in 0.451 s (node uptimes 0.15–0.40 s) on a heavily loaded agent.
> *Root cause*
> `testSchemaCleaning` asserts that a view created in the cache's SQL schema
> disappears from `sys.views` immediately after the cache is destroyed
> (ViewsIntegrationTest.java:186-189). However, view cleanup is applied on
> nodes **asynchronously** relative to `destroyCache` completion:
> 1. `client.destroyCache("CACHE1")` finishes the synchronous destroy exchange;
> on every node `SchemaManager.onCacheStopped` drops the SQL schema when its
> usage count reaches zero (core: `SchemaManager.java:420-466`).
> 2. Removal of SQL views tied to the dropped schema is *not* part of that
> synchronous path: `SchemaManager.java:460` calls
> `SchemaSqlViewManager.clearSchemaViews(schemaName)`
> (`SchemaSqlViewManager.java:120-137`), which runs **only on the coordinator**
> and deletes the `ignite.sql.view.<schema>.*` metastorage keys via
> `metastorage.removeAsync(k)` — **the returned future is ignored**
> (fire-and-forget).
> 3. The actual local view removal on every node happens later, via a
> metastorage notification round trip:
> `DistributedMetaStorageImpl.onGridDataReceived` →
> `SchemaSqlViewManager.processMetastorageUpdate`
> (`SchemaSqlViewManager.java:150-164`) → `SchemaManager.dropView` →
> `H2SchemaManager.onViewDropped`.
> 4. The test queries `sys.views` locally on all grids without waiting, so a
> node that has not yet processed the metastorage update still lists
> `CACHE1.MY_VIEW` → `expected:<0> but was:<1>`.
> The same family of races is visible in the log as benign
> `SchemaManager$CompoundSchemaChangeListener "Failed to notify listener (will
> ignore): Failed to create view: MY_VIEW — Table "MY_TABLE" not found"`
> warnings during node joins of sibling tests — metastorage view updates
> replaying ahead of the local SQL schema state.
> *Proposed fix*
> Make `assertViewsCount` wait for convergence instead of asserting
> immediately: poll all grids with `GridTestUtils.waitForCondition` (timeout 10
> s) until every grid reports the expected count, and fail with a descriptive
> message (grid, schema, view, expected count) on timeout. This covers both the
> drop-side race (`cnt == 0` after `destroyCache`) and the symmetric
> create-side race (`cnt == 1` after `CREATE VIEW`, which is propagated through
> the metastorage the same way). No product code changes; the semantic that
> `destroyCache` does not guarantee metastorage view cleanup propagation is
> left untouched.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
