yuqi1129 opened a new pull request, #13861:
URL: https://github.com/apache/gravitino/pull/13861
### What changes were proposed in this pull request?
- `OccWriteSupport#deleteChildrenWithVersions` runs the versioned batch soft
delete in chunks of 500 children. All chunks stay in the caller's transaction,
so a version miss in any chunk still rolls back the whole cascade.
- `softDeleteSchemaMetasWithVersion` and `softDeleteCatalogMetasWithVersion`
(base and PostgreSQL) add `schema_id IN (...)` / `catalog_id IN (...)` before
the version OR chain, so each chunk finds its rows by primary key.
### Why are the changes needed?
A cascading catalog or metalake drop deleted all schemas with one OR chain.
MySQL planned it superlinearly, H2 scanned the whole table for it, and
PostgreSQL rejected it above 32,767 schemas because of its 65,535 bind
parameter limit.
Cascade drop of one catalog, time in ms:
| Schemas | H2 before | H2 after | MySQL before | MySQL after | PG before |
PG after | PG after, with `ANALYZE` |
|--------:|----------:|---------:|-------------:|------------:|----------:|---------:|-------------------------:|
| 2,000 | 636 | 389 | 246 | 162 | 77 |
73 | 72 |
| 17,000 | 4,475 | 956 | 22,649 | 766 | 1,117 |
1,239 | 399 |
| 40,000 | 20,909 | 2,371 | 117,160 | 1,806 | failed |
8,648 | 971 |
The schemas are bulk-inserted right before the drop. Until PostgreSQL
analyzes the table, it estimates the table as nearly empty and scans a
secondary index instead of the primary key. That accounts for the "PG after"
column; with fresh statistics it uses the primary key.
Fix: #13860
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- `TestOccWriteSupport`: the children are split into chunks, and a short
chunk stops the delete with `OptimisticLockException`.
-
`TestCatalogMetaService#testCascadeDeleteCatalogWithMoreSchemasThanOneChunk`: a
stale version in the second chunk fails the cascade and rolls back the first
chunk; a retry then drops everything. Runs on H2, MySQL and PostgreSQL.
- `TestCatalogMetaService`, `TestMetalakeMetaService`,
`TestSchemaMetaService` and `TestOccWriteSupport` pass on H2, MySQL and
PostgreSQL (`-PskipDockerTests=false`).
- The timings above come from a local stress harness that is not part of
this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
