bobhan1 opened a new pull request, #70478: URL: https://github.com/apache/doris/pull/70478
### Background Cloud Merge-on-Write persists a logical delete bitmap as a blob split into 90 KB key-value pairs. Bitmap writes complete before the corresponding load version or compaction/schema-change output rowset is published. ### What problem does this PR solve? Issue Number: None Related PR: #46969 (transaction-size budgeting), #47097 (existing large-bitmap tests) PR Type: Bugfix Problem Summary: A single bitmap or v2 bitmap-storage value larger than FoundationDB's transaction limit still fails even when the RPC already batches different bitmaps: `blob_put` puts all fragments of one value into the same transaction. Batch fragments of oversized locked writes while retaining the existing blob encoding and publish boundary. Recheck lock ownership, publish version, and pending-writer identity after each intermediate commit. Oversized unlocked aggregation preserves the existing visible bitmaps by skipping that optional optimization. ### Details - **Change map:** 121 changed production/schema lines, 459 lines of tests, and one test-target registration; no generated files. Most of the diff is the regression fixture and failure/retry scenarios. - **Frequency:** Triggered by a single large v1 bitmap or serialized v2 storage value, including a 10,578,008-byte value exceeding the 10,000,000-byte FDB transaction limit. - **Root cause:** The previous budget check runs between logical values; splitting a value into KV fragments does not split its transaction. - **Why existing tests missed it:** Existing large-bitmap tests use MemTxnKv without enforcing FDB transaction-size limits, so even an oversized logical value can pass. - **Fix:** Write oversized locked blobs in bounded batches using the existing fragment keys. Cap the effective soft budget at 9,000,000 bytes to retain headroom if the configured budget is raised. Persist pending cleanup/intent first and revalidate the lock, partition version, and optional pending writer ID in renewed transactions. Preserve the writer ID when explicit sub-transactions append pending keys. - **Other paths and branches:** Cover v1, v2, dual writes, load, compaction, and schema change. Unlocked aggregation with a logical value of at least 10,000,000 bytes returns a successful no-op before any writes or cleanup, retaining its previous representation. The optional PendingDeleteBitmapPB field 2 is unused on branch-3.1, branch-4.0, branch-4.1, and branch-4.2; this PR allocates it on master. - **Behavior change:** Oversized locked bitmap writes can span FDB transactions without changing the Doris transaction boundary. Skipped oversized aggregation can retain more historical bitmap metadata until later cleanup. - **Compatibility:** Existing bitmap keys and encoded values remain unchanged. The new pending-metadata writer ID is optional, and legacy pending metadata remains readable. ### Testing - Six initial regression cases failed with transaction-size errors before the implementation change. - ASAN: `./run-cloud-ut.sh --run --filter='meta_service_test:*DeleteBitmap*' -j100`: **39 tests passed**, including 12 new cases. The size-limited memory store rejects oversized transactions and KV values. Coverage includes v1/v2/dual readback, raised soft budgets, job writes, ambiguous commit retry, lock loss with shorter retry/tail cleanup, partition-version changes, overlapping RPCs, explicit sub-transactions, and preserving visible bitmaps during skipped aggregation. - Production Cloud ASAN build passed via `./build.sh --cloud -j100`, with `MAKE_TEST=OFF`, injection points disabled, and Java extensions disabled. - `build-support/check-format.sh` and `git diff --check` passed. - clang-tidy was stopped and not completed. No real-FDB cluster or FE/BE end-to-end load was run. ### Release note Cloud Merge-on-Write can persist a single delete bitmap larger than the FoundationDB transaction limit by batching its fragments. Oversized optional unlocked bitmap aggregation retains the previous bitmaps. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. - Pre-submission status: Partially verified: ASAN unit tests and production compilation passed; real-FDB and cluster validation remain. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Implementation, tests, and this description. - Human review: Pending author/maintainer review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
