bobhan1 opened a new pull request, #70760:
URL: https://github.com/apache/doris/pull/70760

   ### Background
   
   Cloud merge-on-write loads acquire a table-level lock in Meta Service before 
publishing delete bitmaps. Compaction and schema change also acquire locks 
through Meta Service. Cloud clusters currently select the lock protocol by 
instance using the Meta Service allowlist.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #50173 (introduced the instance allowlist)
   PR Type: Enhancement
   
   Problem Summary:
   
   The allowlist makes enabling V2 an instance-wide operation. Operators have 
to coordinate a protocol switch across clients and Meta Service while existing 
tables are active. Store the protocol choice when a table is created so the 
cluster can use V2 for new tables while serving existing tables with their 
current mode.
   
   ### Details
   
   - **Change map:** The diff has 200 production lines and 228 test lines, 
including nine lines of generated SQL results. Table creation and tablet 
metadata carry the immutable setting. Lock RPCs and existing tablet job records 
carry the setting to Meta Service. The tests cover selection, cleanup, and SQL 
behavior.
   - **Before and after:** Meta Service previously selected V1 or V2 from its 
instance configuration. It now uses the table setting when supplied. FE sends 
it for load locks and BE sends it for compaction, schema change, and locked 
bitmap writes.
   - **Configs and defaults:** Cloud MoW `CREATE TABLE` defaults 
`enable_delete_bitmap_lock_v2` to `true`. Set `"enable_delete_bitmap_lock_v2" = 
"false"` at creation to use V1. `ALTER TABLE` rejects changes to the property. 
The existing `delete_bitmap_lock_v2_white_list` is the fallback for tables 
without the setting.
   - **Impact on existing users:** Upgraded tables with no stored setting keep 
their allowlist-selected mode. New partitions, shadow tablets, and restored 
tablets use the destination table's setting. Roll out support to FE, BE, Meta 
Service and Recycler before creating tables that use the property.
   - **Behavior change:** New cloud MoW tables use V2 by default.
   
   ### Testing
   
   - Local ASAN Cloud UT passed the two new `*TableDeleteBitmapLockVersion*` 
tests for explicit V1 and V2, legacy fallback, locked update and release, and 
V2 lock cleanup through the recorded job.
   - Six existing `*DeleteBitmapUpdateLock*` Cloud UTs passed locally with ASAN.
   - 
`CloudInternalCatalogTest.testCreateTabletsCarriesTableDeleteBitmapLockVersion` 
passed locally for true, false, and absent properties in the create-tablets RPC.
   - `test_table_delete_bitmap_lock_version` passed on a local cloud cluster 
with FoundationDB and S3. It covers the new default, an explicit V1 table, 
invalid DDL, and MoW overwrites in both tables.
   - Manual full compaction on each table merged [2-2] and [3-3]. Meta Service 
used the table's protocol for lock acquisition and job commit, and both queries 
returned the expected rows after compaction.
   
   ### Notes for Reviewers
   
   - The mode comes from existing tablet metadata on BE. Meta Service selects 
the protocol from the RPC and resolves it from the existing job record when 
finishing a job. Snapshot restore takes the mode from the destination tablet 
because the source header may come from a table with a different setting.
   
   ### Release note
   
   New cloud Merge-on-Write tables use delete bitmap lock V2 by default. To use 
V1 for a new table, set `enable_delete_bitmap_lock_v2` to `false` in the CREATE 
TABLE properties. Tables without the property continue to use the Meta Service 
instance allowlist.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: Yes (document the property and cloud upgrade 
behavior in the CREATE TABLE reference)
   - Pre-submission status: Ready
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6)
   - Generated: Implementation, tests, and PR description
   - Human review: Pending at submission
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to