Userwhite opened a new pull request, #63643: URL: https://github.com/apache/doris/pull/63643
### What problem does this PR solve? Issue Number: close https://github.com/apache/doris/issues/61956 #### Support Row Binlog Clone and Compaction ##### Motivation This PR adds BE storage support: 1. row binlog clone ( fix missing version problem) 2. row binlog compaction. (prevent too many small files) ##### Clone clone binlog files like ccr ##### Compaction ``` // Binlog compaction selection rules (tiered, L0..LMax) // // Score / Permits // - For LMax, treat Base([0-x]) as score/permit=1, others use RowsetMeta::get_compaction_score(). // // Trigger (all levels): merge when ANY holds // - size >= binlog_compaction_goal_size_mbytes * 1MB // - score >= binlog_compaction_file_count_threshold // - time >= binlog_compaction_time_threshold_seconds // // LMax "Base + `ENOUGH` + remaining" model (oldest -> newest) // | Base([0-x]) | `ENOUGH` rowsets | remaining rowsets ... | // `ENOUGH` is computed dynamically on LMax (not persisted): // (rowset_size >= goal_size) OR (rowset_score >= file_count_threshold) // // Input Rowsets selection: // - If physical rewrite trigger is NOT met: try quick compact first (requires Base([0-x])). // - If both quick compact and physical rewrite are possible: compare score and pick the higher. // // Quick compact output must be OVERLAPPING. ``` -------- ## Summary This PR makes row binlog a first-class storage object in BE. It adds: 1. ROW_BINLOG copy type for clone/snapshot. 2. Snapshot support for row binlog rowsets and binlog_delvec. 4. Clone support for downloading and linking row binlog files. 5. Tablet meta support for managing row_binlog_rs_metas. 6. Dedicated tiered compaction for row binlog rowsets. The result is: • row binlog can be correctly cloned between replicas • MOW row binlog delete information is preserved • row binlog file/meta count can be reduced by compaction ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [x] Regression test - [x] Unit Test - [x] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [x] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [] No. - [] Yes. <!-- Add document PR link here. eg: https://github.com/apache/doris-website/pull/1214 --> ### Check List (For Reviewer who merge this PR) - [x] Confirm the release note - [x] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
