Mryange opened a new pull request, #70591:
URL: https://github.com/apache/doris/pull/70591

   ### Background
   
   Historical ANN index builds buffer vectors and construct an in-memory index 
for each segment. IndexBuilder also owns the per-segment column and file 
writers until their index files are sealed.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #70394 (extends the build-resource cleanup with per-segment file 
sealing)
   PR Type: Bugfix
   
   Problem Summary:
   
   Building ANN indexes on a rowset with many segments retains completed build 
objects and serialized index buffers, increasing peak memory as more segments 
finish. Release the ANN build resources after each segment and seal each 
segment's independent index file before processing the next segment. Preserve 
the existing rowset publication boundary and propagate write/close failures.
   
   ### Details
   
   - **Frequency:** Multi-segment historical index builds; the 
serialized-buffer accumulation is visible when index RAM directories are 
enabled.
   - **Root cause:** Completed ANN objects/input buffers and per-segment index 
directories outlive their useful build phase because cleanup and file sealing 
occur at rowset scope.
   - **Why existing tests missed it:** Index correctness checks do not assert 
peak memory or the lifetime of completed segment build resources.
   - **Fix:** Release build-only resources on success, skip and error paths, 
free input-buffer capacity before serialization, and close/release segment 
writers after all requested logical indexes for that segment finish. Segments 
with no columns to rebuild still seal inherited indexes.
   - **Other paths and branches:** Apply segment sealing to both regular and 
SNII IndexBuilder paths. The common close placement also applies to 
inverted-index rebuilds; dropping indexes and rowset publication are unchanged. 
No format, protocol or configuration changes.
   - **Behavior change:** Completed segment resources are reclaimed earlier; a 
rowset is still published only after the whole operation succeeds.
   
   ### Testing
   
   - Final patch: `sh build.sh --be -j48` (ASAN) completed successfully on the 
development host.
   - Official `build-support/clang-format.sh`, `build-support/check-format.sh`, 
and `build-support/check-build-hygiene.sh`: passed; formatting made no source 
changes.
   - Official `build-support/run-clang-tidy.sh --base 08231e3a635` with a 
per-file compilation database derived from the ASAN unity build: passed for all 
three changed files in changed-line filtering mode. This does not claim that 
entire files or included headers have no pre-existing diagnostics.
   - `git diff --check`: passed.
   
   Earlier runtime measurements of this resource-release/per-segment-sealing 
implementation, before the final no-columns control-flow adjustment:
   
   - Same 500,000 float32 vectors, dimension 128, one rowset with 14 segments; 
HNSW, max_degree=16, ef_construction=100, L2, flat quantization. RELEASE build, 
local disk, 32 MiB write buffer, automatic compaction disabled, index RAM 
directories enabled.
   - Process-wide peak increases relative to the pre-build baseline (MiB):
   
   | Implementation | RSS peak increase | jemalloc allocated peak increase |
   | --- | ---: | ---: |
   | Baseline master 08231e3a635 | 695.4 | 701.8 |
   | Release ANN build resources only | 403.9 | 366.2 |
   | Release resources + seal each segment | 90.7 | 73.9 |
   
   - The combined implementation reduced the observed RSS peak increase by 
about 87% and allocated peak increase by about 89.5% versus baseline. These are 
single-run, process-wide observations on a shared development host; warm state 
differed and the sampling is not an exact accounting of ANN-only allocations. 
No throughput claim is made.
   - Output remained 500,000 rows / 14 segments with identical per-segment row 
counts, 220,383,392 data bytes and 328,141,266 index bytes across the Doris 
versions. Six indexed self-match queries spanning segment boundaries returned 
the original IDs at distance zero; EXPLAIN showed ANN SORT INFO. This is basic 
readability validation, not a recall benchmark.
   - A separate targeted syscall trace with sync_file_on_close=true observed 14 
index-file fdatasync calls totaling 109.5 ms, 14 directory-sync calls totaling 
2.3 ms and 14 file-close calls totaling 0.42 ms. These exclude serialization, 
buffer copies and resource destruction; tracing adds overhead and this does not 
establish an end-to-end performance regression bound.
   - No new unit tests are included. The final no-columns branch adjustment has 
not been rerun in a live cluster.
   
   ### Release note
   
   Reduce peak memory usage when building ANN indexes on historical rowsets 
with multiple segments by releasing completed segment resources earlier.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details).
   - Does this need documentation: No.
   - Pre-submission status: Partially verified: build/static checks and runtime 
limits are listed above; runtime memory/readability measurements precede the 
final no-columns branch adjustment, and broader inverted-index rebuild 
regression tests were not run.
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6).
   - Generated: Code changes and this description.
   - Human review: The user reviewed the proposed lifecycle approach; final 
revision review is pending.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to