Mryange opened a new pull request, #70591: URL: https://github.com/apache/doris/pull/70591
### Background Historical ANN index builds buffer vectors and construct an in-memory index for each segment. IndexBuilder also owns the per-segment column and file writers until their index files are sealed. ### What problem does this PR solve? Issue Number: None Related PR: #70394 (extends the build-resource cleanup with per-segment file sealing) PR Type: Bugfix Problem Summary: Building ANN indexes on a rowset with many segments retains completed build objects and serialized index buffers, increasing peak memory as more segments finish. Release the ANN build resources after each segment and seal each segment's independent index file before processing the next segment. Preserve the existing rowset publication boundary and propagate write/close failures. ### Details - **Frequency:** Multi-segment historical index builds; the serialized-buffer accumulation is visible when index RAM directories are enabled. - **Root cause:** Completed ANN objects/input buffers and per-segment index directories outlive their useful build phase because cleanup and file sealing occur at rowset scope. - **Why existing tests missed it:** Index correctness checks do not assert peak memory or the lifetime of completed segment build resources. - **Fix:** Release build-only resources on success, skip and error paths, free input-buffer capacity before serialization, and close/release segment writers after all requested logical indexes for that segment finish. Segments with no columns to rebuild still seal inherited indexes. - **Other paths and branches:** Apply segment sealing to both regular and SNII IndexBuilder paths. The common close placement also applies to inverted-index rebuilds; dropping indexes and rowset publication are unchanged. No format, protocol or configuration changes. - **Behavior change:** Completed segment resources are reclaimed earlier; a rowset is still published only after the whole operation succeeds. ### Testing - Final patch: `sh build.sh --be -j48` (ASAN) completed successfully on the development host. - Official `build-support/clang-format.sh`, `build-support/check-format.sh`, and `build-support/check-build-hygiene.sh`: passed; formatting made no source changes. - Official `build-support/run-clang-tidy.sh --base 08231e3a635` with a per-file compilation database derived from the ASAN unity build: passed for all three changed files in changed-line filtering mode. This does not claim that entire files or included headers have no pre-existing diagnostics. - `git diff --check`: passed. Earlier runtime measurements of this resource-release/per-segment-sealing implementation, before the final no-columns control-flow adjustment: - Same 500,000 float32 vectors, dimension 128, one rowset with 14 segments; HNSW, max_degree=16, ef_construction=100, L2, flat quantization. RELEASE build, local disk, 32 MiB write buffer, automatic compaction disabled, index RAM directories enabled. - Process-wide peak increases relative to the pre-build baseline (MiB): | Implementation | RSS peak increase | jemalloc allocated peak increase | | --- | ---: | ---: | | Baseline master 08231e3a635 | 695.4 | 701.8 | | Release ANN build resources only | 403.9 | 366.2 | | Release resources + seal each segment | 90.7 | 73.9 | - The combined implementation reduced the observed RSS peak increase by about 87% and allocated peak increase by about 89.5% versus baseline. These are single-run, process-wide observations on a shared development host; warm state differed and the sampling is not an exact accounting of ANN-only allocations. No throughput claim is made. - Output remained 500,000 rows / 14 segments with identical per-segment row counts, 220,383,392 data bytes and 328,141,266 index bytes across the Doris versions. Six indexed self-match queries spanning segment boundaries returned the original IDs at distance zero; EXPLAIN showed ANN SORT INFO. This is basic readability validation, not a recall benchmark. - A separate targeted syscall trace with sync_file_on_close=true observed 14 index-file fdatasync calls totaling 109.5 ms, 14 directory-sync calls totaling 2.3 ms and 14 file-close calls totaling 0.42 ms. These exclude serialization, buffer copies and resource destruction; tracing adds overhead and this does not establish an end-to-end performance regression bound. - No new unit tests are included. The final no-columns branch adjustment has not been rerun in a live cluster. ### Release note Reduce peak memory usage when building ANN indexes on historical rowsets with multiple segments by releasing completed segment resources earlier. ### Check List (For Author) - Behavior changed: Yes (see Details). - Does this need documentation: No. - Pre-submission status: Partially verified: build/static checks and runtime limits are listed above; runtime memory/readability measurements precede the final no-columns branch adjustment, and broader inverted-index rebuild regression tests were not run. ### AI Disclosure - Tool: Codex (GPT-6). - Generated: Code changes and this description. - Human review: The user reviewed the proposed lifecycle approach; final revision review is pending. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
