This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
     new 547dc0f0e98 [fix](storage) Release ANN resources and seal rebuilt 
indexes per segment (#70591)
547dc0f0e98 is described below

commit 547dc0f0e984775790c1c1b3279171500a7492ef
Author: Mryange <[email protected]>
AuthorDate: Thu Nov 19 16:49:57 2026 +0800

    [fix](storage) Release ANN resources and seal rebuilt indexes per segment 
(#70591)
    
    ### Background
    
    Historical ANN index builds buffer vectors and construct an in-memory
    index for each segment. IndexBuilder also owns the per-segment column
    and file writers until their index files are sealed.
    
    ### What problem does this PR solve?
    
    Issue Number: None
    Related PR: #70394 (extends the build-resource cleanup with per-segment
    file sealing)
    PR Type: Bugfix
    
    Problem Summary:
    
    Building ANN indexes on a rowset with many segments retains completed
    build objects and serialized index buffers, increasing peak memory as
    more segments finish. Release the ANN build resources after each segment
    and seal each segment's independent index file before processing the
    next segment. Preserve the existing rowset publication boundary and
    propagate write/close failures.
    
    ### Details
    
    - **Frequency:** Multi-segment historical index builds; the
    serialized-buffer accumulation is visible when index RAM directories are
    enabled.
    - **Root cause:** Completed ANN objects/input buffers and per-segment
    index directories outlive their useful build phase because cleanup and
    file sealing occur at rowset scope.
    - **Why existing tests missed it:** Index correctness checks do not
    assert peak memory or the lifetime of completed segment build resources.
    - **Fix:** Release build-only resources on success, skip and error
    paths, free input-buffer capacity before serialization, and
    close/release segment writers after all requested logical indexes for
    that segment finish. Segments with no columns to rebuild still seal
    inherited indexes.
    - **Other paths and branches:** Apply segment sealing to both regular
    and SNII IndexBuilder paths. The common close placement also applies to
    inverted-index rebuilds; dropping indexes and rowset publication are
    unchanged. No format, protocol or configuration changes.
    - **Behavior change:** Completed segment resources are reclaimed
    earlier; a rowset is still published only after the whole operation
    succeeds.
    
    ### Testing
    
    - Final patch: `sh build.sh --be -j48` (ASAN) completed successfully on
    the development host.
    - Official `build-support/clang-format.sh`,
    `build-support/check-format.sh`, and
    `build-support/check-build-hygiene.sh`: passed; formatting made no
    source changes.
    - Official `build-support/run-clang-tidy.sh --base 08231e3a635` with a
    per-file compilation database derived from the ASAN unity build: passed
    for all three changed files in changed-line filtering mode. This does
    not claim that entire files or included headers have no pre-existing
    diagnostics.
    - `git diff --check`: passed.
    
    Earlier runtime measurements of this
    resource-release/per-segment-sealing implementation, before the final
    no-columns control-flow adjustment:
    
    - Same 500,000 float32 vectors, dimension 128, one rowset with 14
    segments; HNSW, max_degree=16, ef_construction=100, L2, flat
    quantization. RELEASE build, local disk, 32 MiB write buffer, automatic
    compaction disabled, index RAM directories enabled.
    - Process-wide peak increases relative to the pre-build baseline (MiB):
    
    | Implementation | RSS peak increase | jemalloc allocated peak increase
    |
    | --- | ---: | ---: |
    | Baseline master 08231e3a635 | 695.4 | 701.8 |
    | Release ANN build resources only | 403.9 | 366.2 |
    | Release resources + seal each segment | 90.7 | 73.9 |
    
    - The combined implementation reduced the observed RSS peak increase by
    about 87% and allocated peak increase by about 89.5% versus baseline.
    These are single-run, process-wide observations on a shared development
    host; warm state differed and the sampling is not an exact accounting of
    ANN-only allocations. No throughput claim is made.
    - Output remained 500,000 rows / 14 segments with identical per-segment
    row counts, 220,383,392 data bytes and 328,141,266 index bytes across
    the Doris versions. Six indexed self-match queries spanning segment
    boundaries returned the original IDs at distance zero; EXPLAIN showed
    ANN SORT INFO. This is basic readability validation, not a recall
    benchmark.
    - A separate targeted syscall trace with sync_file_on_close=true
    observed 14 index-file fdatasync calls totaling 109.5 ms, 14
    directory-sync calls totaling 2.3 ms and 14 file-close calls totaling
    0.42 ms. These exclude serialization, buffer copies and resource
    destruction; tracing adds overhead and this does not establish an
    end-to-end performance regression bound.
    - No new unit tests are included. The final no-columns branch adjustment
    has not been rerun in a live cluster.
    
    ### Release note
    
    Reduce peak memory usage when building ANN indexes on historical rowsets
    with multiple segments by releasing completed segment resources earlier.
    
    ### Check List (For Author)
    
    - Behavior changed: Yes (see Details).
    - Does this need documentation: No.
    - Pre-submission status: Partially verified: build/static checks and
    runtime limits are listed above; runtime memory/readability measurements
    precede the final no-columns branch adjustment, and broader
    inverted-index rebuild regression tests were not run.
    
    ### AI Disclosure
    
    - Tool: Codex (GPT-6).
    - Generated: Code changes and this description.
    - Human review: The user reviewed the proposed lifecycle approach; final
    revision review is pending.
---
 be/src/storage/index/ann/ann_index_writer.cpp |  20 ++-
 be/src/storage/index/ann/ann_index_writer.h   |   5 +-
 be/src/storage/task/index_builder.cpp         | 187 +++++++++++++-------------
 3 files changed, 114 insertions(+), 98 deletions(-)

diff --git a/be/src/storage/index/ann/ann_index_writer.cpp 
b/be/src/storage/index/ann/ann_index_writer.cpp
index 686cf15208c..a56c6a97d12 100644
--- a/be/src/storage/index/ann/ann_index_writer.cpp
+++ b/be/src/storage/index/ann/ann_index_writer.cpp
@@ -26,6 +26,7 @@
 #include "common/config.h"
 #include "storage/index/ann/faiss_ann_index.h"
 #include "storage/index/index_fs_directory.h"
+#include "util/defer_op.h"
 
 namespace doris::segment_v2 {
 static std::string get_or_default(const std::map<std::string, std::string>& 
properties,
@@ -91,6 +92,11 @@ Status AnnIndexColumnWriter::add_values(const uint8_t* 
values, size_t count) {
 }
 
 void AnnIndexColumnWriter::close_on_error() {
+    _release_build_resources();
+}
+
+void AnnIndexColumnWriter::_release_build_resources() {
+    _vector_index.reset();
     PODArray<float> empty_buffered_vectors;
     _buffered_vectors.swap(empty_buffered_vectors);
 }
@@ -138,6 +144,7 @@ int64_t AnnIndexColumnWriter::size() const {
 Status AnnIndexColumnWriter::finish() {
     if (_total_rows == 0) {
         LOG_INFO("No data to train/add for ANN index. Skipping index 
building.");
+        _release_build_resources();
         return _index_file_writer->delete_index(_index_meta);
     }
 
@@ -149,8 +156,7 @@ Status AnnIndexColumnWriter::finish() {
                 "Total data size {} is less than minimum {} rows required for 
ANN index build. "
                 "Skipping index building for this segment.",
                 _total_rows, effective_min_rows);
-        PODArray<float> empty_buffered_vectors;
-        _buffered_vectors.swap(empty_buffered_vectors);
+        _release_build_resources();
         return _index_file_writer->delete_index(_index_meta);
     }
 
@@ -158,6 +164,10 @@ Status AnnIndexColumnWriter::finish() {
 }
 
 Status AnnIndexColumnWriter::_build_and_save(Int64 min_train_rows, Int64 
effective_min_rows) {
+    // Release the Faiss index and any remaining input buffer after saving, 
including
+    // on failure, rather than retaining them until this writer is destroyed.
+    Defer release_resources([this] { _release_build_resources(); });
+
     const size_t dim = _vector_index->get_dimension();
     DCHECK(_buffered_vectors.size() % dim == 0);
     const Int64 train_rows = cast_set<Int64>(_buffered_vectors.size() / dim);
@@ -169,8 +179,10 @@ Status AnnIndexColumnWriter::_build_and_save(Int64 
min_train_rows, Int64 effecti
     RETURN_IF_ERROR(_vector_index->add(train_rows, _buffered_vectors.data()));
     // PODArray::clear() keeps the allocated capacity. Swap with an empty 
array so the
     // full-segment build buffer is released before saving the index.
-    PODArray<float> empty_buffered_vectors;
-    _buffered_vectors.swap(empty_buffered_vectors);
+    {
+        PODArray<float> empty_buffered_vectors;
+        _buffered_vectors.swap(empty_buffered_vectors);
+    }
     Status status = _vector_index->save(_dir.get());
     if (!status.ok()) {
         // A failed save keeps whatever it managed to write, and under SNII 
that is
diff --git a/be/src/storage/index/ann/ann_index_writer.h 
b/be/src/storage/index/ann/ann_index_writer.h
index e876a0a3956..587348d28c9 100644
--- a/be/src/storage/index/ann/ann_index_writer.h
+++ b/be/src/storage/index/ann/ann_index_writer.h
@@ -75,14 +75,13 @@ public:
 
 private:
     Status _build_and_save(Int64 min_train_rows, Int64 effective_min_rows);
+    void _release_build_resources();
 
 #ifdef BE_TEST
     friend class TestAnnIndexColumnWriter;
 #endif
 
-    // VectorIndex shoule be managed by some cache.
-    // VectorIndex should be weak shared by AnnIndexWriter and 
VectorIndexReader
-    // This should be a weak_ptr
+    // The build index is owned by this writer and released when the segment 
is finished.
     std::shared_ptr<VectorIndex> _vector_index;
     PODArray<float> _buffered_vectors;
     int64_t _total_rows = 0;
diff --git a/be/src/storage/task/index_builder.cpp 
b/be/src/storage/task/index_builder.cpp
index 84b8edce3de..74f2b1c54be 100644
--- a/be/src/storage/task/index_builder.cpp
+++ b/be/src/storage/task/index_builder.cpp
@@ -719,97 +719,101 @@ Status 
IndexBuilder::handle_single_rowset(RowsetMetaSharedPtr output_rowset_meta
                     _index_file_writers.emplace(seg_ptr->id(), 
std::move(index_file_writer));
             DORIS_CHECK(inserted);
             DORIS_CHECK(index_file_writer_it->second != nullptr);
-            if (return_columns.empty()) {
-                // no columns to read
-                continue;
-            }
-            // create iterator for each segment
-            OlapReaderStatistics stats;
-            StorageReadOptions read_options(stats);
-            auto schema = std::make_shared<ReadSchema>(
-                    
project_columns_by_ordinal(output_rowset_schema->columns(), return_columns));
-            std::unique_ptr<RowwiseIterator> iter;
-            auto res = seg_ptr->new_iterator(schema, read_options, &iter);
-            
DBUG_EXECUTE_IF("IndexBuilder::handle_single_rowset_create_iterator_error", {
-                res = Status::Error<ErrorCode::INTERNAL_ERROR>(
-                        "debug point: 
handle_single_rowset_create_iterator_error");
-            })
-            if (!res.ok()) {
-                LOG(WARNING) << "failed to create iterator[" << seg_ptr->id()
-                             << "]: " << res.to_string();
-                return 
Status::Error<ErrorCode::ROWSET_READER_INIT>(res.to_string());
-            }
+            // Even without columns to rebuild, the inherited indexes must be 
sealed below.
+            if (!return_columns.empty()) {
+                // create iterator for each segment
+                OlapReaderStatistics stats;
+                StorageReadOptions read_options(stats);
+                auto schema = 
std::make_shared<ReadSchema>(project_columns_by_ordinal(
+                        output_rowset_schema->columns(), return_columns));
+                std::unique_ptr<RowwiseIterator> iter;
+                auto res = seg_ptr->new_iterator(schema, read_options, &iter);
+                
DBUG_EXECUTE_IF("IndexBuilder::handle_single_rowset_create_iterator_error", {
+                    res = Status::Error<ErrorCode::INTERNAL_ERROR>(
+                            "debug point: 
handle_single_rowset_create_iterator_error");
+                })
+                if (!res.ok()) {
+                    LOG(WARNING) << "failed to create iterator[" << 
seg_ptr->id()
+                                 << "]: " << res.to_string();
+                    return 
Status::Error<ErrorCode::ROWSET_READER_INIT>(res.to_string());
+                }
 
-            auto block = Block::create_unique(schema->create_read_block());
-            while (true) {
-                auto status = iter->next_batch(block.get());
-                
DBUG_EXECUTE_IF("IndexBuilder::handle_single_rowset_iterator_next_batch_error", 
{
-                    status = 
Status::Error<ErrorCode::SCHEMA_CHANGE_INFO_INVALID>(
-                            "next_batch fault injection");
-                });
-                if (!status.ok()) {
-                    if (status.is<ErrorCode::END_OF_FILE>()) {
-                        break;
+                auto block = Block::create_unique(schema->create_read_block());
+                while (true) {
+                    auto status = iter->next_batch(block.get());
+                    DBUG_EXECUTE_IF(
+                            
"IndexBuilder::handle_single_rowset_iterator_next_batch_error", {
+                                status = 
Status::Error<ErrorCode::SCHEMA_CHANGE_INFO_INVALID>(
+                                        "next_batch fault injection");
+                            });
+                    if (!status.ok()) {
+                        if (status.is<ErrorCode::END_OF_FILE>()) {
+                            break;
+                        }
+                        LOG(WARNING) << "failed to read next block when schema 
change for inverted "
+                                        "index."
+                                     << ", err=" << status.to_string();
+                        return status;
                     }
-                    LOG(WARNING)
-                            << "failed to read next block when schema change 
for inverted index."
-                            << ", err=" << status.to_string();
-                    return status;
-                }
 
-                // write inverted index data, or ann index data
-                status = _write_inverted_index_data(output_rowset_schema, 
iter->data_id(),
-                                                    block.get());
-                DBUG_EXECUTE_IF(
-                        
"IndexBuilder::handle_single_rowset_write_inverted_index_data_error", {
-                            status = Status::Error<ErrorCode::INTERNAL_ERROR>(
-                                    "debug point: "
-                                    
"handle_single_rowset_write_inverted_index_data_error");
-                        })
-                if (!status.ok()) {
-                    return status;
+                    // write inverted index data, or ann index data
+                    status = _write_inverted_index_data(output_rowset_schema, 
iter->data_id(),
+                                                        block.get());
+                    DBUG_EXECUTE_IF(
+                            
"IndexBuilder::handle_single_rowset_write_inverted_index_data_error", {
+                                status = 
Status::Error<ErrorCode::INTERNAL_ERROR>(
+                                        "debug point: "
+                                        
"handle_single_rowset_write_inverted_index_data_error");
+                            })
+                    if (!status.ok()) {
+                        return status;
+                    }
+                    block->clear_column_data();
                 }
-                block->clear_column_data();
-            }
 
-            // finish write inverted index, flush data to compound file
-            for (auto& writer_sign : inverted_index_writer_signs) {
-                try {
-                    auto index_column_writer_it = 
_index_column_writers.find(writer_sign);
-                    DORIS_CHECK(index_column_writer_it != 
_index_column_writers.end());
-                    DORIS_CHECK(index_column_writer_it->second != nullptr);
-                    RETURN_IF_ERROR(index_column_writer_it->second->finish());
-                    
DBUG_EXECUTE_IF("IndexBuilder::handle_single_rowset_index_build_finish_error", {
-                        _CLTHROWA(CL_ERR_IO,
-                                  "debug point: 
handle_single_rowset_index_build_finish_error");
-                    })
-                } catch (const std::exception& e) {
-                    return 
Status::Error<ErrorCode::INVERTED_INDEX_CLUCENE_ERROR>(
-                            "CLuceneError occurred: {}", e.what());
+                // finish write inverted index, flush data to compound file
+                for (auto& writer_sign : inverted_index_writer_signs) {
+                    try {
+                        auto index_column_writer_it = 
_index_column_writers.find(writer_sign);
+                        DORIS_CHECK(index_column_writer_it != 
_index_column_writers.end());
+                        DORIS_CHECK(index_column_writer_it->second != nullptr);
+                        
RETURN_IF_ERROR(index_column_writer_it->second->finish());
+                        DBUG_EXECUTE_IF(
+                                
"IndexBuilder::handle_single_rowset_index_build_finish_error", {
+                                    _CLTHROWA(CL_ERR_IO,
+                                              "debug point: "
+                                              
"handle_single_rowset_index_build_finish_error");
+                                })
+                    } catch (const std::exception& e) {
+                        return 
Status::Error<ErrorCode::INVERTED_INDEX_CLUCENE_ERROR>(
+                                "CLuceneError occurred: {}", e.what());
+                    }
                 }
             }
-        }
-        for (auto&& [seg_id, index_file_writer] : _index_file_writers) {
-            auto st = index_file_writer->begin_close();
-            
DBUG_EXECUTE_IF("IndexBuilder::handle_single_rowset_file_writer_close_error", {
-                st = Status::Error<ErrorCode::INVERTED_INDEX_CLUCENE_ERROR>(
-                        "debug point: 
handle_single_rowset_file_writer_close_error");
-            })
-            if (!st.ok()) {
-                LOG(ERROR) << "close index_file_writer error:" << st;
-                return st;
+            // Each segment owns an independent index file. Seal it before 
building the
+            // next segment: otherwise every completed ANN file remains in its 
RAM directory.
+            for (auto&& [seg_id, completed_file_writer] : _index_file_writers) 
{
+                auto st = completed_file_writer->begin_close();
+                
DBUG_EXECUTE_IF("IndexBuilder::handle_single_rowset_file_writer_close_error", {
+                    st = 
Status::Error<ErrorCode::INVERTED_INDEX_CLUCENE_ERROR>(
+                            "debug point: 
handle_single_rowset_file_writer_close_error");
+                })
+                if (!st.ok()) {
+                    LOG(ERROR) << "close completed_file_writer error:" << st;
+                    return st;
+                }
+                inverted_index_size += 
completed_file_writer->get_index_file_total_size();
             }
-            inverted_index_size += 
index_file_writer->get_index_file_total_size();
-        }
-        for (auto&& [seg_id, index_file_writer] : _index_file_writers) {
-            auto st = index_file_writer->finish_close();
-            if (!st.ok()) {
-                LOG(ERROR) << "wait close index_file_writer error:" << st;
-                return st;
+            for (auto&& [seg_id, completed_file_writer] : _index_file_writers) 
{
+                auto st = completed_file_writer->finish_close();
+                if (!st.ok()) {
+                    LOG(ERROR) << "wait close completed_file_writer error:" << 
st;
+                    return st;
+                }
             }
+            _index_column_writers.clear();
+            _index_file_writers.clear();
         }
-        _index_column_writers.clear();
-        _index_file_writers.clear();
         
output_rowset_meta->set_data_disk_size(output_rowset_meta->data_disk_size());
         
output_rowset_meta->set_total_disk_size(output_rowset_meta->total_disk_size() +
                                                 inverted_index_size);
@@ -828,21 +832,22 @@ Status IndexBuilder::_handle_single_rowset_snii(
     auto input_schema_it = _input_rowset_schemas.find(rowset_id);
     DORIS_CHECK(input_schema_it != _input_rowset_schemas.end());
 
+    size_t inverted_index_size = 0;
     for (auto& seg_ptr : segments) {
         RETURN_IF_ERROR(_rewrite_single_segment_snii(output_rowset_meta->fs(),
                                                      
output_rowset_meta->tablet_schema(),
                                                      *input_schema_it->second, 
rowset_id, seg_ptr));
+        // Blob staging and column writers belong to this segment, not the 
whole rowset.
+        for (auto&& [seg_id, index_file_writer] : _index_file_writers) {
+            RETURN_IF_ERROR(index_file_writer->begin_close());
+            inverted_index_size += 
index_file_writer->get_index_file_total_size();
+        }
+        for (auto&& [seg_id, index_file_writer] : _index_file_writers) {
+            RETURN_IF_ERROR(index_file_writer->finish_close());
+        }
+        _index_column_writers.clear();
+        _index_file_writers.clear();
     }
-    size_t inverted_index_size = 0;
-    for (auto&& [seg_id, index_file_writer] : _index_file_writers) {
-        RETURN_IF_ERROR(index_file_writer->begin_close());
-        inverted_index_size += index_file_writer->get_index_file_total_size();
-    }
-    for (auto&& [seg_id, index_file_writer] : _index_file_writers) {
-        RETURN_IF_ERROR(index_file_writer->finish_close());
-    }
-    _index_column_writers.clear();
-    _index_file_writers.clear();
     
output_rowset_meta->set_data_disk_size(output_rowset_meta->data_disk_size());
     
output_rowset_meta->set_total_disk_size(output_rowset_meta->total_disk_size() +
                                             inverted_index_size);


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to