This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/master by this push:
     new 0826f7e3c75 [fix](be) Fix data race on Segment::_footer_pb (#67720)
0826f7e3c75 is described below

commit 0826f7e3c7530c68ebe68a820ed71b53997ce7a0
Author: stinger1206 <[email protected]>
AuthorDate: Mon Nov 16 12:13:27 2026 +0200

    [fix](be) Fix data race on Segment::_footer_pb (#67720)
    
    ### What problem does this PR solve?
    
    Issue Number: close #67719
    
    Problem Summary:
    
    `Segment::_footer_pb` (`std::weak_ptr<SegmentFooterPB>`) was a second
    footer
    access path outside `StoragePageCache`:
    
    - `_get_segment_footer()` reads it via `lock()` and rebinds it
    (`_footer_pb = footer_pb_shared`) from query/compaction threads whenever
    the
    cached footer expires, while `get_metadata_size()` also accesses the
    same
    non-atomic `std::weak_ptr`. Concurrent assignment and `lock()` on the
    same
    weak_ptr, if these paths overlap, is a data race and therefore undefined
      behavior.
    - Moreover, the weak_ptr fast path could bypass the cache lookup
    whenever the
    footer object was still alive — including while the cache entry was
    present —
    so the cache could not observe real footer accesses and its LRU order
    could
    diverge from actual usage. After eviction the object could even remain
    alive
      through other `shared_ptr` references.
    
    In production (4.1.3) we observed intermittent BE SIGSEGVs, with
    innermost
    frames such as `SegmentFooterPB::ByteSizeLong`
    (`Segment::get_metadata_size`)
    and `~SegmentFooterPB` under `LRUCache::prune_if`, plus crashes at
    unrelated
    sites. Every crash was healed by a BE restart without repair and the
    implicated
    segment files stayed byte-identical, which is consistent with transient
    in-process memory/lifetime corruption rather than persistent on-disk
    corruption. The exact production failure interleaving has not been
    proven.
    See #67719 for a TSAN micro-repro of the unsafe `std::weak_ptr` access
    pattern.
    
    Fix:
    
    - remove `_footer_pb` entirely;
    - `_get_segment_footer()` always accesses the footer through
    `StoragePageCache`
    (single access path: lookup → parse & insert on miss), so eviction and
    LRU
      recency are handled consistently;
    - `get_metadata_size()` no longer reads the footer: footer memory is
    tracked by
    `StoragePageCache`'s own memory tracking, and doing a lookup there only
    for
    accounting would artificially refresh LRU recency on every metadata-size
      sweep without any real query access. This also matches `_open()`'s
      `_meta_mem_usage` estimation, which already excludes the footer;
    - update the stale `StoragePageCache::insert` comment that recommended
    storing
    a weak_ptr beside the cache (with `Segment::_get_segment_footer` as the
    example) to instead recommend going through cache lookup for every
    access.
    
    ### Release note
    
    Fix unsafe Segment footer caching that could cause BE crashes and bypass
    StoragePageCache LRU accounting.
    
    ### Check List (For Author)
    
    - Test <!-- At least one of them must be included. -->
        - [x] Regression test
        - [x] Unit Test
        - [x] Manual test (add detailed scripts or steps below)
        - [ ] No need to test or manual test. Explain why:
    - [ ] This is a refactor/code format and no logic has been changed.
            - [ ] Previous test can cover this change.
            - [ ] No code files have been changed.
            - [ ] Other reason <!-- Add your reason?  -->
    
    Unit test:
    `SegmentFooterCacheTest.ConcurrentGetSegmentFooterWithEviction`
    (adapted from #69811, thanks to @airborne12 for the original
    reproducer):
    8 threads concurrently call `_get_segment_footer()` on a segment while
    half
    of them keep erasing the footer's cache page, verifying that every
    caller
    gets a valid, identical footer regardless of concurrent evictions. A
    similar
    original workload in #69811 reproduced a heap-use-after-free under ASAN
    within 20 runs on the old weak_ptr implementation (a thread read the
    footer
    through `SegmentFooterPB::ByteSizeLong` after another thread freed it in
      `std::weak_ptr` assignment); with the cache-only design of this PR the
      specific unsafe weak_ptr reassignment path is no longer present.
    
    Manual test: the standalone TSAN micro-repro of the unsafe
    `std::weak_ptr`
    access pattern (attached to #67719) demonstrates the data race. The
    current
    cache-only implementation is deployed on a production canary BE and the
    crash
      family has not recurred under the previously-crashing workload.
    
    - Behavior changed:
        - [x] No.
        - [ ] Yes. <!-- Explain the behavior change -->
    
    - Does this need documentation?
        - [x] No.
    - [ ] Yes. <!-- Add document PR link here. eg:
    https://github.com/apache/doris-website/pull/1214 -->
    
    ### Check List (For Reviewer who merge this PR)
    
    - [ ] Confirm the release note
    - [ ] Confirm test cases
    - [ ] Confirm document
    - [ ] Add branch pick label <!-- Add branch pick label that this PR
    should merge into -->
    
    ---------
    
    Co-authored-by: stinger1206 <[email protected]>
---
 be/src/storage/cache/page_cache.h                  |   5 +-
 be/src/storage/segment/segment.cpp                 |  31 +--
 be/src/storage/segment/segment.h                   |   6 +-
 .../storage/segment/segment_footer_cache_test.cpp  | 283 ++++++++++++++++++++-
 4 files changed, 299 insertions(+), 26 deletions(-)

diff --git a/be/src/storage/cache/page_cache.h 
b/be/src/storage/cache/page_cache.h
index 6f88ff1571d..feec965213b 100644
--- a/be/src/storage/cache/page_cache.h
+++ b/be/src/storage/cache/page_cache.h
@@ -183,9 +183,8 @@ public:
     // size should be the size of the page instead of shared_ptr.
     // Internal implementation will wrap shared_ptr with 
MemoryTrackedPageWithPagePtr
     // Since we are using std::shared_ptr, so lify cycle of the page is not 
managed by
-    // this cache alone.
-    // User could store a weak_ptr to the page, and lock it when needed.
-    // See Segment::_get_segment_footer for example.
+    // this cache alone. Access the page through cache lookup 
(PageCacheHandle) so the
+    // cache observes accesses and the LRU order reflects actual usage.
     template <typename T>
     void insert(const CacheKey& key, T data, size_t size, PageCacheHandle* 
handle,
                 segment_v2::PageTypePB page_type, bool in_memory = false);
diff --git a/be/src/storage/segment/segment.cpp 
b/be/src/storage/segment/segment.cpp
index 30d03a2457c..e5ceb8264f3 100644
--- a/be/src/storage/segment/segment.cpp
+++ b/be/src/storage/segment/segment.cpp
@@ -367,9 +367,8 @@ io::UInt128Wrapper Segment::file_cache_key(std::string_view 
rowset_id, uint32_t
 }
 
 int64_t Segment::get_metadata_size() const {
-    std::shared_ptr<SegmentFooterPB> footer_pb_shared = _footer_pb.lock();
-    return sizeof(Segment) + (_pk_index_meta ? _pk_index_meta->ByteSizeLong() 
: 0) +
-           (footer_pb_shared ? footer_pb_shared->ByteSizeLong() : 0);
+    // The segment footer is owned and tracked by StoragePageCache, so it is 
not counted here.
+    return sizeof(Segment) + (_pk_index_meta ? _pk_index_meta->ByteSizeLong() 
: 0);
 }
 
 void Segment::update_metadata_size() {
@@ -550,7 +549,7 @@ Status Segment::new_iterator(ReadSchemaSPtr schema, const 
StorageReadOptions& re
 }
 
 Status Segment::_write_error_file(size_t file_size, size_t offset, size_t 
bytes_read, char* data,
-                                  io::IOContext& io_ctx) {
+                                  io::IOContext& io_ctx) const {
     if (!config::enbale_dump_error_file || !doris::config::is_cloud_mode()) {
         return Status::OK();
     }
@@ -610,8 +609,12 @@ Status Segment::_write_error_file(size_t file_size, size_t 
offset, size_t bytes_
     return Status::OK(); // already exists
 };
 
+// Parses the footer into the caller-owned output object without modifying any
+// Segment member (the reason this method is const). Concurrent cache misses 
may
+// parse independent, content-identical footer copies; sharing is handled by
+// StoragePageCache in _get_segment_footer().
 Status Segment::_parse_footer(std::shared_ptr<SegmentFooterPB>& footer, 
OlapReaderStatistics* stats,
-                              const io::IOContext* source_io_ctx) {
+                              const io::IOContext* source_io_ctx) const {
     // Footer := SegmentFooterPB, FooterPBSize(4), FooterPBChecksum(4), 
MagicNumber(4)
     auto file_size = _file_reader->size();
     if (file_size < 12) {
@@ -1474,16 +1477,6 @@ Status Segment::seek_and_read_by_rowid(const 
TabletColumn& read_column, SlotDesc
 Status Segment::_get_segment_footer(std::shared_ptr<SegmentFooterPB>& 
footer_pb,
                                     OlapReaderStatistics* stats,
                                     const io::IOContext* source_io_ctx) {
-    std::shared_ptr<SegmentFooterPB> footer_pb_shared = _footer_pb.lock();
-    if (footer_pb_shared != nullptr) {
-        footer_pb = footer_pb_shared;
-        return Status::OK();
-    }
-
-    VLOG_DEBUG << fmt::format("Segment footer of {}:{}:{} is missing, try to 
load it",
-                              _file_reader->path().native(), 
_file_reader->size(),
-                              _file_reader->size() - 12);
-
     StoragePageCache* segment_footer_cache = 
ExecEnv::GetInstance()->get_storage_page_cache();
     DCHECK(segment_footer_cache != nullptr);
 
@@ -1498,6 +1491,10 @@ Status 
Segment::_get_segment_footer(std::shared_ptr<SegmentFooterPB>& footer_pb,
     //   as other index/metadata pages and avoids competing with DATA_PAGE 
budget.
     if (!segment_footer_cache->lookup(cache_key, &cache_handle,
                                       segment_v2::PageTypePB::INDEX_PAGE)) {
+        VLOG_DEBUG << fmt::format("Segment footer of {}:{}:{} is missing, try 
to load it",
+                                  _file_reader->path().native(), 
_file_reader->size(),
+                                  _file_reader->size() - 12);
+        std::shared_ptr<SegmentFooterPB> footer_pb_shared;
         RETURN_IF_ERROR(_parse_footer(footer_pb_shared, stats, source_io_ctx));
         segment_footer_cache->insert(cache_key, footer_pb_shared, 
footer_pb_shared->ByteSizeLong(),
                                      &cache_handle, 
segment_v2::PageTypePB::INDEX_PAGE);
@@ -1506,9 +1503,7 @@ Status 
Segment::_get_segment_footer(std::shared_ptr<SegmentFooterPB>& footer_pb,
                                   _file_reader->path().native(), 
_file_reader->size(),
                                   _file_reader->size() - 12);
     }
-    footer_pb_shared = cache_handle.get<std::shared_ptr<SegmentFooterPB>>();
-    _footer_pb = footer_pb_shared;
-    footer_pb = footer_pb_shared;
+    footer_pb = cache_handle.get<std::shared_ptr<SegmentFooterPB>>();
     return Status::OK();
 }
 
diff --git a/be/src/storage/segment/segment.h b/be/src/storage/segment/segment.h
index 7a0228b0d29..05aa1383e05 100644
--- a/be/src/storage/segment/segment.h
+++ b/be/src/storage/segment/segment.h
@@ -285,13 +285,13 @@ private:
     Status _open(OlapReaderStatistics* stats, const io::IOContext* io_ctx = 
nullptr);
     Status _parse_footer(std::shared_ptr<SegmentFooterPB>& footer,
                          OlapReaderStatistics* stats = nullptr,
-                         const io::IOContext* io_ctx = nullptr);
+                         const io::IOContext* io_ctx = nullptr) const;
     Status _create_column_meta(const SegmentFooterPB& footer, 
OlapReaderStatistics* stats = nullptr,
                                const io::IOContext* io_ctx = nullptr);
     Status _load_pk_bloom_filter(OlapReaderStatistics* stats,
                                  const io::IOContext* io_ctx = nullptr);
     Status _write_error_file(size_t file_size, size_t offset, size_t 
bytes_read, char* data,
-                             io::IOContext& io_ctx);
+                             io::IOContext& io_ctx) const;
 
     Status _open_index_file_reader();
 
@@ -362,8 +362,6 @@ private:
 
     DorisCallOnce<Status> _create_column_meta_once_call;
 
-    std::weak_ptr<SegmentFooterPB> _footer_pb;
-
     // Cached raw_data_bytes per column unique id, populated once in 
_create_column_meta().
     std::unordered_map<int32_t, uint64_t> _column_uid_to_raw_bytes;
 
diff --git a/be/test/storage/segment/segment_footer_cache_test.cpp 
b/be/test/storage/segment/segment_footer_cache_test.cpp
index ab83da3a5ea..cc76e0e51fe 100644
--- a/be/test/storage/segment/segment_footer_cache_test.cpp
+++ b/be/test/storage/segment/segment_footer_cache_test.cpp
@@ -17,11 +17,15 @@
 
 #include <gtest/gtest.h>
 
+#include <atomic>
 #include <random>
+#include <thread>
 
 #include "core/field.h"
 #include "cpp/sync_point.h"
 #include "io/cache/remote_scan_cache_write_limiter.h"
+#include "runtime/memory/cache_manager.h"
+#include "runtime/thread_context.h"
 #include "storage/cache/page_cache.h"
 #include "storage/segment/segment.h"
 #include "storage/segment/vertical_segment_writer.h"
@@ -152,6 +156,69 @@ private:
     Segments _segments;
 };
 
+namespace {
+
+// Under BE_TEST, CacheManager::register_cache() replaces same-type entries and
+// unregister_cache() is a no-op, so constructing any StoragePageCache hijacks 
the
+// CacheManager registrations and leaves them dangling after that cache is
+// destroyed. This restores the current global cache's registrations (its 
policy
+// members are private; BE UT builds with -fno-access-control).
+void reregister_global_storage_page_cache_policies() {
+    auto* global_cache = ExecEnv::GetInstance()->get_storage_page_cache();
+    auto* cache_manager = CacheManager::instance();
+    cache_manager->register_cache(global_cache->_data_page_cache.get());
+    cache_manager->register_cache(global_cache->_index_page_cache.get());
+    cache_manager->register_cache(global_cache->_pk_index_page_cache.get());
+}
+
+// Restores the global cache's CacheManager registrations on destruction, 
including
+// on ASSERT failure (see reregister_global_storage_page_cache_policies()).
+class ScopedGlobalCachePolicyRegistration {
+public:
+    ScopedGlobalCachePolicyRegistration() = default;
+    ~ScopedGlobalCachePolicyRegistration() { 
reregister_global_storage_page_cache_policies(); }
+
+    ScopedGlobalCachePolicyRegistration(const 
ScopedGlobalCachePolicyRegistration&) = delete;
+    ScopedGlobalCachePolicyRegistration& operator=(const 
ScopedGlobalCachePolicyRegistration&) =
+            delete;
+};
+
+// Swaps the global storage page cache for the lifetime of the guard, so a 
test can
+// run against a cache with controlled capacity. The destructor restores the
+// previous global cache and its CacheManager registrations (see
+// reregister_global_storage_page_cache_policies()).
+class ScopedStoragePageCache {
+public:
+    explicit ScopedStoragePageCache(StoragePageCache* cache)
+            : _original(ExecEnv::GetInstance()->get_storage_page_cache()) {
+        DCHECK(_original != nullptr);
+        ExecEnv::GetInstance()->set_storage_page_cache(cache);
+    }
+    ~ScopedStoragePageCache() {
+        ExecEnv::GetInstance()->set_storage_page_cache(_original);
+        reregister_global_storage_page_cache_policies();
+    }
+
+    ScopedStoragePageCache(const ScopedStoragePageCache&) = delete;
+    ScopedStoragePageCache& operator=(const ScopedStoragePageCache&) = delete;
+
+private:
+    StoragePageCache* _original;
+};
+
+// Enables SyncPoint processing for the lifetime of the guard; without it, an
+// ASSERT failure would leave processing enabled for subsequent tests.
+class ScopedSyncPointProcessing {
+public:
+    ScopedSyncPointProcessing() { 
SyncPoint::get_instance()->enable_processing(); }
+    ~ScopedSyncPointProcessing() { 
SyncPoint::get_instance()->disable_processing(); }
+
+    ScopedSyncPointProcessing(const ScopedSyncPointProcessing&) = delete;
+    ScopedSyncPointProcessing& operator=(const ScopedSyncPointProcessing&) = 
delete;
+};
+
+} // namespace
+
 TEST_F(SegmentFooterCacheTest, TestGetSegmentFooter) {
     for (auto segment_ptr : _segments) {
         std::shared_ptr<segment_v2::SegmentFooterPB> footer;
@@ -166,13 +233,223 @@ TEST_F(SegmentFooterCacheTest, TestGetSegmentFooter) {
     }
 }
 
+// The segment footer is owned by the index page cache and every access goes 
through the cache
+// lookup: a miss re-parses the footer and re-inserts it, while other threads 
keep reading it
+// (variant sub-path readers call _get_segment_footer() once per path). A 
similar concurrent
+// eviction + access workload exposed a heap-use-after-free in the old 
weak_ptr implementation
+// (#69811); this test checks that every caller gets a valid, identical footer 
regardless of
+// concurrent evictions.
+TEST_F(SegmentFooterCacheTest, ConcurrentGetSegmentFooterWithEviction) {
+    auto segment_ptr = _segments.front();
+    auto* segment_footer_cache = 
ExecEnv::GetInstance()->get_storage_page_cache();
+    ASSERT_NE(segment_footer_cache, nullptr);
+    const auto cache_key = segment_ptr->get_segment_footer_cache_key();
+
+    std::shared_ptr<segment_v2::SegmentFooterPB> baseline;
+    ASSERT_TRUE(segment_ptr->_get_segment_footer(baseline, nullptr).ok());
+    ASSERT_NE(baseline, nullptr);
+    const auto expected_num_rows = baseline->num_rows();
+    const auto expected_size = baseline->ByteSizeLong();
+    baseline.reset();
+
+    constexpr int kThreads = 8;
+    constexpr int kIterations = 2000;
+    std::atomic<int> failures {0};
+    std::vector<std::thread> threads;
+    for (int t = 0; t < kThreads; ++t) {
+        threads.emplace_back([&, t] {
+            // Reading the footer from the file needs the thread context a 
scanner thread has.
+            SCOPED_ATTACH_TASK(ExecEnv::GetInstance()->orphan_mem_tracker());
+            for (int i = 0; i < kIterations; ++i) {
+                if (t % 2 == 0) {
+                    // simulate the LRU eviction of the footer page
+                    segment_footer_cache->erase(cache_key, 
segment_v2::PageTypePB::INDEX_PAGE);
+                }
+                std::shared_ptr<segment_v2::SegmentFooterPB> footer;
+                Status st = segment_ptr->_get_segment_footer(footer, nullptr);
+                if (!st.ok() || footer == nullptr || footer->num_rows() != 
expected_num_rows ||
+                    footer->ByteSizeLong() != expected_size) {
+                    ++failures;
+                }
+            }
+        });
+    }
+    for (auto& th : threads) {
+        th.join();
+    }
+    EXPECT_EQ(failures.load(), 0);
+}
+
+// Every footer access goes through StoragePageCache, so repeated accesses must
+// keep the footer page recent in the LRU order. The test runs against a
+// single-shard index cache that fits exactly two footer pages plus half a 
page of
+// slack: after A and B are loaded, A is re-accessed (refreshing its recency,
+// served from the cache), and a third footer-sized page is inserted. The 
insertion
+// must evict B — the least recently used entry — and keep A, proving that the
+// repeated access refreshed A's position.
+TEST_F(SegmentFooterCacheTest, RepeatedFooterAccessesRefreshLruOrder) {
+    auto segment_a = _segments[0];
+    auto segment_b = _segments[1];
+    const auto key_a = segment_a->get_segment_footer_cache_key();
+    const auto key_b = segment_b->get_segment_footer_cache_key();
+    const StoragePageCache::CacheKey pressure_key("footer_lru_pressure_test", 
0, 0);
+
+    // Measure the cache space (charge + handle overhead) of the entries 
involved:
+    // footer A, footers A+B, and the pressure page. The pressure footer 
object is
+    // reused in the scenario phase, so its charge is identical by 
construction.
+    int64_t usage_a = 0;
+    int64_t usage_ab = 0;
+    int64_t usage_abp = 0;
+    std::shared_ptr<segment_v2::SegmentFooterPB> pressure_footer;
+    {
+        StoragePageCache measure_cache(1 << 20, 100, 0, 1);
+        ScopedStoragePageCache scoped_cache(&measure_cache);
+        std::shared_ptr<segment_v2::SegmentFooterPB> footer;
+        ASSERT_TRUE(segment_a->_get_segment_footer(footer, nullptr).ok());
+        usage_a = 
measure_cache._get_page_cache(segment_v2::PageTypePB::INDEX_PAGE)->get_usage();
+        ASSERT_TRUE(segment_b->_get_segment_footer(footer, nullptr).ok());
+        usage_ab = 
measure_cache._get_page_cache(segment_v2::PageTypePB::INDEX_PAGE)->get_usage();
+        pressure_footer = 
std::make_shared<segment_v2::SegmentFooterPB>(*footer);
+        {
+            PageCacheHandle handle;
+            measure_cache.insert(pressure_key, pressure_footer, 
pressure_footer->ByteSizeLong(),
+                                 &handle, segment_v2::PageTypePB::INDEX_PAGE);
+        }
+        usage_abp = 
measure_cache._get_page_cache(segment_v2::PageTypePB::INDEX_PAGE)->get_usage();
+        ASSERT_GT(usage_ab, usage_a);
+        ASSERT_GT(usage_abp, usage_ab);
+    }
+    const int64_t weight_a = usage_a;
+    const int64_t weight_b = usage_ab - usage_a;
+    const int64_t weight_pressure = usage_abp - usage_ab;
+
+    // Capacity for exactly two footer pages plus half of A's page as slack. 
The
+    // slack guarantees — verified here explicitly — that the pressure insert 
both
+    // exceeds the capacity (it must evict) and fits again after evicting 
exactly
+    // one entry (it must evict no more than the least recently used page).
+    const int64_t capacity = usage_ab + weight_a / 2;
+    ASSERT_GT(weight_a + weight_b + weight_pressure, capacity);
+    ASSERT_LE(weight_a + weight_pressure, capacity);
+
+    StoragePageCache small_cache(capacity, 100, 0, 1);
+    ScopedStoragePageCache scoped_cache(&small_cache);
+
+    ScopedSyncPointProcessing scoped_sync_point;
+    int parse_count = 0;
+    SyncPoint::CallbackGuard guard;
+    SyncPoint::get_instance()->set_call_back(
+            "Segment::_parse_footer::io_ctx", [&](auto&&) { ++parse_count; }, 
&guard);
+
+    std::shared_ptr<segment_v2::SegmentFooterPB> footer;
+
+    // Load A and B: both are cache misses and are parsed; A is now the least
+    // recently used footer page.
+    ASSERT_TRUE(segment_a->_get_segment_footer(footer, nullptr).ok());
+    ASSERT_TRUE(segment_b->_get_segment_footer(footer, nullptr).ok());
+    ASSERT_EQ(parse_count, 2);
+
+    // Repeated access of A: served from the cache (no additional parse) and
+    // refreshes A as the most recently used footer page.
+    ASSERT_TRUE(segment_a->_get_segment_footer(footer, nullptr).ok());
+    ASSERT_NE(footer, nullptr);
+    ASSERT_EQ(parse_count, 2);
+
+    // Insert the third footer-sized index page: it does not fit, so the cache
+    // evicts the least recently used page, which must be B — not the 
re-accessed A.
+    {
+        PageCacheHandle pressure_handle;
+        small_cache.insert(pressure_key, pressure_footer, 
pressure_footer->ByteSizeLong(),
+                           &pressure_handle, 
segment_v2::PageTypePB::INDEX_PAGE);
+    }
+
+    {
+        PageCacheHandle probe;
+        EXPECT_TRUE(small_cache.lookup(key_a, &probe, 
segment_v2::PageTypePB::INDEX_PAGE));
+    }
+    {
+        PageCacheHandle probe;
+        EXPECT_FALSE(small_cache.lookup(key_b, &probe, 
segment_v2::PageTypePB::INDEX_PAGE));
+    }
+    {
+        PageCacheHandle probe;
+        EXPECT_TRUE(small_cache.lookup(pressure_key, &probe, 
segment_v2::PageTypePB::INDEX_PAGE));
+    }
+    ASSERT_EQ(parse_count, 2);
+}
+
+// VariantExternalMetaReader keeps the footer shared_ptr returned by
+// _get_segment_footer() alive for the reader's whole lifetime. This test pins 
the
+// lifetime and cache-accounting semantics under eviction: after the cached 
entry
+// is dropped, the held copy stays valid; the next access re-parses the footer
+// (counted via sync point) into a footer that is valid and content-identical; 
and
+// the index cache tracker is charged exactly while the entry is cached (footer
+// bytes plus entry overhead), uncharged at eviction, re-charged identically at
+// reload, and unaffected by holders that outlive the entry. This verifies
+// cache-tracker bookkeeping and the retained footer's validity, not actual 
heap
+// deallocation.
+TEST_F(SegmentFooterCacheTest, 
HeldFooterStaysValidAcrossCacheEvictionAndReload) {
+    auto* segment_footer_cache = 
ExecEnv::GetInstance()->get_storage_page_cache();
+    ASSERT_NE(segment_footer_cache, nullptr);
+    auto segment_ptr = _segments.front();
+    const auto cache_key = segment_ptr->get_segment_footer_cache_key();
+    const auto tracker = 
segment_footer_cache->mem_tracker(segment_v2::PageTypePB::INDEX_PAGE);
+
+    // Drop whatever Segment::open() during SetUp may have left in the cache.
+    segment_footer_cache->erase(cache_key, segment_v2::PageTypePB::INDEX_PAGE);
+
+    ScopedSyncPointProcessing scoped_sync_point;
+    int parse_count = 0;
+    SyncPoint::CallbackGuard guard;
+    SyncPoint::get_instance()->set_call_back(
+            "Segment::_parse_footer::io_ctx", [&](auto&&) { ++parse_count; }, 
&guard);
+
+    // First access: the footer is parsed; the cache charges the entry — the
+    // footer's own bytes plus the cache entry overhead (handle + key).
+    const int64_t usage_before = tracker->consumption();
+    std::shared_ptr<segment_v2::SegmentFooterPB> held;
+    ASSERT_TRUE(segment_ptr->_get_segment_footer(held, nullptr).ok());
+    ASSERT_NE(held, nullptr);
+    ASSERT_EQ(parse_count, 1);
+    const auto num_rows = held->num_rows();
+    const auto byte_size = held->ByteSizeLong();
+    const int64_t tracked_bytes = tracker->consumption() - usage_before;
+    ASSERT_GT(tracked_bytes, static_cast<int64_t>(byte_size));
+
+    // Evict the entry: the cache uncharges it while the held copy keeps the 
footer
+    // object alive (the reader's lifetime guarantee).
+    segment_footer_cache->erase(cache_key, segment_v2::PageTypePB::INDEX_PAGE);
+    EXPECT_EQ(tracker->consumption(), usage_before);
+
+    // The next access re-parses the footer (a new parse is counted) and 
charges
+    // the same measured amount again. Whether the reloaded footer is the same
+    // object as the held one is an implementation detail; what matters is 
that it
+    // is valid and content-identical.
+    std::shared_ptr<segment_v2::SegmentFooterPB> reloaded;
+    ASSERT_TRUE(segment_ptr->_get_segment_footer(reloaded, nullptr).ok());
+    ASSERT_NE(reloaded, nullptr);
+    ASSERT_EQ(parse_count, 2);
+    EXPECT_EQ(tracker->consumption() - usage_before, tracked_bytes);
+    EXPECT_EQ(reloaded->num_rows(), num_rows);
+    EXPECT_EQ(reloaded->ByteSizeLong(), byte_size);
+    EXPECT_EQ(held->SerializeAsString(), reloaded->SerializeAsString());
+
+    // The held copy stays valid across the eviction and the reload.
+    EXPECT_EQ(held->num_rows(), num_rows);
+    EXPECT_EQ(held->ByteSizeLong(), byte_size);
+
+    // Releasing the held copy frees that reference without touching the 
cache's
+    // accounting (the cached copy stays charged).
+    held.reset();
+    EXPECT_EQ(tracker->consumption() - usage_before, tracked_bytes);
+    ASSERT_EQ(parse_count, 2);
+}
+
 TEST_F(SegmentFooterCacheTest, GetSegmentFooterPropagatesIoContext) {
     auto segment_ptr = _segments.front();
     auto* segment_footer_cache = 
ExecEnv::GetInstance()->get_storage_page_cache();
     ASSERT_NE(segment_footer_cache, nullptr);
     segment_footer_cache->erase(segment_ptr->get_segment_footer_cache_key(),
                                 segment_v2::PageTypePB::INDEX_PAGE);
-    segment_ptr->_footer_pb.reset();
 
     TUniqueId query_id;
     query_id.hi = 100;
@@ -280,6 +557,10 @@ TEST_F(SegmentFooterCacheTest, 
TestGetSegmentFooterCacheKey) {
 }
 
 TEST_F(SegmentFooterCacheTest, TestSemgnetFooterPBPage) {
+    // The local cache below registers its policies with CacheManager (BE_TEST
+    // unregister is a no-op); restore the global cache's registrations on 
exit,
+    // including on ASSERT failure.
+    ScopedGlobalCachePolicyRegistration registration_guard;
     StoragePageCache cache(16 * 2048, 0, 0, 16);
     for (auto segment_ptr : _segments) {
         std::shared_ptr<segment_v2::SegmentFooterPB> footer;


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to