This is an automated email from the ASF dual-hosted git repository.
hello-stephen pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 51e384b919e branch-4.2: [fix](be) Fix data race on Segment::_footer_pb
#67720 (#70638)
51e384b919e is described below
commit 51e384b919e228a6b0512dd1b00824a57c89f0ef
Author: stinger1206 <[email protected]>
AuthorDate: Tue Nov 17 10:44:22 2026 +0200
branch-4.2: [fix](be) Fix data race on Segment::_footer_pb #67720 (#70638)
Cherry-pick of #67720 to branch-4.2. The change is identical to the
merged master commit; the only adaptation is that the test file keeps
this branch's `#include "storage/segment/segment_writer.h"` (master's
later rename to `vertical_segment_writer.h` is not part of this branch).
### What problem does this PR solve?
Issue Number: close #67719
Related PR: #67720
Problem Summary:
`Segment::_footer_pb` (`std::weak_ptr<SegmentFooterPB>`) was a second
footer
access path outside `StoragePageCache`:
- `_get_segment_footer()` reads it via `lock()` and rebinds it
(`_footer_pb = footer_pb_shared`) from query/compaction threads whenever
the
cached footer expires, while `get_metadata_size()` also accesses the
same
non-atomic `std::weak_ptr`. Concurrent assignment and `lock()` on the
same
weak_ptr, if these paths overlap, is a data race and therefore undefined
behavior.
- Moreover, the weak_ptr fast path could bypass the cache lookup
whenever the
footer object was still alive — including while the cache entry was
present —
so the cache could not observe real footer accesses and its LRU order
could
diverge from actual usage.
In production (4.1.3) this crash family manifested as intermittent BE
SIGSEGVs, with innermost frames such as `SegmentFooterPB::ByteSizeLong`
(`Segment::get_metadata_size`) and `~SegmentFooterPB` under
`LRUCache::prune_if`, healed by BE restart with segment files
byte-identical.
See #67719 for a TSAN micro-repro of the unsafe `std::weak_ptr` access
pattern.
Fix (identical to #67720):
- remove `_footer_pb` entirely;
- `_get_segment_footer()` always accesses the footer through
`StoragePageCache` (single access path: lookup → parse & insert on
miss), so
eviction and LRU recency are handled consistently;
- `get_metadata_size()` no longer reads the footer: footer memory is
tracked
by `StoragePageCache`'s own memory tracking, and doing a lookup there
only
for accounting would artificially refresh LRU recency on every
metadata-size sweep without any real query access;
- update the stale `StoragePageCache::insert` comment to instead
recommend
going through cache lookup for every access;
- mark `Segment::_parse_footer()` and its helper `_write_error_file()`
const;
- tests: concurrent eviction regression (adapted from #69811), LRU
recency of
repeated footer accesses, and held-footer lifetime/cache-accounting
across
eviction and reload.
### Release note
Fix unsafe Segment footer caching that could cause BE crashes and bypass
StoragePageCache LRU accounting.
---
be/src/storage/cache/page_cache.h | 5 +-
be/src/storage/segment/segment.cpp | 31 +--
be/src/storage/segment/segment.h | 6 +-
.../storage/segment/segment_footer_cache_test.cpp | 283 ++++++++++++++++++++-
4 files changed, 299 insertions(+), 26 deletions(-)
diff --git a/be/src/storage/cache/page_cache.h
b/be/src/storage/cache/page_cache.h
index 1cc98c5cb49..30c3c860237 100644
--- a/be/src/storage/cache/page_cache.h
+++ b/be/src/storage/cache/page_cache.h
@@ -183,9 +183,8 @@ public:
// size should be the size of the page instead of shared_ptr.
// Internal implementation will wrap shared_ptr with
MemoryTrackedPageWithPagePtr
// Since we are using std::shared_ptr, so lify cycle of the page is not
managed by
- // this cache alone.
- // User could store a weak_ptr to the page, and lock it when needed.
- // See Segment::_get_segment_footer for example.
+ // this cache alone. Access the page through cache lookup
(PageCacheHandle) so the
+ // cache observes accesses and the LRU order reflects actual usage.
template <typename T>
void insert(const CacheKey& key, T data, size_t size, PageCacheHandle*
handle,
segment_v2::PageTypePB page_type, bool in_memory = false);
diff --git a/be/src/storage/segment/segment.cpp
b/be/src/storage/segment/segment.cpp
index 90bddf887f6..c49e3105300 100644
--- a/be/src/storage/segment/segment.cpp
+++ b/be/src/storage/segment/segment.cpp
@@ -370,9 +370,8 @@ io::UInt128Wrapper Segment::file_cache_key(std::string_view
rowset_id, uint32_t
}
int64_t Segment::get_metadata_size() const {
- std::shared_ptr<SegmentFooterPB> footer_pb_shared = _footer_pb.lock();
- return sizeof(Segment) + (_pk_index_meta ? _pk_index_meta->ByteSizeLong()
: 0) +
- (footer_pb_shared ? footer_pb_shared->ByteSizeLong() : 0);
+ // The segment footer is owned and tracked by StoragePageCache, so it is
not counted here.
+ return sizeof(Segment) + (_pk_index_meta ? _pk_index_meta->ByteSizeLong()
: 0);
}
void Segment::update_metadata_size() {
@@ -544,7 +543,7 @@ Status Segment::new_iterator(SchemaSPtr schema, const
StorageReadOptions& read_o
}
Status Segment::_write_error_file(size_t file_size, size_t offset, size_t
bytes_read, char* data,
- io::IOContext& io_ctx) {
+ io::IOContext& io_ctx) const {
if (!config::enbale_dump_error_file || !doris::config::is_cloud_mode()) {
return Status::OK();
}
@@ -604,8 +603,12 @@ Status Segment::_write_error_file(size_t file_size, size_t
offset, size_t bytes_
return Status::OK(); // already exists
};
+// Parses the footer into the caller-owned output object without modifying any
+// Segment member (the reason this method is const). Concurrent cache misses
may
+// parse independent, content-identical footer copies; sharing is handled by
+// StoragePageCache in _get_segment_footer().
Status Segment::_parse_footer(std::shared_ptr<SegmentFooterPB>& footer,
OlapReaderStatistics* stats,
- const io::IOContext* source_io_ctx) {
+ const io::IOContext* source_io_ctx) const {
// Footer := SegmentFooterPB, FooterPBSize(4), FooterPBChecksum(4),
MagicNumber(4)
auto file_size = _file_reader->size();
if (file_size < 12) {
@@ -1300,16 +1303,6 @@ Status Segment::seek_and_read_by_rowid(const
TabletSchema& schema, SlotDescripto
Status Segment::_get_segment_footer(std::shared_ptr<SegmentFooterPB>&
footer_pb,
OlapReaderStatistics* stats,
const io::IOContext* source_io_ctx) {
- std::shared_ptr<SegmentFooterPB> footer_pb_shared = _footer_pb.lock();
- if (footer_pb_shared != nullptr) {
- footer_pb = footer_pb_shared;
- return Status::OK();
- }
-
- VLOG_DEBUG << fmt::format("Segment footer of {}:{}:{} is missing, try to
load it",
- _file_reader->path().native(),
_file_reader->size(),
- _file_reader->size() - 12);
-
StoragePageCache* segment_footer_cache =
ExecEnv::GetInstance()->get_storage_page_cache();
DCHECK(segment_footer_cache != nullptr);
@@ -1324,6 +1317,10 @@ Status
Segment::_get_segment_footer(std::shared_ptr<SegmentFooterPB>& footer_pb,
// as other index/metadata pages and avoids competing with DATA_PAGE
budget.
if (!segment_footer_cache->lookup(cache_key, &cache_handle,
segment_v2::PageTypePB::INDEX_PAGE)) {
+ VLOG_DEBUG << fmt::format("Segment footer of {}:{}:{} is missing, try
to load it",
+ _file_reader->path().native(),
_file_reader->size(),
+ _file_reader->size() - 12);
+ std::shared_ptr<SegmentFooterPB> footer_pb_shared;
RETURN_IF_ERROR(_parse_footer(footer_pb_shared, stats, source_io_ctx));
segment_footer_cache->insert(cache_key, footer_pb_shared,
footer_pb_shared->ByteSizeLong(),
&cache_handle,
segment_v2::PageTypePB::INDEX_PAGE);
@@ -1332,9 +1329,7 @@ Status
Segment::_get_segment_footer(std::shared_ptr<SegmentFooterPB>& footer_pb,
_file_reader->path().native(),
_file_reader->size(),
_file_reader->size() - 12);
}
- footer_pb_shared = cache_handle.get<std::shared_ptr<SegmentFooterPB>>();
- _footer_pb = footer_pb_shared;
- footer_pb = footer_pb_shared;
+ footer_pb = cache_handle.get<std::shared_ptr<SegmentFooterPB>>();
return Status::OK();
}
diff --git a/be/src/storage/segment/segment.h b/be/src/storage/segment/segment.h
index 5ec9f9b3c8a..543f910bd7d 100644
--- a/be/src/storage/segment/segment.h
+++ b/be/src/storage/segment/segment.h
@@ -255,14 +255,14 @@ private:
Status _open(OlapReaderStatistics* stats, const io::IOContext* io_ctx =
nullptr);
Status _parse_footer(std::shared_ptr<SegmentFooterPB>& footer,
OlapReaderStatistics* stats = nullptr,
- const io::IOContext* io_ctx = nullptr);
+ const io::IOContext* io_ctx = nullptr) const;
Status _create_column_meta(const SegmentFooterPB& footer,
OlapReaderStatistics* stats = nullptr,
const io::IOContext* io_ctx = nullptr);
Status _load_pk_bloom_filter(OlapReaderStatistics* stats,
const io::IOContext* io_ctx = nullptr);
Status _write_error_file(size_t file_size, size_t offset, size_t
bytes_read, char* data,
- io::IOContext& io_ctx);
+ io::IOContext& io_ctx) const;
Status _open_index_file_reader();
@@ -319,8 +319,6 @@ private:
DorisCallOnce<Status> _create_column_meta_once_call;
- std::weak_ptr<SegmentFooterPB> _footer_pb;
-
// Cached raw_data_bytes per column unique id, populated once in
_create_column_meta().
std::unordered_map<int32_t, uint64_t> _column_uid_to_raw_bytes;
diff --git a/be/test/storage/segment/segment_footer_cache_test.cpp
b/be/test/storage/segment/segment_footer_cache_test.cpp
index d6af7b5146f..4004761fe84 100644
--- a/be/test/storage/segment/segment_footer_cache_test.cpp
+++ b/be/test/storage/segment/segment_footer_cache_test.cpp
@@ -17,11 +17,15 @@
#include <gtest/gtest.h>
+#include <atomic>
#include <random>
+#include <thread>
#include "core/field.h"
#include "cpp/sync_point.h"
#include "io/cache/remote_scan_cache_write_limiter.h"
+#include "runtime/memory/cache_manager.h"
+#include "runtime/thread_context.h"
#include "storage/cache/page_cache.h"
#include "storage/segment/segment.h"
#include "storage/segment/segment_writer.h"
@@ -152,6 +156,69 @@ private:
Segments _segments;
};
+namespace {
+
+// Under BE_TEST, CacheManager::register_cache() replaces same-type entries and
+// unregister_cache() is a no-op, so constructing any StoragePageCache hijacks
the
+// CacheManager registrations and leaves them dangling after that cache is
+// destroyed. This restores the current global cache's registrations (its
policy
+// members are private; BE UT builds with -fno-access-control).
+void reregister_global_storage_page_cache_policies() {
+ auto* global_cache = ExecEnv::GetInstance()->get_storage_page_cache();
+ auto* cache_manager = CacheManager::instance();
+ cache_manager->register_cache(global_cache->_data_page_cache.get());
+ cache_manager->register_cache(global_cache->_index_page_cache.get());
+ cache_manager->register_cache(global_cache->_pk_index_page_cache.get());
+}
+
+// Restores the global cache's CacheManager registrations on destruction,
including
+// on ASSERT failure (see reregister_global_storage_page_cache_policies()).
+class ScopedGlobalCachePolicyRegistration {
+public:
+ ScopedGlobalCachePolicyRegistration() = default;
+ ~ScopedGlobalCachePolicyRegistration() {
reregister_global_storage_page_cache_policies(); }
+
+ ScopedGlobalCachePolicyRegistration(const
ScopedGlobalCachePolicyRegistration&) = delete;
+ ScopedGlobalCachePolicyRegistration& operator=(const
ScopedGlobalCachePolicyRegistration&) =
+ delete;
+};
+
+// Swaps the global storage page cache for the lifetime of the guard, so a
test can
+// run against a cache with controlled capacity. The destructor restores the
+// previous global cache and its CacheManager registrations (see
+// reregister_global_storage_page_cache_policies()).
+class ScopedStoragePageCache {
+public:
+ explicit ScopedStoragePageCache(StoragePageCache* cache)
+ : _original(ExecEnv::GetInstance()->get_storage_page_cache()) {
+ DCHECK(_original != nullptr);
+ ExecEnv::GetInstance()->set_storage_page_cache(cache);
+ }
+ ~ScopedStoragePageCache() {
+ ExecEnv::GetInstance()->set_storage_page_cache(_original);
+ reregister_global_storage_page_cache_policies();
+ }
+
+ ScopedStoragePageCache(const ScopedStoragePageCache&) = delete;
+ ScopedStoragePageCache& operator=(const ScopedStoragePageCache&) = delete;
+
+private:
+ StoragePageCache* _original;
+};
+
+// Enables SyncPoint processing for the lifetime of the guard; without it, an
+// ASSERT failure would leave processing enabled for subsequent tests.
+class ScopedSyncPointProcessing {
+public:
+ ScopedSyncPointProcessing() {
SyncPoint::get_instance()->enable_processing(); }
+ ~ScopedSyncPointProcessing() {
SyncPoint::get_instance()->disable_processing(); }
+
+ ScopedSyncPointProcessing(const ScopedSyncPointProcessing&) = delete;
+ ScopedSyncPointProcessing& operator=(const ScopedSyncPointProcessing&) =
delete;
+};
+
+} // namespace
+
TEST_F(SegmentFooterCacheTest, TestGetSegmentFooter) {
for (auto segment_ptr : _segments) {
std::shared_ptr<segment_v2::SegmentFooterPB> footer;
@@ -166,13 +233,223 @@ TEST_F(SegmentFooterCacheTest, TestGetSegmentFooter) {
}
}
+// The segment footer is owned by the index page cache and every access goes
through the cache
+// lookup: a miss re-parses the footer and re-inserts it, while other threads
keep reading it
+// (variant sub-path readers call _get_segment_footer() once per path). A
similar concurrent
+// eviction + access workload exposed a heap-use-after-free in the old
weak_ptr implementation
+// (#69811); this test checks that every caller gets a valid, identical footer
regardless of
+// concurrent evictions.
+TEST_F(SegmentFooterCacheTest, ConcurrentGetSegmentFooterWithEviction) {
+ auto segment_ptr = _segments.front();
+ auto* segment_footer_cache =
ExecEnv::GetInstance()->get_storage_page_cache();
+ ASSERT_NE(segment_footer_cache, nullptr);
+ const auto cache_key = segment_ptr->get_segment_footer_cache_key();
+
+ std::shared_ptr<segment_v2::SegmentFooterPB> baseline;
+ ASSERT_TRUE(segment_ptr->_get_segment_footer(baseline, nullptr).ok());
+ ASSERT_NE(baseline, nullptr);
+ const auto expected_num_rows = baseline->num_rows();
+ const auto expected_size = baseline->ByteSizeLong();
+ baseline.reset();
+
+ constexpr int kThreads = 8;
+ constexpr int kIterations = 2000;
+ std::atomic<int> failures {0};
+ std::vector<std::thread> threads;
+ for (int t = 0; t < kThreads; ++t) {
+ threads.emplace_back([&, t] {
+ // Reading the footer from the file needs the thread context a
scanner thread has.
+ SCOPED_ATTACH_TASK(ExecEnv::GetInstance()->orphan_mem_tracker());
+ for (int i = 0; i < kIterations; ++i) {
+ if (t % 2 == 0) {
+ // simulate the LRU eviction of the footer page
+ segment_footer_cache->erase(cache_key,
segment_v2::PageTypePB::INDEX_PAGE);
+ }
+ std::shared_ptr<segment_v2::SegmentFooterPB> footer;
+ Status st = segment_ptr->_get_segment_footer(footer, nullptr);
+ if (!st.ok() || footer == nullptr || footer->num_rows() !=
expected_num_rows ||
+ footer->ByteSizeLong() != expected_size) {
+ ++failures;
+ }
+ }
+ });
+ }
+ for (auto& th : threads) {
+ th.join();
+ }
+ EXPECT_EQ(failures.load(), 0);
+}
+
+// Every footer access goes through StoragePageCache, so repeated accesses must
+// keep the footer page recent in the LRU order. The test runs against a
+// single-shard index cache that fits exactly two footer pages plus half a
page of
+// slack: after A and B are loaded, A is re-accessed (refreshing its recency,
+// served from the cache), and a third footer-sized page is inserted. The
insertion
+// must evict B — the least recently used entry — and keep A, proving that the
+// repeated access refreshed A's position.
+TEST_F(SegmentFooterCacheTest, RepeatedFooterAccessesRefreshLruOrder) {
+ auto segment_a = _segments[0];
+ auto segment_b = _segments[1];
+ const auto key_a = segment_a->get_segment_footer_cache_key();
+ const auto key_b = segment_b->get_segment_footer_cache_key();
+ const StoragePageCache::CacheKey pressure_key("footer_lru_pressure_test",
0, 0);
+
+ // Measure the cache space (charge + handle overhead) of the entries
involved:
+ // footer A, footers A+B, and the pressure page. The pressure footer
object is
+ // reused in the scenario phase, so its charge is identical by
construction.
+ int64_t usage_a = 0;
+ int64_t usage_ab = 0;
+ int64_t usage_abp = 0;
+ std::shared_ptr<segment_v2::SegmentFooterPB> pressure_footer;
+ {
+ StoragePageCache measure_cache(1 << 20, 100, 0, 1);
+ ScopedStoragePageCache scoped_cache(&measure_cache);
+ std::shared_ptr<segment_v2::SegmentFooterPB> footer;
+ ASSERT_TRUE(segment_a->_get_segment_footer(footer, nullptr).ok());
+ usage_a =
measure_cache._get_page_cache(segment_v2::PageTypePB::INDEX_PAGE)->get_usage();
+ ASSERT_TRUE(segment_b->_get_segment_footer(footer, nullptr).ok());
+ usage_ab =
measure_cache._get_page_cache(segment_v2::PageTypePB::INDEX_PAGE)->get_usage();
+ pressure_footer =
std::make_shared<segment_v2::SegmentFooterPB>(*footer);
+ {
+ PageCacheHandle handle;
+ measure_cache.insert(pressure_key, pressure_footer,
pressure_footer->ByteSizeLong(),
+ &handle, segment_v2::PageTypePB::INDEX_PAGE);
+ }
+ usage_abp =
measure_cache._get_page_cache(segment_v2::PageTypePB::INDEX_PAGE)->get_usage();
+ ASSERT_GT(usage_ab, usage_a);
+ ASSERT_GT(usage_abp, usage_ab);
+ }
+ const int64_t weight_a = usage_a;
+ const int64_t weight_b = usage_ab - usage_a;
+ const int64_t weight_pressure = usage_abp - usage_ab;
+
+ // Capacity for exactly two footer pages plus half of A's page as slack.
The
+ // slack guarantees — verified here explicitly — that the pressure insert
both
+ // exceeds the capacity (it must evict) and fits again after evicting
exactly
+ // one entry (it must evict no more than the least recently used page).
+ const int64_t capacity = usage_ab + weight_a / 2;
+ ASSERT_GT(weight_a + weight_b + weight_pressure, capacity);
+ ASSERT_LE(weight_a + weight_pressure, capacity);
+
+ StoragePageCache small_cache(capacity, 100, 0, 1);
+ ScopedStoragePageCache scoped_cache(&small_cache);
+
+ ScopedSyncPointProcessing scoped_sync_point;
+ int parse_count = 0;
+ SyncPoint::CallbackGuard guard;
+ SyncPoint::get_instance()->set_call_back(
+ "Segment::_parse_footer::io_ctx", [&](auto&&) { ++parse_count; },
&guard);
+
+ std::shared_ptr<segment_v2::SegmentFooterPB> footer;
+
+ // Load A and B: both are cache misses and are parsed; A is now the least
+ // recently used footer page.
+ ASSERT_TRUE(segment_a->_get_segment_footer(footer, nullptr).ok());
+ ASSERT_TRUE(segment_b->_get_segment_footer(footer, nullptr).ok());
+ ASSERT_EQ(parse_count, 2);
+
+ // Repeated access of A: served from the cache (no additional parse) and
+ // refreshes A as the most recently used footer page.
+ ASSERT_TRUE(segment_a->_get_segment_footer(footer, nullptr).ok());
+ ASSERT_NE(footer, nullptr);
+ ASSERT_EQ(parse_count, 2);
+
+ // Insert the third footer-sized index page: it does not fit, so the cache
+ // evicts the least recently used page, which must be B — not the
re-accessed A.
+ {
+ PageCacheHandle pressure_handle;
+ small_cache.insert(pressure_key, pressure_footer,
pressure_footer->ByteSizeLong(),
+ &pressure_handle,
segment_v2::PageTypePB::INDEX_PAGE);
+ }
+
+ {
+ PageCacheHandle probe;
+ EXPECT_TRUE(small_cache.lookup(key_a, &probe,
segment_v2::PageTypePB::INDEX_PAGE));
+ }
+ {
+ PageCacheHandle probe;
+ EXPECT_FALSE(small_cache.lookup(key_b, &probe,
segment_v2::PageTypePB::INDEX_PAGE));
+ }
+ {
+ PageCacheHandle probe;
+ EXPECT_TRUE(small_cache.lookup(pressure_key, &probe,
segment_v2::PageTypePB::INDEX_PAGE));
+ }
+ ASSERT_EQ(parse_count, 2);
+}
+
+// VariantExternalMetaReader keeps the footer shared_ptr returned by
+// _get_segment_footer() alive for the reader's whole lifetime. This test pins
the
+// lifetime and cache-accounting semantics under eviction: after the cached
entry
+// is dropped, the held copy stays valid; the next access re-parses the footer
+// (counted via sync point) into a footer that is valid and content-identical;
and
+// the index cache tracker is charged exactly while the entry is cached (footer
+// bytes plus entry overhead), uncharged at eviction, re-charged identically at
+// reload, and unaffected by holders that outlive the entry. This verifies
+// cache-tracker bookkeeping and the retained footer's validity, not actual
heap
+// deallocation.
+TEST_F(SegmentFooterCacheTest,
HeldFooterStaysValidAcrossCacheEvictionAndReload) {
+ auto* segment_footer_cache =
ExecEnv::GetInstance()->get_storage_page_cache();
+ ASSERT_NE(segment_footer_cache, nullptr);
+ auto segment_ptr = _segments.front();
+ const auto cache_key = segment_ptr->get_segment_footer_cache_key();
+ const auto tracker =
segment_footer_cache->mem_tracker(segment_v2::PageTypePB::INDEX_PAGE);
+
+ // Drop whatever Segment::open() during SetUp may have left in the cache.
+ segment_footer_cache->erase(cache_key, segment_v2::PageTypePB::INDEX_PAGE);
+
+ ScopedSyncPointProcessing scoped_sync_point;
+ int parse_count = 0;
+ SyncPoint::CallbackGuard guard;
+ SyncPoint::get_instance()->set_call_back(
+ "Segment::_parse_footer::io_ctx", [&](auto&&) { ++parse_count; },
&guard);
+
+ // First access: the footer is parsed; the cache charges the entry — the
+ // footer's own bytes plus the cache entry overhead (handle + key).
+ const int64_t usage_before = tracker->consumption();
+ std::shared_ptr<segment_v2::SegmentFooterPB> held;
+ ASSERT_TRUE(segment_ptr->_get_segment_footer(held, nullptr).ok());
+ ASSERT_NE(held, nullptr);
+ ASSERT_EQ(parse_count, 1);
+ const auto num_rows = held->num_rows();
+ const auto byte_size = held->ByteSizeLong();
+ const int64_t tracked_bytes = tracker->consumption() - usage_before;
+ ASSERT_GT(tracked_bytes, static_cast<int64_t>(byte_size));
+
+ // Evict the entry: the cache uncharges it while the held copy keeps the
footer
+ // object alive (the reader's lifetime guarantee).
+ segment_footer_cache->erase(cache_key, segment_v2::PageTypePB::INDEX_PAGE);
+ EXPECT_EQ(tracker->consumption(), usage_before);
+
+ // The next access re-parses the footer (a new parse is counted) and
charges
+ // the same measured amount again. Whether the reloaded footer is the same
+ // object as the held one is an implementation detail; what matters is
that it
+ // is valid and content-identical.
+ std::shared_ptr<segment_v2::SegmentFooterPB> reloaded;
+ ASSERT_TRUE(segment_ptr->_get_segment_footer(reloaded, nullptr).ok());
+ ASSERT_NE(reloaded, nullptr);
+ ASSERT_EQ(parse_count, 2);
+ EXPECT_EQ(tracker->consumption() - usage_before, tracked_bytes);
+ EXPECT_EQ(reloaded->num_rows(), num_rows);
+ EXPECT_EQ(reloaded->ByteSizeLong(), byte_size);
+ EXPECT_EQ(held->SerializeAsString(), reloaded->SerializeAsString());
+
+ // The held copy stays valid across the eviction and the reload.
+ EXPECT_EQ(held->num_rows(), num_rows);
+ EXPECT_EQ(held->ByteSizeLong(), byte_size);
+
+ // Releasing the held copy frees that reference without touching the
cache's
+ // accounting (the cached copy stays charged).
+ held.reset();
+ EXPECT_EQ(tracker->consumption() - usage_before, tracked_bytes);
+ ASSERT_EQ(parse_count, 2);
+}
+
TEST_F(SegmentFooterCacheTest, GetSegmentFooterPropagatesIoContext) {
auto segment_ptr = _segments.front();
auto* segment_footer_cache =
ExecEnv::GetInstance()->get_storage_page_cache();
ASSERT_NE(segment_footer_cache, nullptr);
segment_footer_cache->erase(segment_ptr->get_segment_footer_cache_key(),
segment_v2::PageTypePB::INDEX_PAGE);
- segment_ptr->_footer_pb.reset();
TUniqueId query_id;
query_id.hi = 100;
@@ -280,6 +557,10 @@ TEST_F(SegmentFooterCacheTest,
TestGetSegmentFooterCacheKey) {
}
TEST_F(SegmentFooterCacheTest, TestSemgnetFooterPBPage) {
+ // The local cache below registers its policies with CacheManager (BE_TEST
+ // unregister is a no-op); restore the global cache's registrations on
exit,
+ // including on ASSERT failure.
+ ScopedGlobalCachePolicyRegistration registration_guard;
StoragePageCache cache(16 * 2048, 0, 0, 16);
for (auto segment_ptr : _segments) {
std::shared_ptr<segment_v2::SegmentFooterPB> footer;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]