stinger1206 opened a new pull request, #70638:
URL: https://github.com/apache/doris/pull/70638

   Cherry-pick of #67720 to branch-4.2. The change is identical to the merged 
master commit; the only adaptation is that the test file keeps this branch's 
`#include "storage/segment/segment_writer.h"` (master's later rename to 
`vertical_segment_writer.h` is not part of this branch).
   
   ### What problem does this PR solve?
   
   Issue Number: close #67719
   
   Related PR: #67720
   
   Problem Summary:
   
   `Segment::_footer_pb` (`std::weak_ptr<SegmentFooterPB>`) was a second footer
   access path outside `StoragePageCache`:
   
   - `_get_segment_footer()` reads it via `lock()` and rebinds it
     (`_footer_pb = footer_pb_shared`) from query/compaction threads whenever 
the
     cached footer expires, while `get_metadata_size()` also accesses the same
     non-atomic `std::weak_ptr`. Concurrent assignment and `lock()` on the same
     weak_ptr, if these paths overlap, is a data race and therefore undefined
     behavior.
   - Moreover, the weak_ptr fast path could bypass the cache lookup whenever the
     footer object was still alive — including while the cache entry was 
present —
     so the cache could not observe real footer accesses and its LRU order could
     diverge from actual usage.
   
   In production (4.1.3) this crash family manifested as intermittent BE
   SIGSEGVs, with innermost frames such as `SegmentFooterPB::ByteSizeLong`
   (`Segment::get_metadata_size`) and `~SegmentFooterPB` under
   `LRUCache::prune_if`, healed by BE restart with segment files byte-identical.
   See #67719 for a TSAN micro-repro of the unsafe `std::weak_ptr` access
   pattern.
   
   Fix (identical to #67720):
   
   - remove `_footer_pb` entirely;
   - `_get_segment_footer()` always accesses the footer through
     `StoragePageCache` (single access path: lookup → parse & insert on miss), 
so
     eviction and LRU recency are handled consistently;
   - `get_metadata_size()` no longer reads the footer: footer memory is tracked
     by `StoragePageCache`'s own memory tracking, and doing a lookup there only
     for accounting would artificially refresh LRU recency on every
     metadata-size sweep without any real query access;
   - update the stale `StoragePageCache::insert` comment to instead recommend
     going through cache lookup for every access;
   - mark `Segment::_parse_footer()` and its helper `_write_error_file()` const;
   - tests: concurrent eviction regression (adapted from #69811), LRU recency of
     repeated footer accesses, and held-footer lifetime/cache-accounting across
     eviction and reload.
   
   ### Release note
   
   Fix unsafe Segment footer caching that could cause BE crashes and bypass
   StoragePageCache LRU accounting.
   
   ### Check List (For Author)
   
   - Test <!-- At least one of them must be included. -->
       - [x] Regression test
       - [x] Unit Test
       - [x] Manual test (add detailed scripts or steps below)
       - [ ] No need to test or manual test. Explain why:
           - [ ] This is a refactor/code format and no logic has been changed.
           - [ ] Previous test can cover this change.
           - [ ] No code files have been changed.
           - [ ] Other reason <!-- Add your reason?  -->
   
     Unit test: the same `SegmentFooterCacheTest` suite as #67720, including
     `ConcurrentGetSegmentFooterWithEviction` (8 threads call
     `_get_segment_footer()` while half of them keep erasing the footer's cache
     page), `RepeatedFooterAccessesRefreshLruOrder`, and
     `HeldFooterStaysValidAcrossCacheEvictionAndReload`. All 8 tests passed
     locally under ASAN on master (#67720); CI re-runs them on this branch.
   
     Manual test: the cache-only implementation is deployed on a production
     canary BE and the crash family has not recurred under the
     previously-crashing workload.
   
   - Behavior changed:
       - [x] No.
       - [ ] Yes. <!-- Explain the behavior change -->
   
   - Does this need documentation?
       - [x] No.
       - [ ] Yes. <!-- Add document PR link here. eg: 
https://github.com/apache/doris-website/pull/1214 -->
   
   ### Check List (For Reviewer who merge this PR)
   
   - [ ] Confirm the release note
   - [ ] Confirm test cases
   - [ ] Confirm document
   - [ ] Add branch pick label <!-- Add branch pick label that this PR should 
merge into -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to