morningman opened a new pull request, #70727:
URL: https://github.com/apache/doris/pull/70727

   ### Background
   
   Lance readers share metadata and index caches across datasets, and can use a 
persistent Foyer cache for immutable data-file ranges. The BE creates these 
caches through the native lance-c library and retains their state across 
queries.
   
   ### What problem does this PR solve?
   
   Issue Number: None
   Related PR: #68613 (introduced the Foyer patch), #69581 (integrated the 
shared native caches into BE readers)
   PR Type: Bugfix
   
   Problem Summary:
   
   The default 100 GiB data cache retains about 51,200 block-file descriptors, 
and caches can mix data from different storage endpoints that expose the same 
bucket and path. Use a single disk file and storage namespaces across the data, 
metadata and index caches, while retaining their shared capacity limits. Add a 
native cache-clear operation and tests for these paths; BE integration of that 
operation follows the rebuilt precompiled package.
   
   | Create the default 100 GiB cache | Before | After |
   |---|---:|---:|
   | Additional process descriptors, measured locally | 51,206 | 7 |
   
   ### Details
   
   - **Change map:** +926/-52 across two thirdparty files, classified as 
build/CI by the path profile. Inside the patch, the native change is +331/-18 
implementation, +460/-5 tests and +15 documentation; patch metadata accounts 
for the remaining representation changes. The shell harness adds 24 lines. Read 
the single-file device and migration first, then the shared cache namespace 
adapter, then cache clearing and its tests.
   - **Frequency:** Always when the default data cache initializes; endpoint 
aliasing requires one session to open the same dataset URI on different storage 
endpoints.
   - **Root cause:** FsDevice retains one file per 2 MiB engine block. Data 
keys omit storage options, while native metadata/index caches use dataset URIs 
that also omit the endpoint.
   - **Why existing tests missed it:** BE session-manager and reader tests use 
small cache capacities and one storage origin. The Lance catalog regression 
uses one origin. Existing native range tests omit default-capacity descriptor 
counts, distinct endpoint configurations and cache-clear lifecycle checks.
   - **Fix:** Share one FileDevice across disk blocks and namespace every cache 
tier before metadata loading. Keep one bounded physical backend per memory 
tier, preserve cumulative statistics, and expose cache clearing without 
invalidating datasets. Use eviction to release Foyer's weighted usage correctly.
   - **Other paths and branches:** Isolation also applies with the data cache 
disabled and to restored handles. Both branch-4.1 and branch-4.2 retain the 
descriptor/key defects; their dependency baselines differ. Older branches were 
not checked. No release branch is changed here.
   - **Behavior change:** The disk layout/key version changes, so upgrade 
starts a cold data cache. After successful device initialization, remove only 
the old device's reserved block files. Cache capacities and configuration 
defaults remain the same.
   
   ### Testing
   
   - Local macOS arm64, Rust 1.98.1: the extraction/refresh/failure-marker 
harness and 30 native tests passed from the original archive plus the 
repository patch: 17 cache tests, 4 shared-session C API tests, 2 data-cache 
tests, 6 prewarm tests and 1 restored-handle test.
   - The default-capacity probe measured 51,206 additional descriptors before 
and 7 after. Targeted reverts of data and metadata isolation both failed their 
assertions; the latter read 5 rows where the second endpoint contains 9.
   
   ### Notes for Reviewers
   
   The upstream pin and lockfile stay unchanged. Merge this native prerequisite 
before updating precompiled packages and BE consumers. Query/workload-group 
memory accounting, a complete scanner budget, and BE `/api/clear_cache/all` 
wiring remain follow-ups; this PR does not implement them. Native tests require 
`LANCE_C_RUN_NATIVE_TESTS=ON` in the supplied harness and are not automatically 
run by ordinary BE PR jobs.
   
   ### Release note
   
   Lance caches use a bounded number of file descriptors and isolate storage 
configurations while retaining their configured capacities. Upgrading starts a 
cold disk cache and removes legacy block files after the new cache initializes. 
As before, one session/process must own the cache directory.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes (see Details)
   - Does this need documentation: Yes (document the cold-cache upgrade in 
apache/doris-website when the rebuilt package is adopted; native API 
documentation is included here)
   - Pre-submission status: Partially verified: Linux builds, Linux/glibc 2.17 
artifact validation, and Doris BE integration with the rebuilt precompiled 
package were not run
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6)
   - Generated: native implementation, tests, and this description
   - Human review: not stated
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to