morningman opened a new pull request, #70727: URL: https://github.com/apache/doris/pull/70727
### Background Lance readers share metadata and index caches across datasets, and can use a persistent Foyer cache for immutable data-file ranges. The BE creates these caches through the native lance-c library and retains their state across queries. ### What problem does this PR solve? Issue Number: None Related PR: #68613 (introduced the Foyer patch), #69581 (integrated the shared native caches into BE readers) PR Type: Bugfix Problem Summary: The default 100 GiB data cache retains about 51,200 block-file descriptors, and caches can mix data from different storage endpoints that expose the same bucket and path. Use a single disk file and storage namespaces across the data, metadata and index caches, while retaining their shared capacity limits. Add a native cache-clear operation and tests for these paths; BE integration of that operation follows the rebuilt precompiled package. | Create the default 100 GiB cache | Before | After | |---|---:|---:| | Additional process descriptors, measured locally | 51,206 | 7 | ### Details - **Change map:** +926/-52 across two thirdparty files, classified as build/CI by the path profile. Inside the patch, the native change is +331/-18 implementation, +460/-5 tests and +15 documentation; patch metadata accounts for the remaining representation changes. The shell harness adds 24 lines. Read the single-file device and migration first, then the shared cache namespace adapter, then cache clearing and its tests. - **Frequency:** Always when the default data cache initializes; endpoint aliasing requires one session to open the same dataset URI on different storage endpoints. - **Root cause:** FsDevice retains one file per 2 MiB engine block. Data keys omit storage options, while native metadata/index caches use dataset URIs that also omit the endpoint. - **Why existing tests missed it:** BE session-manager and reader tests use small cache capacities and one storage origin. The Lance catalog regression uses one origin. Existing native range tests omit default-capacity descriptor counts, distinct endpoint configurations and cache-clear lifecycle checks. - **Fix:** Share one FileDevice across disk blocks and namespace every cache tier before metadata loading. Keep one bounded physical backend per memory tier, preserve cumulative statistics, and expose cache clearing without invalidating datasets. Use eviction to release Foyer's weighted usage correctly. - **Other paths and branches:** Isolation also applies with the data cache disabled and to restored handles. Both branch-4.1 and branch-4.2 retain the descriptor/key defects; their dependency baselines differ. Older branches were not checked. No release branch is changed here. - **Behavior change:** The disk layout/key version changes, so upgrade starts a cold data cache. After successful device initialization, remove only the old device's reserved block files. Cache capacities and configuration defaults remain the same. ### Testing - Local macOS arm64, Rust 1.98.1: the extraction/refresh/failure-marker harness and 30 native tests passed from the original archive plus the repository patch: 17 cache tests, 4 shared-session C API tests, 2 data-cache tests, 6 prewarm tests and 1 restored-handle test. - The default-capacity probe measured 51,206 additional descriptors before and 7 after. Targeted reverts of data and metadata isolation both failed their assertions; the latter read 5 rows where the second endpoint contains 9. ### Notes for Reviewers The upstream pin and lockfile stay unchanged. Merge this native prerequisite before updating precompiled packages and BE consumers. Query/workload-group memory accounting, a complete scanner budget, and BE `/api/clear_cache/all` wiring remain follow-ups; this PR does not implement them. Native tests require `LANCE_C_RUN_NATIVE_TESTS=ON` in the supplied harness and are not automatically run by ordinary BE PR jobs. ### Release note Lance caches use a bounded number of file descriptors and isolate storage configurations while retaining their configured capacities. Upgrading starts a cold disk cache and removes legacy block files after the new cache initializes. As before, one session/process must own the cache directory. ### Check List (For Author) - Behavior changed: Yes (see Details) - Does this need documentation: Yes (document the cold-cache upgrade in apache/doris-website when the rebuilt package is adopted; native API documentation is included here) - Pre-submission status: Partially verified: Linux builds, Linux/glibc 2.17 artifact validation, and Doris BE integration with the rebuilt precompiled package were not run ### AI Disclosure - Tool: Codex (GPT-6) - Generated: native implementation, tests, and this description - Human review: not stated -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
