Doris-Breakwater commented on issue #69360: URL: https://github.com/apache/doris/issues/69360#issuecomment-6307020021
Thanks for the detailed report. **There is a relevant synchronous-clear fix missing from the public 4.1.2 source, but the supplied logs do not yet establish the cause of the warnings before clearing, or their connection to the Spark failure.** I checked the issue metadata (currently open, no labels or comments) and traced the public versioned source without modifying Doris code. ### Verified source findings 1. **The statement that 4.1.2 includes #64578 needs correction.** Public tag `4.1.2` resolves to `aec169d20256a788de448f46737b5fb53ee3b4e3`. Its HTTP `op=clear&sync=true` path calls `FileCacheFactory::clear_file_caches(true)` → `BlockFileCache::clear_file_cache_directly()` → `FSFileCacheStorage::clear()`. This deletes cache directories and clears the manager's maps/FD entries without preserving the normal lifecycle of blocks still held by readers. Existing shared `FileBlock` references can survive that operation. See the [4.1.2 factory](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53ee3b4e3/be/src/io/cache/block_file_cache_factory.cpp#L246) and [direct-clear implementation](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53ee3b4e3/be/src/io/cache/block_file_cache.cpp#L2197). [#64578](https://github.com/apache/doris/pull/64578), backported to 4.1 as [#64692](https://github.com/apache/doris/pull/64692) / `b317c8e2be3b837f0e2ca72aa919a8bc6964993a`, replaces that wipe with block-aware removal: busy blocks are marked for deletion and retained until their holders release them. Git ancestry and source inspection confirm that this backport is absent from public `4.1.2` and present in public `4.1.3` and `4.1.4`. Thus clearing under active readers is a concrete candidate for **post-clear** warnings; it does not explain warnings that preceded the clear. 2. **Remote fallback already exists in 4.1.2.** `FSFileCacheStorage::read()` emits the quoted warning when both local-path attempts fail. In `CachedRemoteFileReader::read_at_impl()`, a failed local read of a `DOWNLOADED` block falls back to the remote reader. That path also counts local `NOT_FOUND` events and schedules hash invalidation after the read loop. However, an unsuccessful remote read returns its error before that final invalidation. The direct-cache fast path first breaks out on a failed local read and proceeds through the ordinary cache/remote path; it does not itself discard its retained block reference. Consequently, repeated warnings alone do not prove that remote fallback is missing or that every query failed. See the [versioned reader](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53ee3b4e3/be/src/io/cache/cached_remote_file_reader.cpp#L340) and [fallback/self-healing code](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53 ee3b4e3/be/src/io/cache/cached_remote_file_reader.cpp#L497). 3. **The filenames do not identify a Disposable eviction bug.** V3 uses `<hash>_0/<offset>`; V2 uses an expiration-based directory and a cache-type suffix where applicable. Normal, non-expiring V2 and V3 paths can therefore be identical, as in the examples. A V2 candidate ending in `_disposable` reflects the requested block's type; it does not prove the file previously existed there or why it disappeared. See the [path builders](https://github.com/apache/doris/blob/aec169d20256a788de448f46737b5fb53ee3b4e3/be/src/io/cache/fs_file_cache_storage.cpp#L330). The reported 5.5 GB after clearing establishes neither historical cache occupancy nor filesystem/inode headroom at failure time. It gives no reason to increase the 100 GB limit. Also, [#41280](https://github.com/apache/doris/issues/41280) describes orphaned disk data and space leakage; that is useful consistency-check context, not evidence for the cause of these missing files. ### Evidence needed to confirm the mechanism - Full BE build/version output from **all three BEs**, including commit IDs and whether the binaries contain downstream patches; Spark and Doris Spark Connector versions as well. - A time-aligned BE INFO/WARNING window covering the first warning **before** clearing, the clear start/completion, and recurrence afterward. Include the HTTP response and identify which BEs were cleared: this endpoint acts on the addressed BE, not the whole cluster. Group recurrence by BE and hash/offset to distinguish the same stale blocks from newly affected blocks. - Adjacent `Read data failed from file cache downloaded by others`, self-healing, remote-storage, and query-error messages; the delta of the affected BE's bvar counter `cached_remote_reader_self_heal_on_not_found` over that window. Include effective `enable_read_cache_file_directly`, relevant cache/TTL settings, and sanitized table DDL. - The complete Spark exception chain and failing operation, request/query or scan identifiers, and matching BE errors. The connector's bare `NOT_FOUND` with a null message cannot establish that a local cache miss caused it. - Reproduction SQL/DDL, workload concurrency, restart/upgrade/clear timeline, and whether another process or cleanup job shares/removes cache files. Capture `df -h`, `df -i`, mount details, and cache metrics at recurrence, with credentials and sensitive paths redacted. ### Suggested next steps Validate on a supported build containing #64692; the checked public 4.1.3/4.1.4 tags contain this specific fix. Treat this as removal of a verified clear-path hazard, not a guarantee that every reported symptom is resolved. Until then, avoid repeated synchronous clears under active workload. If clearing is necessary, the 4.1.2 asynchronous clear path (`sync=false`) preserves busy-block lifetimes, although reclamation is deferred. In a test environment, compare the same workload with no clear and with clear during active reads, on 4.1.2 and a build containing #64692. If warnings persist without clearing, or remain on the fixed build, investigate the first disappearance using the evidence above rather than attributing it to capacity or Disposable eviction. Separately correlate the Spark failure with its server-side request. No production reproduction or runtime tests were available for this analysis. Breakwater-GitHub-Analysis-Slot: slot_dcdc39add527 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
