Doris-Breakwater commented on issue #67934: URL: https://github.com/apache/doris/issues/67934#issuecomment-5659602044
Breakwater-GitHub-Analysis-Slot: slot_1dc63241fb95 ## Initial assessment **Disposition: confirmed configuration inconsistency; the load/write scheduling gap is supported by the code, but the broader claim that this causes general query failures is not yet demonstrated.** No labels are currently applied to the issue. ### Verified facts - The local mirror has `3.1.4-rc01` and `3.1.4-rc02` rather than a plain `v3.1.4` tag. Both release-candidate tags, current `branch-3.1`, and current `master` define the FE default as 95 and the BE default as 90. - In `3.1.4-rc02`, FE defines [`storage_flood_stage_usage_percent = 95`](https://github.com/apache/doris/blob/3.1.4-rc02/fe/fe-common/src/main/java/org/apache/doris/common/Config.java#L884-L892). BE defines the corresponding default as [`90`](https://github.com/apache/doris/blob/3.1.4-rc02/be/src/common/config.cpp#L762-L767). Both byte thresholds default to 1 GiB. - FE's [`DiskInfo.exceedLimit(true)`](https://github.com/apache/doris/blob/3.1.4-rc02/fe/fe-core/src/main/java/org/apache/doris/catalog/DiskInfo.java#L167-L186) requires both `available < 1 GiB` and `used > 95%`. BE's [`DataDir::reach_capacity_limit`](https://github.com/apache/doris/blob/3.1.4-rc02/be/src/olap/data_dir.cpp#L922-L932) requires both `available-after-write <= 1 GiB` and `used-after-write >= 90%`. - FE uses its flood-stage check when preparing OLAP load destinations in [`OlapTableSink`](https://github.com/apache/doris/blob/3.1.4-rc02/fe/fe-core/src/main/java/org/apache/doris/planner/OlapTableSink.java#L767-L772), while BE independently enforces its local threshold on disk-consuming operations. Therefore, with effective defaults and a path at 92% usage **and at most about 1 GiB free**, FE can admit a load that BE later rejects. Usage of 92% alone is not sufficient because both components use a two-part capacity condition. - The equality boundaries also differ (`>`/`<` in FE versus `>=`/`<=` in BE), so merely making the numeric defaults equal would still leave exact-threshold behavior inconsistent. - [`Backend.isQueryAvailable()`](https://github.com/apache/doris/blob/3.1.4-rc02/fe/fe-core/src/main/java/org/apache/doris/system/Backend.java#L542-L550) checks liveness and administrative query-disable/shutdown flags, not disk capacity. Thus the observation that the BE can remain a query candidate is correct. However, `reach_capacity_limit()` is a per-operation guard, not a persistent global flood-state transition. Ordinary read queries do not inherently require new local data files; failures are expected only on disk-consuming paths such as load/segment finalization, migration/clone/restore, or a query that actually spills to that path. ### History and likely scope The original disk-flood-stage implementation introduced FE and BE defaults together at 95 in [commit `7e981b2`](https://github.com/apache/doris/commit/7e981b2b14d08cacc6726c962249395c69f6d2e9). BE alone changed from 95 to 90 in [commit `206a711`](https://github.com/apache/doris/commit/206a711f9b77648f0d6e34bf7e6f87f2f642bea3); that same commit adjusted trash-sweep guard space, but neither the configuration comment nor tests explain an intended FE/BE policy split. This supports treating the mismatch as a valid maintenance issue, while the reason for the 2021 BE change remains unverified. ### Information still needed To establish the claimed user-visible failure rather than only the code-level mismatch, please provide: 1. The exact FE and BE build strings or Git commit hashes for the reported `v3.1.4` deployment. 2. The effective runtime values of both flood-stage settings on FE and on every relevant BE; they are mutable and may differ from compiled defaults. 3. A black-box reproduction using unchanged binaries, including the exact SQL/load/restore operation. The current reproduction proposes hard-coding a value and artificially enlarging the byte threshold, so it does not establish what happens with shipped defaults. 4. The affected data-path total/available capacity and path hash at the failure time, plus the corresponding BE `DISK_REACH_CAPACITY_LIMIT` message and FE/coordinator error. If the failed operation was a read query, include whether spilling was enabled/observed and its query profile. ### Recommended next steps 1. Confirm whether BE's 90% value is an intentional safety floor. If it is, make FE's load/restore preflight at least as strict (or propagate the BE's effective threshold to FE) rather than relaxing BE to 95 without a separate disk-safety assessment. 2. If independent component policies are intentional, document their distinct roles explicitly and avoid presenting the same names as equivalent cluster-wide settings. FE should still avoid scheduling writes that BE will predictably reject from the same reported disk state. 3. Align the comparison boundaries as part of any fix and add FE/BE boundary tests covering just below, exactly at, and just above both percentage and remaining-byte thresholds. Add an integration test showing that FE preflight and BE enforcement agree for a load. 4. Do not make general query eligibility depend on disk flood stage as part of this issue without a separate design decision; that could unnecessarily remove readable replicas. Limit the fix to admission of disk-consuming operations unless a reproducible non-spill read-query failure shows otherwise. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
