Doris-Breakwater commented on issue #69280: URL: https://github.com/apache/doris/issues/69280#issuecomment-6264250677
Breakwater-GitHub-Analysis-Slot: slot_ea3ea7429492 **Assessment:** High-confidence FE statistics accounting defect, confirmed by source inspection at `5a561bd6f9820ecc6fabb3f4eec3a13003f80a82` and #69028 head `dc2d63c6987ff9d63bc71dabc27497dad18d1bfd`. The analyzed-table path predates #69028; that PR additionally exposes it for load-created, never-analyzed records. This investigation did not run a Doris cluster or reproduce the SQL dynamically. Other release branches have not been verified. **Verified mechanism** - Partial truncate captures the old partitions' base-index row counts, replaces the partitions, and passes those removed counts to `updateUpdatedRows()`. `replayUpdateRowsRecord()` adds their sum to `updatedRows`. Replay repeats this through `Env.replayTruncateTable()` using the persisted `TruncateTableInfo.updateRecords`. Whole-table truncate instead removes the statistics record. See [truncate producer](https://github.com/apache/doris/blob/5a561bd6f9820ecc6fabb3f4eec3a13003f80a82/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java#L3735-L3769), [statistics transition](https://github.com/apache/doris/blob/5a561bd6f9820ecc6fabb3f4eec3a13003f80a82/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/AnalysisManager.java#L1413-L1431), and [replay](https://github.com/apache/doris/blob/5a561bd6f9820ecc6fabb3f4eec3a13003f80a82/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java#L6827-L6837). - Normal partition drop and its replay set `partitionChanged` without adjusting the collected index count or the load-derived delta. The journal record contains no removed-row count. This leaves a stale fallback; it does not necessarily increase the immediately displayed estimate, because the backend-count path can still win. See [drop and replay](https://github.com/apache/doris/blob/5a561bd6f9820ecc6fabb3f4eec3a13003f80a82/fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java#L2114-L2202). - `StatsCalculator.getOlapTableRowCount()` uses collected count plus delta only when the strict backend-count lookup returns unknown. #69028 changes that fallback to `getRowCountWithDeltaRows()` but preserves the truncate increment. Its load-created record starts with collected count/baseline zero, so the reported `0 + (4 + 2 + 1) = 7` follows when this fallback is selected. See [original consumer](https://github.com/apache/doris/blob/5a561bd6f9820ecc6fabb3f4eec3a13003f80a82/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java#L463-L480) and [PR counter handling](https://github.com/apache/doris/blob/dc2d63c6987ff9d63bc71dabc27497dad18d1bfd/fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/AnalysisManager.java#L1392-L1468). - A backend report bypasses the bad fallback for scan row count; it does **not** repair the counter. `computeOlapScan()` still attaches `computeDeltaRowCount()` to statistics, and `FilterEstimation` uses a positive delta when a predicate otherwise estimates zero rows. Include this consumer in the fix and backend-report tests. See [delta attachment](https://github.com/apache/doris/blob/5a561bd6f9820ecc6fabb3f4eec3a13003f80a82/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/StatsCalculator.java#L565-L567) and [filter recovery](https://github.com/apache/doris/blob/5a561bd6f9820ecc6fabb3f4eec3a13003f80a82/fe/fe-core/src/main/java/org/apache/doris/nereids/stats/FilterEstimation.java#L92-L118). **Reproduction and fix details to resolve** 1. Distinguish unmodified master from master plus #69028, which is currently open. On the inspected master revision, loads do not create an absent record; therefore the never-analyzed example does not establish cardinality 7 there. The analyzed-table defect still applies. Also, if the first record is created by ANALYZE after the initial loads, its update-counter baseline can be zero rather than four; the faulty result remains `4 + (3 - 0) = 7`. 2. Use executable syntax `TRUNCATE TABLE t PARTITION (p2)`. For DROP, provide a separate script: a subsequent row must target a surviving partition, and the test must explicitly ensure a surviving index has an unreported count. Merely dropping p2 and loading into already-reported p1 need not enter the fallback. [Strict lookup checks only surviving normal partitions](https://github.com/apache/doris/blob/5a561bd6f9820ecc6fabb3f4eec3a13003f80a82/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java#L1912-L1931); [first-load handling clears the report flag](https://github.com/apache/doris/blob/5a561bd6f9820ecc6fabb3f4eec3a13003f80a82/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java#L2629-L2635). 3. The proposed `removedRows` formula needs a semantic change too. If `updatedRows` still includes truncate's `+2`, subtracting removed rows once gives **5**, not 3: `7 - 2 = 5`, or `4 + (7 - 4) - 2 = 5`. Keep the auto-analyze change counter separately from a genuine load/removal balance, or invalidate cardinality accounting. Preserve a signed net delta: analyzed count 4 with two removed and one loaded requires delta **-1**; clamp the final known estimate rather than clamping that delta to zero. 4. Captured removal counts are asynchronously reported estimates, not guaranteed exact counts, especially with recent loads. If completeness cannot be established, explicit invalidation is safer than claiming an exact subtraction. Simply deleting the record is insufficient with #69028: the next load recreates a zero-baseline record that forgets surviving rows. Any invalidation must also suppress the stale filter-recovery delta. **Suggested next steps and remaining evidence** - Add deterministic FE tests for analyzed and never-analyzed records, truncate, drop, repeated removals, signed negative deltas, backend-report takeover **plus filter estimation**, and journal replay/failover. Check snapshot/baseline consistency and compatibility with older journal/image records. If persisting removal counts for DROP, replay the recorded value rather than recomputing it from a follower's independently reported counts. - Keep cloud coverage explicit. The inspected `TableStatsPB` carries an updated-row count and table version, but no removed-row count; a new balance protocol needs corresponding cloud accounting or invalidation work. An FE-only counter addition does not establish cloud correctness. See [protocol](https://github.com/apache/doris/blob/5a561bd6f9820ecc6fabb3f4eec3a13003f80a82/gensrc/proto/cloud.proto#L1106-L1111). - To close the runtime gap, attach the exact build commit and whether #69028 is applied, local/cloud mode, corrected SQL, synchronous ANALYZE completion, and before/after `SELECT COUNT(*)`, `SHOW TABLE STATS t`, and `EXPLAIN` output. Capture the fallback inputs—record existence, collected index count, baseline, `updatedRows`, and each surviving index's `rowCountReported`—before removal, after the next load, and after backend reporting. These establish which path produced the estimate; a runtime profile is only needed to quantify any resulting query-performance impact. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
