xylaaaaa commented on issue #69884: URL: https://github.com/apache/doris/issues/69884#issuecomment-6569084501
## Basic statistics follow-up — 2026-11-02 Implemented and pushed to draft PR [#69904](https://github.com/apache/doris/pull/69904): - [598756e03e](https://github.com/apache/doris/commit/598756e03e): ordinary pinned queries consume snapshot statistics, reuse the FE size estimate, and forward snapshots through the Hive sibling gateway. - [cc12b56819](https://github.com/apache/doris/commit/cc12b56819): Delta caches logical rows by user/path/table UUID/version, subtracts DV cardinality, and uses same-version CRC bytes while bounded background collection runs. Missing file row statistics remain unknown; CRC contains no row count. Exact COUNT remains independent of estimates. Cache tests cover concurrent single-flight, user/version/table identity isolation, invalidation/close, late completion, retry, missing/invalid stats and background authentication/classloader handling. Eight concurrent scans share one real cached checkpoint/DV snapshot; early-closing iterators do not affect the other scans. Validation on the final source: - Full FE build and Checkstyle passed; 87 Delta + 2 metastore tests passed, plus broad 267 Hive / 114 FE core tests and focused 22 Paimon / 7 Iceberg compatibility tests. - Seven p0 suites passed against the final FE build. The new statistics suite covers filesystem/HMS/Hive cardinalities, DV, V2 checkpoints, refresh, empty tables and disabled collection. Its 21 expected query tags were generated with the official script and then independently verified. - Million-file statistics in a 512 MiB JVM: cold lookup **0.491 ms**, completion **2.170 s**, sampled heap peak **100,255,200 bytes (~96 MiB)**. This fixture measures metadata collection, not Parquet data scanning. - All **1,227** retained MinIO objects still match the preserved path/size/SHA-256 manifest. FE/BE and the three temporarily started containers have been stopped; fixtures and metadata are retained. Connector README, local design/tasklist and paired EN/ZH website patch are updated. The checklist is **63/67**; the issue body now reflects verified work and remaining gates. The previous TPC-DS +26.1% measurement predates this change. A full post-statistics benchmark has not been rerun, and the entire gap has not been attributed to statistics. Column NDV/histograms remain later work; performance acceptance, cloud/Kerberos environments, website PR/build and master integration/final acceptance remain open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
