furquan39 commented on issue #66298: URL: https://github.com/apache/doris/issues/66298#issuecomment-5422544969
**Scale validation of the layout-side mitigation** (since our last update; may be useful to anyone hit by E-3113, and as background for the fix work discussed above). On `doris-4.1.0-rc03` we validated the skew-avoiding layout at production scale: 520M synthetic rows across 25 monthly partitions (one month deliberately loaded at 2x volume), distributed `HASH(document_id)` with 96 buckets and ZSTD, on a table whose VARIANT/string content previously made E-3113 deterministic. Results: zero E-3113 across the full load and all subsequent compaction; tablets uniform at ~207k rows (the 2x month at ~418k rows/tablet, still roughly 2x below the 4 GiB cliff by measured on-disk bytes); compaction kept up throughout (max 5 rowset versions per tablet). Re-proven afterwards on a live, continuously written table at full scale. To be clear, this does not touch the sizing defect itself (compressed-bytes segment sizing vs the uncompressed uint32 limit stands as analyzed); it only confirms that bounding per-tablet data volume via distribution/partitioning is a reliable operator-side mitigation at scale, and gives a tested reference point for what tablet sizes stay safely under the limit for high-compression-ratio data. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
