matanper commented on issue #70771: URL: https://github.com/apache/doris/issues/70771#issuecomment-7027415556
Thanks for the review. Some follow-up details: **Table properties** (DUPLICATE KEY, cloud mode): `compaction_policy = time_series`, `time_series_compaction_goal_size_mbytes = 1024`, `time_series_compaction_file_count_threshold = 1000`, `time_series_compaction_time_threshold_seconds = 3600`, `time_series_compaction_empty_rowsets_threshold = 5`, `time_series_compaction_level_threshold = 2`. The threshold was raised from 1 to 2 with `ALTER TABLE ... SET`. The table was under continuous load at the time. **One affected tablet, read from the meta-service** (cumulative point 6315): | versions | rowsets | segments | compaction_level | created | |---|---|---|---|---| | `[6315-6342]` … `[6427-6450]` | 5 | 1 each | 0 | within about a minute of the ALTER | | `[6451-…]` … `[11711-11714]` | 203 | 1 each | 1 | after the ALTER | | `[11715-11715]` | 1 | 0 (empty) | 0 | a late load | All rowsets above the point are NONOVERLAPPING, and none has a delete predicate. `/api/compaction/show` shows `last cumulative status: [E-2000]` and no cumulative success. **Creation history:** we have not reconstructed it. Below threshold 2 the cumulative point normally moves past each cumulative output, so we can't yet say why these level-0 outputs stayed at or above it. On other tablets of the same table, the level-0 merged rowsets at the point were created up to three days before the ALTER. The blocked state itself is consistent across tablets: on every affected tablet, the first rowset above the point is a merged level-0 rowset from before the ALTER, followed by level-1 rowsets. **Patch:** we treat a merged (start != end), non-empty, stored level-0 rowset as level 1 in the level-2 scan of `pick_input_rowsets`, in `calc_cumulative_compaction_score`'s initial classification and level-2 scan, and in `get_compaction_level`. The output is labelled one level above its *lowest* effective input, so a level-1 pick that starts at such a rowset and includes new loads still writes level 1. Conditions 2 and 3 skip a lone non-overlapping rowset and fall through. Unit tests cover: - the mixed prefix with no tail, an aged empty tail and an aged non-empty tail, asserting the selected versions; - output level 2, plus cumulative-point movement for a non-empty output and no movement for an empty one; - a mixed pick writing level 1; - a lone load past the time threshold; - condition 2 with file-count threshold 1; - an overlapping lone load still being picked. With the patch all 19 time-series policy tests pass; without it the new cases fail. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
