lasdf1234 commented on PR #13386: URL: https://github.com/apache/gravitino/pull/13386#issuecomment-5788322400
> @lasdf1234 Sai, this matches my intent on the first two points. Commit as a compaction accelerator only, and the scheduler as the clock for all four activities, is what I asked for. The ordered pass of compaction → manifest rewrite → snapshot expiration is right, with each step gated by its own effective policy, the recommender and `minIntervalMs`. > > On orphan cleanup, agreed: it runs outside the hot pipeline. I had it as step 4 so a failed compaction's leftovers were collected the same night, and the `olderThan` window makes that impossible anyway, since those files are hours old. Moving it out costs nothing. > > Two conditions on the separate track. The eligibility cadence is a per-table floor rather than a global weekly cron: a table becomes eligible again a set interval after its own last cleanup, so the prefix listings spread across the week instead of the whole estate listing on one night. And the `olderThan` floor is enforced server side, not only validated in the form, since policies can be written over REST. > > Worst-first ordering stays on the compaction queue, where the score is real. Orphan cleanup has no cheap metadata signal to rank on, so its queue is ordered by oldest cleanup first, and it runs within the same maintenance window and concurrency limit, with the remainder carrying to the next night. > > Both tracks write `job_run_meta` rows, including in-process runs, so Jobs → Runs stays the complete history and the maintenance page reads from one place. OK, Mark. I agree with all your points. I will complete the rewriting of the design document as soon as possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
