laserninja commented on PR #12937: URL: https://github.com/apache/gravitino/pull/12937#issuecomment-5564454296
@lasdf1234 done - #12942, following the convention that every design doc lands as its own PR. It covers why manifest bloat is distinct from data-file bloat and why neither `rewrite_data_files` nor `expire_snapshots` addresses it, the four approaches considered with the reasons the Iceberg Java action lost to the Spark SQL procedure, the parameter surface and its validation, and the `JobManager.replacePlaceholder` behaviour that shapes the job's input handling. The follow-up section sequences the manifest metrics and the policy that would fire on them, aligned with table compaction per your comment on #11700 - `DataRequirement.TABLE_STATISTICS`, and any last-run signal in `statistic_meta` as a `custom-` statistic rather than bespoke state. Two things changed on this PR while writing it: - `rewrite_manifests` takes a third in-parameter, `spec_id`, which was not exposed. It is what makes the partition-spec half of #11196's motivation reachable, so it is now a `jobConf` key, validated as a non-negative integer and placeholder-filtered like `use_caching`. - The doc records one gap this PR does not close: no Spark-backed integration test, unlike the two sibling `*WithSpark` tests. Happy to add it here if you would prefer that over a follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
