SaketaChalamchala opened a new pull request, #11651: URL: https://github.com/apache/ozone/pull/11651
## What changes were proposed in this pull request? Developed with the help of Cursor AI. This PR is final orchestration piece that integrates the optimized snapshot diff pipeline (DAG/full sequential readers, SnapDiffJobStore, MergeJoinSnapDiffWriter) into live OM job execution behind a feature flag, with legacy fallback, FSO dependency-ordered report control, per-stage progress, and clearer CLI/RPC behavior.. See design doc: ([HDDS-9154](https://issues.apache.org/jira/browse/HDDS-9154)). **Approach** - **`ozone.om.snapshot.diff.optimized.enabled` (default `false`):** when enabled and native DAG diff is viable, `SnapshotDiffManager.generateSnapshotDiffReport` runs file/dir scans via `FullDiffSequentialReader` or `DagDiffSequentialReader`, then `MergeJoinSnapDiffWriter.writeReport`. Otherwise the existing legacy path is unchanged. - **Submit / job metadata:** optional ordered on `SubmitSnapshotDiffRequest` and `SnapshotDiffJobProto` (CLI `--ordered`). Stored on `SnapshotDiffJob` and passed through the async report closure. - **FSO ordered resubmit:** if the client requests `ordered=true`, the stored job had `ordered=false`, and status is `QUEUED`|`IN_PROGRESS`|`DONE`, OM creates a new job and reruns instead of reusing the cached unordered result. - **Progress:** per sub-status `keysProcessedPct` via `DoubleConsumer` → `updateProgress` (same model as object-id map stages). Scan readers report in-stage (10% steps / slice milestones); report generation uses FSO slices (classify 0–50%, resolve 50–90% or 50–100%, order 90–100% when ordered). Denominators come from `SnapDiffJobStore` counters and table/SST estimates. - **Observability:** `INFO` logs for scan file/dir and report sub-stages (classify, delete filter/resolve, path resolve, ordering). DONE SnapshotDiffResponse.toString() prints dependency ordering via existing reason (FSO honors job flag; OBS always reported as ordered). - **Robustness:** `updateJobStatusToFailed(jobKey, jobId, reason)` skips updates when a superseding job replaced the table entry (mirrors done-path jobId guard). **Compatibility** - New proto fields are optional; older clients/OM ignore ordered. - Optimized path requires native libs for DAG mode; full-diff optimized path still allowed when non-native is forced. - Legacy behavior remains when the optimized flag is off or DAG native diff is unavailable. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-15397 ## How was this patch tested? Unit Tests added, Acceptance and Integration tests (TBD) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
