smengcl opened a new pull request, #11597:
URL: https://github.com/apache/ozone/pull/11597
Generated-by: Claude Code (Opus 5.5)
## What changes were proposed in this pull request?
Follow-up of HDDS-17079 (#11576). This branch is stacked on #11576; only the
last commit is new until #11576 is merged.
Building the snapshot version graph at OM startup needs only `snapshotId`,
`previousSnapshotId`, `version` and each version's `previousSnapshotVersion`.
`loadWithoutSstFiles` skips SST file entries, but still reads the whole file,
because each `previousSnapshotVersion` is stored inside `versionSstFileInfos`
next to its SST file list. With large YAMLs (about 9 MB median, 175 GB per OM
on a test cluster with about 19,350 snapshots) startup is disk bound on this
read.
This PR writes a derived top level map, `previousSnapshotVersions` (version
to previousSnapshotVersion), into the YAML. Keys are emitted in sorted order,
so it precedes `versionSstFileInfos` together with the ids and version. When
the summary is present, `loadWithoutSstFiles` returns as soon as it reaches the
`versionSstFileInfos` key. Files without it are streamed in full as before.
```yaml
previousSnapshotId: 5f796c35-...
previousSnapshotVersions: {0: 0, 1: 3}
snapshotId: 4a7e4e9c-...
version: 1
versionSstFileInfos:
...
```
Compatibility:
- The full loader, unchanged here and in released versions, reads keys by
name and ignores unknown keys, so older OMs load new files.
- The summary is left out while the checksum is computed, so the checksum
does not change with or without it. Old and new code verify both layouts.
- An older OM rewriting a file drops the summary, and the file falls back to
the full stream.
- No layout feature is needed: the change is additive and readable by every
released version.
For files with the summary, startup no longer validates the SST file lists;
malformed content there surfaces on the next full load of that snapshot.
Existing YAMLs keep the old layout until they are rewritten.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17098
## How was this patch tested?
- New and updated tests in `TestOmSnapshotLocalDataYaml`: summary written
before the SST file lists, early stop (a file cut after the
`versionSstFileInfos` key still loads when it has the summary and fails without
it), old layout falls back to the full stream, checksum is identical with and
without the summary, and re-saving an old layout file adds the summary back.
- `TestOmSnapshotLocalDataManager`, `TestSnapshotDefragService`,
`TestOmSnapshotManager`, `TestOMSnapshotPurgeRequestAndResponse` pass.
- Manually loaded a file written by this patch with the 2.2.1
`OmSnapshotLocalDataYaml`: it loads and verifies its checksum, and its rewrite
differs only by the missing summary line, which this patch loads and verifies.
- On a warm 14 MB synthetic YAML, `loadWithoutSstFiles` takes 0.5 ms with
the summary and about 150 to 200 ms without it.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
