swamirishi commented on PR #8981:
URL: https://github.com/apache/ozone/pull/8981#issuecomment-3237663147
This commit makes significant changes to the way Ozone's snapshot local data
tracks and serializes SST (Sorted String Table) file information. Here’s a
summary of all changes:
---
### 1. **Refactoring SST File Tracking Structure**
- **Old Approach:** SST files (both uncompacted and compacted) were tracked
using maps:
- `Map<String, Set<String>>` for uncompacted SST file lists
- `Map<Integer, Map<String, Set<String>>>` for compacted SST file lists
- **New Approach:**
- Introduces a new class `SstFileInfo` to encapsulate SST file metadata:
file name, start key, end key, column family.
- Compacted/uncompacted SST files are now tracked as lists of
`SstFileInfo` objects grouped by snapshot version in a new `VersionMeta` class.
- `VersionMeta` stores:
- `previousSnapshotVersion`
- `List<SstFileInfo> sstFiles`
- The maps are replaced by `Map<Integer, VersionMeta> versionSstFileInfos`.
### 2. **Class and Code Structure Changes**
- **New Class:**
- `SstFileInfo` is added, encapsulating SST file details and supporting
deep copy, equality, and string representation.
- `VersionMeta` is added as a static inner class of `OmSnapshotLocalData`,
representing a snapshot version's SST file metadata.
- **Existing Classes Modified:**
- `CompactionFileInfo` now extends `SstFileInfo` instead of keeping its
own fields for SST file metadata.
- Constructors and methods throughout `OmSnapshotLocalData`,
`OmSnapshotLocalDataYaml`, and related test classes are updated to use the new
data structures.
### 3. **YAML Serialization/Deserialization Improvements**
- **Custom YAML tags and representers** for `OmSnapshotLocalDataYaml`,
`VersionMeta`, and `SstFileInfo` are introduced for more precise serialization.
- The code now omits null properties in YAML output.
- Reading/writing YAML files for snapshot data now uses the new structures
and custom serialization logic.
### 4. **API and Method Changes**
- Methods for getting/setting uncompacted and compacted SST files are
replaced with versions that use `VersionMeta` and `SstFileInfo`.
- Methods like `addUncompactedSSTFileList` and `addCompactedSSTFileList` are
replaced with `addVersionSSTFileInfos`.
### 5. **Test Updates**
- All relevant tests (`TestOmSnapshotLocalDataYaml`,
`TestOmSnapshotManager`) are updated to use the new data structures.
- Mocks for `LiveFileMetaData` now include start/end keys and column family.
- Assertions now verify the new model (`VersionMeta`, `SstFileInfo`) rather
than simple sets/maps.
### 6. **Constants and Field Additions**
- New string constants added to `OzoneConsts` for the new YAML field names.
- New fields added for previous snapshot IDs and more detailed SST file
tracking.
### 7. **Code Clean-Up**
- Removes unused imports and code related to the old data structure.
- Refactors deep copy and equality/hashCode logic to match the new design.
---
## **Impact and Motivation**
- **More Structured Metadata:** SST file information is now stored as rich
objects instead of simple strings, enabling more precise tracking and
manipulation.
- **Future-Proofing:** The new model is more extensible for future metadata
additions.
- **Serialization Reliability:** Custom YAML serialization ensures the new
data structures are reliably persisted and restored.
- **Test Coverage:** The changes are well covered by updates to existing
unit tests.
---
**In summary:**
This commit replaces set/map-based SST file tracking in snapshot local data
with a model based on rich metadata objects (`SstFileInfo`, `VersionMeta`),
improves YAML serialization/deserialization, and updates all related code and
tests to support this more robust and extensible structure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
