Attenders: Yiyang, Hui, Chung en, Sammi, Peter, Wei-ChiuKey Achievements
This meeting covered three core topics: Version Control (Versioning) design
updates, Object Lock/Locking feature design review, and discussion of
Snapshot and Fork features.
------------------------------
Version Control Design Progress
- Completed four revisions: supplemented OEP description with diagrams,
replaced Version Cleanup Service with App Lifecycle, added layout
explanations for Snapshot and Version, changed Version ID to
timestamp-based generation (referencing Microsoft and Google solutions)
- Currently have approximately 7 subtasks, pushed multiple commits this
week (squash merged after internal review)
- Sammi plans to organize an internal design review in two weeks,
inviting team colleagues to participate and gather more feedback
- Rationale for keeping two tables: most users use regular buckets,
versioned bucket users are fewer; separate tables avoid tombstone impact
from version table affecting regular query performance
------------------------------
Object Locking Design Review
- Locking process adopts double-check: Pre-check for quick filtering,
reconfirmation at Commit to ensure linearizability
- Locking can be set at two points: specified directly when uploading
objects, or set via separate API after upload
-
Things to Clarify:
- In Compliance Mode, S3 documentation states account deletion is
required to delete objects, how this map to Ozone case, corresponding
behavior needs clarification
- Two APIs (Enable Lock / Set Retention Config) called separately may
lead to incomplete state, suggest merging into a single API
- After Lock Enable, Versioning also cannot be disabled; this binding
behavior needs to be clearly documented
- Default behavior when Retention Config is not set needs
confirmation
-
Permission Control:
- Existing Ranger plugin has not yet refined to Lock-related permission
granularity, can reference Lifecycle mapping approach (map to
Bucket Write
permission)
- Permission control method for enabling/suspending Versioning also
needs clarification (Native ACL vs Ranger)
-
Interaction with Lifecycle:
- If objects are locked, Lifecycle needs to check lock status when
cleaning old versions; when all versions are locked (e.g., exceeding max
version limit), cleanup behavior definition needs to reference
S3 handling
approach
- Current version control design has removed max version
configuration, now fully handled by Lifecycle
------------------------------
Snapshot and Fork Feature Discussion
- Current Snapshot primary purpose is locking files to prevent deletion,
does not support multi-branch or rollback
- Fork feature use case: in AI workflows, multiple agents run parallel
experiments based on the same dataset, requiring isolated writes and
protection of original data
-
Implementation Approach Analysis:
- Approach 1: Add additional table to handle two dimensions (multiple
forks in same bucket, large number of changes after fork), complex
implementation
- Approach 2: Implement based on existing SST checkpoint mechanism,
theoretically suitable but requires underlying changes, high difficulty
- Merge back is feasible (sort by timeline based on globally
incrementing ID, no need to resolve conflicts), but overall engineering
effort is large
- Conclusion: Not proceeding with Fork POC for now, currently lacking
suitable underlying mechanism support
------------------------------
Other Updates
- Ratis 3.3 Version Upgrade: In progress, encountered some logic-related
errors, currently remaining Compliance-related issues
- Automated Release Process: Wei-Chiu is using AI tools to build
automated release process, completed versions 2.1.1 and 2.1.2
- IPv6 / Multi-home Support: Currently does not fully support automatic
switching, lacking test environment
- Secure Ozone (without Kerberos): Discussing approach of obtaining
Delegation Token through Mutual TLS + Knox, CA trust chain establishment
method still under discussion
------------------------------
Action Items
- Chung: Update locking design document, clarify behavior definitions
(Compliance Mode deletion behavior, Lock and Versioning binding
relationship, default Retention behavior)
- yiyang: Complete final updates to version control design document
- Sammi: Complete locking design document review, add comments
- peter: Share Fork-related Jira links and POC reference materials (
https://issues.apache.org/jira/browse/HDDS-15125)
- Wei-Chiu: Continue refining Secure Ozone without Kerberos solution
design
