symious commented on code in PR #10822: URL: https://github.com/apache/ozone/pull/10822#discussion_r3900088777
########## hadoop-hdds/docs/content/design/s3-versioning.md: ########## @@ -0,0 +1,630 @@ +--- +title: S3-compatible Object Versioning +summary: Bucket-level, S3-compatible object versioning with O(1) version writes, reclaimed by the existing lifecycle engine +date: 2026-07-21 +jira: HDDS-15728 +status: accepted +author: Symious +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Summary + +Add S3-compatible object versioning to Ozone: the full three-state bucket state +machine (Unversioned / Enabled / Suspended), per-key version chains with delete +markers and null versions, the S3 versioning APIs on the S3 Gateway, and version +reclamation expressed as lifecycle rules on the lifecycle engine Ozone already +has — with O(1) metadata cost per version operation and zero regression on +non-versioned paths. + +# Status + +Defined in the markdown header. + +# Problem statement (Motivation / Abstract) + +Amazon S3 provides bucket-level object versioning: a single key can retain multiple +versions, so users can recover objects that were accidentally overwritten or +deleted. A large part of the S3 ecosystem (backup software, data lake components, +DR tooling) depends on the versioning APIs (`PutBucketVersioning`, +`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an +S3-compatible API through the S3 Gateway but does not support object versioning +today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended +state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record +rather than object versions, and the gateway has no versioning endpoints. + +This proposal implements versioning with S3-compatible semantics, usable by +standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a +version operation is decoupled from the number of versions (one extra small KV +write per operation). + +Reclamation is a first-class part of the feature rather than an afterthought. The +S3 troubleshooting guide documents list degradation and throttling on keys with +millions of versions and leaves the fix to user-configured Lifecycle rules that +are often forgotten. Ozone already has a lifecycle engine (HDDS-8342, in master), +so versioning does not need a reclamation mechanism of its own: it adds the three +version-aware actions S3 defines to that engine, and the lifecycle service +becomes enabled by default, so that a rule a user writes is a rule that runs. Review Comment: Updated, PTAL. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
