symious commented on code in PR #10822: URL: https://github.com/apache/ozone/pull/10822#discussion_r3900041922
########## hadoop-hdds/docs/content/design/s3-versioning.md: ########## @@ -0,0 +1,630 @@ +--- +title: S3-compatible Object Versioning +summary: Bucket-level, S3-compatible object versioning with O(1) version writes, reclaimed by the existing lifecycle engine +date: 2026-07-21 +jira: HDDS-15728 +status: accepted +author: Symious +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# Summary + +Add S3-compatible object versioning to Ozone: the full three-state bucket state +machine (Unversioned / Enabled / Suspended), per-key version chains with delete +markers and null versions, the S3 versioning APIs on the S3 Gateway, and version +reclamation expressed as lifecycle rules on the lifecycle engine Ozone already +has — with O(1) metadata cost per version operation and zero regression on +non-versioned paths. + +# Status + +Defined in the markdown header. + +# Problem statement (Motivation / Abstract) + +Amazon S3 provides bucket-level object versioning: a single key can retain multiple +versions, so users can recover objects that were accidentally overwritten or +deleted. A large part of the S3 ecosystem (backup software, data lake components, +DR tooling) depends on the versioning APIs (`PutBucketVersioning`, +`ListObjectVersions`, object operations with a `versionId`). Ozone exposes an +S3-compatible API through the S3 Gateway but does not support object versioning +today: the bucket-level `isVersionEnabled` boolean cannot express the Suspended +state, `OmKeyInfo.keyLocationVersions` tracks block locations within one record +rather than object versions, and the gateway has no versioning endpoints. + +This proposal implements versioning with S3-compatible semantics, usable by +standard S3 clients (AWS CLI / SDKs) without modification. The metadata cost of a +version operation is decoupled from the number of versions (one extra small KV +write per operation). + +Reclamation is a first-class part of the feature rather than an afterthought. The +S3 troubleshooting guide documents list degradation and throttling on keys with +millions of versions and leaves the fix to user-configured Lifecycle rules that +are often forgotten. Ozone already has a lifecycle engine (HDDS-8342, in master), +so versioning does not need a reclamation mechanism of its own: it adds the three +version-aware actions S3 defines to that engine, and the lifecycle service +becomes enabled by default, so that a rule a user writes is a rule that runs. + +# Non-goals + +- **MFA delete** — depends on the AWS IAM/MFA device ecosystem; Ozone has no + counterpart infrastructure. The `MfaDelete` field of `PutBucketVersioning` + returns NotImplemented. +- **Version-aware lifecycle transitions** (`NoncurrentVersionTransition`, and + `Transition` generally) — Ozone has no storage-class tiering to transition to. +- **Version-aware cross-cluster replication.** +- **Versioning for FSO / LEGACY bucket layouts** — the first version supports + OBJECT_STORE buckets only; enabling versioning on other layouts returns + NotImplemented. Combining FSO's directory/rename semantics with per-key version + chains is disproportionately complex, and S3 tooling scenarios essentially use + the OBS layout. FSO support can be evaluated as an independent follow-up. +- **Coexistence with Ozone snapshots on the same bucket** — the version-aware + reclamation and the version-aware diff the combination needs are deferred past + the first version, so OM rejects the combination outright rather than leaving it + to convention. The snapshot section below states the enforcement and what lifts + it. + +# Technical Description (Architecture and implementation details) + +## Bucket state machine + +```mermaid +stateDiagram-v2 + direction LR + [*] --> Unversioned: bucket created + Unversioned --> Enabled: PutBucketVersioning(Enabled) + Enabled --> Suspended: PutBucketVersioning(Suspended) + Suspended --> Enabled: PutBucketVersioning(Enabled) + Unversioned --> Unversioned: no versions exist + Enabled --> Enabled: every write creates a version + Suspended --> Suspended: writes reuse the null slot +``` + +There is no edge back to Unversioned. Once a bucket has been Enabled it can only +move between Enabled and Suspended, so no single state change can silently +destroy the versions the bucket already holds — S3's data-protection promise. OM +enforces this on `SetBucketProperty` and rejects a transition to `UNVERSIONED` +with `INVALID_REQUEST`. + +The only inbound edge is `PutBucketVersioning`, as in S3, where `CreateBucket` +carries no versioning parameter: a bucket is always created Unversioned. OM +rejects a `CreateBucket` that carries a versioning status, also with +`INVALID_REQUEST`, so the state a bucket starts in cannot be chosen and every +status it ever holds has been through the transition check above. The field is +on `BucketInfo` because that message is the bucket's on-disk record and the +shape `InfoBucket` and `ListBuckets` return, not because a create needs it. + +A `BucketVersioningStatusProto` enum (`UNVERSIONED` / `VERSIONING_ENABLED` / +`VERSIONING_SUSPENDED`) is added as an optional field on `BucketInfo` and +`BucketArgs`. The legacy `isVersionEnabled` boolean is kept, and a status +derives it (`ENABLED → true`, otherwise `false`). The reverse does not hold: a +record without the enum reads as `UNVERSIONED` whatever the boolean says, and a +legacy client setting the boolean on such a bucket leaves it without a status, +so it keeps using the in-record block version list rather than being opted into +S3 versioning semantics behind its owner's back. A legacy client's boolean is +mapped onto the state machine only on a bucket that already carries a status, +where the two have to stay consistent. Old and new clients/OMs coexist during +rolling upgrades either way. + +## Metadata layout: keyTable (current) + versionedKeyTable (noncurrent) + +A new column family, **versionedKeyTable**, splits responsibilities with keyTable: + +- **keyTable** (existing, semantics unchanged) always holds each key's **current + version** — a regular object or a delete marker. Plain GET / HEAD / ListObjects + read paths are unchanged. +- **versionedKeyTable** (new) holds all **noncurrent** versions (including + noncurrent delete markers), each as a complete `OmKeyInfo`. The RocksDB key is + + ``` + /{volume}/{bucket}/{keyName}\x00{Long.MAX_VALUE - versionId} + ``` + + (fixed-width hex suffix; the separator is `0x00` rather than `/` because OBS key + names contain `/` verbatim, which would interleave a key's versions with those of + keys nested under it), so all versions of a key are physically adjacent and + ordered newest to oldest. + +```mermaid +flowchart LR + subgraph KT["keyTable — one entry per key"] + K1["/vol/buck/photo.jpg<br/>versionId 104 · current"] + end + subgraph VKT["versionedKeyTable — noncurrent, newest → oldest"] + V1["/vol/buck/photo.jpg 0x00 MAX-103<br/>delete marker 103"] + V2["/vol/buck/photo.jpg 0x00 MAX-102<br/>object 102"] + V3["/vol/buck/photo.jpg 0x00 MAX-101<br/>object 101"] + end + K1 -. "one seek reaches the newest noncurrent version" .-> V1 + V1 --> V2 --> V3 +``` + +Because the suffix is `MAX - versionId`, a single `seek` on the key's prefix lands +on the newest noncurrent version: `ListObjectVersions` and version promotion are a +seek plus a sequential read, never a scan-and-sort. The table is registered in +`OmMetadataManagerImpl.getTableBucketPrefix` alongside the other key tables, so +bucket-prefixed iteration and SST filtering resolve its prefix. + +`OmKeyInfo` gains three optional proto fields (old records deserialize +compatibly): `versionId` (int64, assigned once at version creation, then frozen), +`isDeleteMarker` (a marker is a record with this flag and no data blocks — no +datanode storage), and `isNullVersion` (the single overwritable "null version" +slot per key). Keys written before versioning was enabled are interpreted as null +versions on read — **zero migration**, matching S3's "enabling versioning does +not change existing objects". + +Every keyTable ↔ versionedKeyTable update rides OM's existing atomic +multi-table WriteBatch commit (the same double-buffer pattern used today for +keyTable + deletedTable on overwrite): no new transaction mechanism and no +cross-table consistency problem. The new column family is introduced under the OM +layout feature / finalization framework (`OMLayoutFeature.OBJECT_VERSIONING`): +before finalization, requests carrying a versioning status are rejected. + +## How a key's versions evolve + +The two mechanisms that are hardest to read out of prose — the delete marker and +the null version — are easiest to follow as one continuous story of a single key +`k`. `obj(n)` is a data version with versionId `n`, `marker(n)` a delete marker, +and `null` marks the record occupying the key's null slot. + +| # | Operation | Bucket state | keyTable (current) | versionedKeyTable (newest → oldest) | +|---|---|---|---|---| +| 0 | `PUT k` | Unversioned | `obj(—)` no versionId | – | +| 1 | `PutBucketVersioning(Enabled)` | Enabled | `obj(—)` read as the null version | – | +| 2 | `PUT k` | Enabled | `obj(101)` | `obj(—)` | +| 3 | `PUT k` | Enabled | `obj(102)` | `obj(101)`, `obj(—)` | +| 4 | `DELETE k` | Enabled | `marker(103)` | `obj(102)`, `obj(101)`, `obj(—)` | +| 5 | `DELETE k?versionId=103` | Enabled | `obj(102)` **promoted back** | `obj(101)`, `obj(—)` | +| 6 | `PutBucketVersioning(Suspended)` | Suspended | `obj(102)` | `obj(101)`, `obj(—)` | +| 7 | `PUT k` | Suspended | `obj(104, null)` | `obj(102)`, `obj(101)` | +| 8 | `PUT k` | Suspended | `obj(105, null)` overwritten in place | unchanged | +| 9 | `DELETE k` | Suspended | `marker(106, null)` overwritten in place | unchanged | +| 10 | `PutBucketVersioning(Enabled)`, `PUT k` | Enabled | `obj(107)` | `marker(106, null)`, `obj(102)`, `obj(101)` | + +Three things to read out of it: + +- **A delete marker is an ordinary version that happens to carry no blocks** + (step 4). It becomes the current version, so a plain `GET` returns 404 with + `x-amz-delete-marker: true` while every earlier version stays intact and + addressable by versionId. Deleting the marker by versionId (step 5) is exactly + S3's "restore an object": the newest remaining version is promoted back into + keyTable. +- **Suspended does not stop versioning, it stops *accumulating*** (steps 7–9). + Each write lands in the key's single null slot: the first suspended write pushes + the previous current version down into versionedKeyTable and drops the key's + older null record (step 7); every later suspended write overwrites the null slot + in place, so the version count stops growing. Versions accumulated while Enabled + are never touched. +- **The null version is a slot, not a reserved id** (step 7 onwards). It carries a + normally generated versionId like any other version and is marked by + `isNullVersion`. Pinning it to a fixed low id would misorder a null created + between two versioned writes, which is the middle version of the key, not its + oldest. Step 10 shows a null delete marker sliding into the noncurrent chain in + its correct chronological position once the bucket is Enabled again. + +### Where a write lands + +The table above is one trace; the rule behind it is the same for every write that +carries no versionId. Suspended is the branch worth reading twice: it does not +stop versioning, it collapses every write into the key's one null slot, which is +why the version count stops growing while the versions accumulated under Enabled +stay untouched. + +```mermaid +flowchart TB + W["PUT k / DELETE k<br/>(no versionId)"] --> S{"bucket state"} + S -->|Unversioned| U["overwrite keyTable in place<br/>old blocks → deletedTable<br/><i>unchanged from today</i>"] + S -->|Enabled| E["move current version → versionedKeyTable<br/>write the new record into keyTable"] + S -->|Suspended| N{"is the current version<br/>already the null slot?"} + N -->|yes| N1["overwrite the null record in place<br/>its blocks → deletedTable, a marker carries none<br/><b>version count does not grow</b>"] + N -->|no| N2["move current version → versionedKeyTable<br/>write the new record as current, isNullVersion<br/>drop the key's older null record, if any"] + E --> R["new current version:<br/>an object on PUT, a delete marker on DELETE"] + N1 --> R + N2 --> R +``` + +### The life of a delete marker + +A delete marker is not a tombstone the read path has to special-case away — it is +an ordinary version that carries no blocks, and it occupies the current slot like +any other. That single fact produces all of its externally visible behaviour: + +```mermaid +stateDiagram-v2 + direction LR + Obj: current = obj(102) + Marker: current = marker(103) + Gone: key no longer exists + [*] --> Obj: PUT k + Obj --> Marker: DELETE k (no versionId) + Marker --> Obj: DELETE k?versionId=103<br/>promotion restores obj(102) + Marker --> Gone: lifecycle ExpiredObjectDeleteMarker<br/>once it is the only version left + Obj --> Obj: GET k → 200 + Marker --> Marker: GET k → 404, x-amz-delete-marker +``` + +Deleting the marker by versionId is exactly S3's "restore an object": nothing is +rewritten, the newest remaining version is moved back into keyTable. + +### Version promotion + +The invariant is that keyTable always holds a key's current version. When a +permanent delete removes the current version, promotion restores the invariant in +the same WriteBatch, under the bucket write lock: + +```mermaid +flowchart TB + A["DELETE k?versionId = current"] --> B["remove the current entry from keyTable<br/>blocks → deletedTable"] + B --> C{"seek versionedKeyTable<br/>on k's prefix"} + C -->|"a noncurrent version exists"| D["move the newest one back into keyTable<br/>record content stays frozen — a pure positional move"] + C -->|"none left"| E["the key disappears entirely"] +``` + +## VersionId generation + +`versionId` generation is abstracted behind a `VersionIdGenerator` interface, Review Comment: Some issues will happen with this approch, one is versionId will be reset when all versions of the key being removed, users will have incorrect value when key being recreated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
