chungen0126 commented on code in PR #11053: URL: https://github.com/apache/ozone/pull/11053#discussion_r4036081830
########## hadoop-hdds/docs/content/design/s3-object-lock.md: ########## @@ -0,0 +1,559 @@ +--- +title: S3 Object Lock +summary: Design to support S3 object lock. +date: 2026-08-18 +jira: HDDS-15945 +status: accepted +author: Chung En Lee +--- +<!-- + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. See accompanying LICENSE file. +--> + +# S3 Object Lock Design Doc + +## Summary + +This design document aims to plan and implement the Object Lock mechanism for OBS buckets integrated with Ranger. +The primary objective is to provide data immutability and tamper-proof protection through the object locking feature. + +## Problem statement + +With growing demands for data security and compliance, ensuring that critical data stored in OBS (Object Storage) is +protected from accidental or malicious deletion and overwriting has become an essential system protection requirement. +To establish a more rigorous data protection mechanism, we plan to introduce the Object Lock feature. + +Considering the current system architecture and access control strategies, +this design integrates with existing Apache Ranger to manage Object Lock permissions on OBS buckets. +Meanwhile, to accelerate core feature delivery, we have decided to exclude complex multi-version locking (Versioning Lock) +, Legacy buckets, and FSO buckets from this initial release. In addition, support for Native ACLs is excluded; Native ACLs typically grant permissions +at the granular bucket or object level, whereas Object Lock permission management favors broad, role-based authorization, +creating a conflict in design philosophies. Narrowing the scope allows us to focus on the core functionality and ensure a rapid, +stable rollout of baseline tamper-proof protection. + +**Goal:** + +* Implement the Object Lock feature on standard OBS buckets, fully integrated with Ranger for permission and access control. +* Support single-version objects only. + +## Non-Goal + +* Versioning Lock: Support for locking across multiple object versions is deferred (multi-version core features are currently under development). +* FSO Legacy Buckets: Object Lock support for legacy FSO buckets is excluded. +* Native ACL Support: Native ACLs will not be used for access control or advanced configuration such as Retention Mode (Governance); access management is centralized exclusively via Ranger. + +## Technical Description + +### Terminology + +**Legal Hold** + +* **Definition**: Applies an indefinite lock status to an object. The object remains protected until an administrator explicitly removes the lock (Remove Legal Hold). +* **Restricted Operations**: + * Put Object + * Delete Object + * Multipart Initial / Complete +* **Allowed Operations**: + * Get Object + * Get Legal Hold + * Put Legal Hold (Depends on permission) + +**Retention** + +* Definition: Configures a retention policy for an object to prevent deletion or modification. Retention can be duration-based (configured in days or years, establishing a fixed `RetainUntilDate`) or event-driven (**Event Hold / Event-based Retention**, where an object remains protected indefinitely until an external business or legal event triggers the final retention countdown). +* Retention Modes: + * Compliance Mode: The strictest protection tier. Once applied, no user (including root/admin) can remove the lock, shorten the duration, or overwrite the object before the retention period expires. + * Governance Mode: A flexible protection tier. Standard users are restricted by locking rules, but users with `BypassGovernanceRetention` permissions can bypass restrictions to modify or delete the object. +* Event Hold / Event-based Retention: + * Used for records management where the retention lifecycle depends on external events (e.g., contract termination, employee departure, loan closure). + * Keeps the object immutable while awaiting event notification; once the event occurs, the definitive expiration date (`RetainUntilDate`) is calculated and applied. +* Restricted Operations: + * Put Object / Copy Object (Overwrites) + * Delete Object + * Multipart Upload (Initial / Complete) + * Shorten Retention Period (in Compliance Mode) +* Allowed Operations: + * Get Object + * Extend Retention Period + +#### Event Hold Trigger Workflow +For enterprise governance workflows, the retention lifecycle often depends on specific business triggers rather than a static configuration at creation. The Event Hold mechanism handles this dynamic lifecycle through the following state transitions: + +```text +object created + | +event hold applied (indefinite WORM protection) + | +business event occurs + | +hold released + | +retention timer starts (RetainUntilDate is calculated) + | +WORM until timer expires +``` + +> _**Note**: +> * Background & Root Cause: A prerequisite for enabling WORM (Write Once, Read Many) in AWS S3 is that Object Versioning must be enabled. Under S3 architecture, executing a Put on a locked object generates a new version without affecting the protected prior version; thus, S3 Object Lock primarily restricts Delete Object. +> * Ozone Implementation Status: Because Ozone's versioning feature is still under development, to guarantee absolute immutability during the lock period, Ozone will directly block and reject all overwrite operations (such as any form of Put or overwrite) on locked objects. + +### Table Changes + +**Bucket Table** + +Two new fields: objectLockEnabled & defaultRetention. + +```protobuf + message BucketInfo { + // ... existing fields + required bool objectLockEnabled = 24 [default = false]; + optional RetentionConfig defaultRetention = 25; + } + + message RetentionConfig { + optional Rule rule = 1; + optional EventHold eventHold = 2; + + } + + message EventHold { + required bool enabled = 1; + required Rule rule = 2; + } + + message Rule { + required RetentionMode retentionMode = 1; + required TimeUnit timeUnit = 2; + required uint64 duration = 3; + } + + enum TimeUnit { + DAYS = 1; + YEARS = 2; + } + + enum RetentionMode { + GOVERNANCE = 1; + COMPLIANCE = 2; + } +``` + + + +**Key Table** + +* **retentionDate**: Represents the expiration timestamp of the retention lock. While clients or API calls specify retention duration in terms of days or years, Ozone calculates the definitive expiration date upon applying the rule and persists it as a timestamp into key table. +* **legalHold**: Indicates whether an explicit legal hold is active on the key. + +Two new fields: retentionDate & legalHold. + +```protobuf + message KeyInfo { + // ... existing fields + optional string retentionDate = 23; + optional RetentionConfig retentionConfig = 24; + optional bool legalHold = 25 [default = false]; + } +``` + + + +### Ranger Access Control + +Ozone Object Lock enforces a dual-gate protection mechanism combining **Ranger policy-based authorization** and **underlying OM WORM state validation**: + +* **Standard Data Operations (Put / Delete)**: Even if a principal has valid Ranger `WRITE` or `DELETE` access, any attempt to mutate, overwrite, or delete an object under an active Legal Hold or an unexpired Retention period is immediately rejected with `403 Access Denied` (`WORMProtectionException`). +* **Fine-Grained S3 Action Matching**: Object Lock management integrates with the action-matching authorization framework introduced via STS. S3 Object Lock actions map 1:1 to AWS S3 action names. See [AWS STS Design for Ozone S3](ozone-sts.md) for details. + +--- + +#### 1. Configuration & Component Updates + +To enable action-level authorization and onboard Object Lock actions into the Ozone & Ranger ecosystem, the following updates are required: + +**Prerequisite: Feature Flag Activation**: +* Enable the Ranger action matcher condition in the Ozone configuration: + ```properties + ranger.servicedef.ozone.enableActionMatcherInPoliciesCondition=true + ``` +* When enabled, Ozone S3 Gateway (S3G) intercepts the high-level S3 API operation and populates the S3 action name into the `RangerAccessRequest`, allowing Ranger's `RangerActionMatcher` to evaluate policy conditions against incoming requests. + +1. **UI & Metadata Registration**: + * Update `security-admin/src/main/webapp/react-webapp/src/utils/actionRequirements/ozone.json` in Apache Ranger: + Register the 7 new Object Lock actions along with their prerequisite primitive permissions (e.g., `READ`, `WRITE`) mapped by resource level (Bucket vs. Key). This enforces design-time validation in the Ranger React UI, preventing administrators from creating invalid policy items. + +2. **S3 Gateway & STS Mapping**: + * Update **`S3GActionIamMapper`**: Map incoming HTTP request context and sub-resources (e.g., `?legal-hold`, `?retention`, `?object-lock`) to the corresponding action string. + * Update **`IamSessionPolicyResolver`**: Ensure session-scoped policies (STS AssumeRole / federation tokens) recognize and evaluate the new Object Lock action strings. + +--- + +#### 2. Action to Ranger Access Type Mapping + +Following the declarative contract in `ozone.json`, each S3 action requires a baseline primitive Ranger permission at the designated resource hierarchy: + +| S3 Action | Resource Level | Required Ranger Access Type | Description | +| :--- | :--- | :--- | :--- | +| **`GetBucketObjectLockConfiguration`** | Bucket | `READ` | Retrieve default Object Lock settings on a bucket. | +| **`PutBucketObjectLockConfiguration`** | Bucket | `WRITE` | Configure default retention mode and period on a bucket. | +| **`GetObjectRetention`** | Key | `READ` | Read the retention mode and Retain-Until date of an object version. | +| **`PutObjectRetention`** | Key | `WRITE` | Set or extend object retention mode and duration. | +| **`GetObjectLegalHold`** | Key | `READ` | Query the current Legal Hold status (`ON` or `OFF`). | +| **`PutObjectLegalHold`** | Key | `WRITE` | Toggle the Legal Hold state (`ON` or `OFF`). | +| **`BypassGovernanceRetention`** | Key | `WRITE` / `DELETE` | Privileged entitlement to bypass retention in Governance Mode. | + +--- + +#### 3. Value-Level Authorization & Ranger Condition Evaluator + +While API invocation is controlled by the S3 Action (e.g., granting `PutObjectLegalHold`), real-world governance often requires **role segregation based on request values** (e.g., Compliance Officers can toggle `ON`, but only external Auditors can toggle `OFF`; or preventing operators from setting `COMPLIANCE` mode). Review Comment: Thanks for the feedback! Actually, AWS IAM does support this kind of value-level condition matching for S3. According to the [AWS IAM Condition Operators documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_condition_operators.html), S3 provides condition keys that allow restricting specific actions based on request values. The reason I included this additional complexity is based on use cases like the one described in this [AWS re:Post article: Denying removal of Amazon S3 Object Lock Legal Hold](https://repost.aws/articles/ARHP6wZxPyQ-6xGawE5MFPuw/denying-removal-of-amazon-s3-object-lock-legal-hold). The article demonstrates how administrators separate the permissions for turning a legal hold `ON` versus `OFF` by leveraging the `StringEquals` condition operator. As mentioned in the quote: > As legal hold has no expiration date, users may wish to use this mode to apply an indefinite lock on objects they wish to protect from accidental or malicious deletion. In this scenario, it may be desirable to restrict permissions to remove legal hold from objects. You can do this with a condition key in the Condition element of an IAM policy, specifically "StringEquals": "s3:object-lock-legal-hold": "OFF" applied to the action "PutObjectLegalHold". That being said, I am completely open to discussing this. If we feel that `PutObjectLegalHold` (without distinguishing between ON/OFF payloads) is enough for Ozone at this stage, we can definitely drop this custom condition evaluator to keep the initial implementation simple. Let me know what @fmorg-git and @ChenSammi think! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
