[
https://issues.apache.org/jira/browse/HDDS-17245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17245:
------------------------------
Affects Version/s: 2.3.0
> SetLifecycleConfiguration validated against one bucket can be stored on a
> different bucket created under the same name before it is applied
> -------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17245
> URL: https://issues.apache.org/jira/browse/HDDS-17245
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-7-bind-lifecycle-configuration-to-validated-bucket.patch
>
>
> h3. Mechanism
> {{OMLifecycleConfigurationSetRequest.preExecute}} runs on the leader before
> the request is replicated. It resolves the bucket and checks that the layout
> tag of the configuration matches the layout of the bucket.
> {{validateAndUpdateCache}} runs when the request is applied. It checks only
> that a bucket with that name exists and that the rules are valid for the
> layout tag in the request, and then binds the configuration to that bucket's
> object ID. If the bucket is deleted and another one is created with the same
> name between the two steps, the configuration is stored on a bucket it was
> never validated against, and {{KeyLifecycleService}} accepts it because the
> stored object ID matches.
> With a layout change this stores rules that the new bucket would reject. A
> configuration tagged OBJECT_STORE with the prefix {{dir1}} (no trailing
> slash) is stored on a FILE_SYSTEM_OPTIMIZED bucket, while the same request
> sent directly to that bucket is rejected, either for the layout mismatch or
> because an FSO prefix must end with {{/}}.
> h3. Trigger
> # A client sends SetLifecycleConfiguration for bucket {{b}}, and preExecute
> passes on the leader.
> # Before the request is applied, {{b}} (empty) is deleted and created again
> under the same name.
> # The request is applied and returns OK.
> The window is the time between preExecute on the leader and the apply of the
> request. I reached the state through the request methods directly (see
> Reproduction) and did not race it over RPC.
> h3. Impact
> * The request succeeds and the configuration is attached to a bucket it was
> not validated for. It stays until it is replaced or deleted.
> * For the OBJECT_STORE prefix stored on an FSO bucket above,
> {{KeyLifecycleService.getRuleUnion}} skips an FSO rule whose prefix does not
> end with {{/}} and logs a warning, so the rule silently does nothing (run
> with a scan of the bucket). When the configuration also has a rule without a
> prefix, every rule is evaluated per key and the rule acts with OBJECT_STORE
> prefix matching, which is what the client asked for (argued, not run).
> h3. Reproduction
> No standalone reproduction is attached. The regression test in the patch
> shows the behaviour: {{testValidateAndUpdateCacheRejectsRecreatedBucket}}
> runs {{preExecute}} against a bucket, replaces the bucket in the bucket table
> with another one of the same name, and applies the request. On the unfixed
> code the apply returns OK and binds the configuration to the new bucket.
> For the layout case I ran the real {{preExecute}} and
> {{validateAndUpdateCache}} against a real {{OmMetadataManagerImpl}}, with
> bucket link resolution stubbed to report the layout of the bucket currently
> in the table and the delete and create of the bucket written to the bucket
> table between the two steps. Output on the unfixed code:
> {noformat}
> preExecute passed against OBJECT_STORE bucket objectID=1000
> apply on recreated FSO bucket: status=OK
> stored configuration: layout=OBJECT_STORE bucketObjectID=2000 prefix=dir1 |
> bucket: layout=FILE_SYSTEM_OPTIMIZED objectID=2000
> control, OBJECT_STORE tagged request on FSO bucket: INVALID_REQUEST Bucket
> layout mismatch: requested lifecycle configuration has bucket layout
> OBJECT_STORE but the actual bucket has layout FILE_SYSTEM_OPTIMIZED
> control, FSO tagged request on FSO bucket: INVALID_REQUEST
> FILE_SYSTEM_OPTIMIZED bucket prefix must end with '/'.
> {noformat}
> h3. Patch
> [^MC-7-bind-lifecycle-configuration-to-validated-bucket.patch], against
> 20d95ed37fe35013b5320713665db03c06ee36fa. It also applies to master at
> 64d4a791502.
> {{preExecute}} records the object ID of the bucket it validated in the
> existing {{bucketObjectID}} field of the request's
> {{LifecycleConfiguration}}, and {{validateAndUpdateCache}} rejects the
> request with BUCKET_NOT_FOUND if the bucket now has a different object ID.
> This does not depend on what changed with the bucket, so I preferred it to
> re-checking the layout at apply. A request without the field is applied as
> before. SetLifecycleConfiguration is behind the {{S3_LIFECYCLE_SUPPORT}}
> layout feature, which is not in a release yet, so no released OM applies
> these requests differently (only OMs built from master before this change
> would accept such a request in the window).
> Covered by {{testValidateAndUpdateCacheRejectsRecreatedBucket}} in the
> existing {{TestOMLifecycleConfigurationSetRequest}}, which also checks that a
> request validated against the new bucket is applied to it. Without the change
> it fails with {{expected: <BUCKET_NOT_FOUND> but was: <OK>}}. With it the
> lifecycle request and helper suites (32 tests) and
> {{TestKeyLifecycleService}} (722 tests, 18 skipped) pass and checkstyle is
> clean. With the patch the layout case above is rejected at the apply with
> BUCKET_NOT_FOUND.
> Found by TLA+ model checking and code review of the OM volume, bucket and
> snapshot lifecycle paths under HDDS-15926, on commit
> 20d95ed37fe35013b5320713665db03c06ee36fa. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
