[
https://issues.apache.org/jira/browse/HDDS-17099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17099:
------------------------------
Description:
{code}
Verify Role-Scoped Token Accesses | FAIL |
'
An error occurred (AccessDenied) when calling the GetObject operation: User
doesn't have the right to access this resource.' does not contain
'"AcceptRanges": "bytes"'
{code}
The failing step is {{Get Object Should Succeed ${TENANT_ONE_LINKED_BUCKET}
${ANOTHER_BUCKET_TESTFILE}}}. OM audit:
{code}
08:44:16,856 | ERROR | OMAudit | ... op=READ_KEY
{"volume":"sts-tenant-one","bucket":"tenant-one-a-linked-bucket","key":"testfile00",...,"sourceVolume":"sts-tenant-two","sourceBucket":"tenant-two-another-bucket"}
| ret=FAILURE | PERMISSION_DENIED ... doesn't have READ permission to access
key Volume:sts-tenant-two Bucket:tenant-two-another-bucket Key:testfile00
{code}
h3. Root cause
HDDS-16560 added a {{Wait Until Keyword Succeeds}} on the first {{Get Object
Should Succeed}} in {{Verify Role-Scoped Token Accesses}}, meant to wait for
all role policies to reach the Ranger plugin cache. But that read
({{${TENANT_ONE}/${TENANT_ONE_ICEBERG_BUCKET}}}) only depends on an early
policy. The linked bucket read depends on the {{${TENANT_TWO}
${TENANT_TWO_ANOTHER_BUCKET} table access}} policy, which is created last.
In the failing run, the leader OM (om1) loaded Ranger policy version 70 at
08:43:48 and version 72 at 08:44:18. The wait passed on version 70, and the
linked bucket read at 08:44:16 was denied because the last policy was not in
the cache yet.
h3. Fix
Wait on the linked bucket read instead. It needs the last created policy, so
once it succeeds every earlier policy is in the plugin cache too.
The tenant two and tenant three {{AssumeRole}} calls in {{Create Role-Scoped
Tokens for All Three Tenants}} have the same race (only the first call waits),
so they get the same wait.
- https://github.com/apache/ozone/actions/runs/39016022509/job/117350675536
was:
{code}
Verify Role-Scoped Token Accesses | FAIL |
'
An error occurred (AccessDenied) when calling the GetObject operation: User
doesn't have the right to access this resource.' does not contain
'"AcceptRanges": "bytes"'
{code}
The failing step is {{Get Object Should Succeed ${TENANT_ONE_LINKED_BUCKET}
${ANOTHER_BUCKET_TESTFILE}}}. OM audit:
{code}
08:44:16,856 | ERROR | OMAudit | ... op=READ_KEY
{"volume":"sts-tenant-one","bucket":"tenant-one-a-linked-bucket","key":"testfile00",...,"sourceVolume":"sts-tenant-two","sourceBucket":"tenant-two-another-bucket"}
| ret=FAILURE | PERMISSION_DENIED ... doesn't have READ permission to access
key Volume:sts-tenant-two Bucket:tenant-two-another-bucket Key:testfile00
{code}
h3. Root cause
HDDS-16560 added a {{Wait Until Keyword Succeeds}} on the first {{Get Object
Should Succeed}} in {{Verify Role-Scoped Token Accesses}}, meant to wait for
all role policies to reach the Ranger plugin cache. But that read
({{${TENANT_ONE}/${TENANT_ONE_ICEBERG_BUCKET}}}) only depends on an early
policy. The linked bucket read depends on the {{${TENANT_TWO}
${TENANT_TWO_ANOTHER_BUCKET} table access}} policy, which is created last.
In the failing run, the leader OM (om1) loaded Ranger policy version 70 at
08:43:48 and version 72 at 08:44:18. The wait passed on version 70, and the
linked bucket read at 08:44:16 was denied because the last policy was not in
the cache yet.
h3. Fix
Wait on the linked bucket read instead. It needs the last created policy, so
once it succeeds every earlier policy is in the plugin cache too.
- https://github.com/apache/ozone/actions/runs/39016022509/job/117350675536
> Intermittent AccessDenied on linked bucket in
> ozone-secure-sts-multitenant.robot
> --------------------------------------------------------------------------------
>
> Key: HDDS-17099
> URL: https://issues.apache.org/jira/browse/HDDS-17099
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: S3, test
> Reporter: Siyao Meng
> Assignee: Siyao Meng
> Priority: Major
> Labels: pull-request-available
>
> {code}
> Verify Role-Scoped Token Accesses | FAIL |
> '
> An error occurred (AccessDenied) when calling the GetObject operation: User
> doesn't have the right to access this resource.' does not contain
> '"AcceptRanges": "bytes"'
> {code}
> The failing step is {{Get Object Should Succeed ${TENANT_ONE_LINKED_BUCKET}
> ${ANOTHER_BUCKET_TESTFILE}}}. OM audit:
> {code}
> 08:44:16,856 | ERROR | OMAudit | ... op=READ_KEY
> {"volume":"sts-tenant-one","bucket":"tenant-one-a-linked-bucket","key":"testfile00",...,"sourceVolume":"sts-tenant-two","sourceBucket":"tenant-two-another-bucket"}
> | ret=FAILURE | PERMISSION_DENIED ... doesn't have READ permission to access
> key Volume:sts-tenant-two Bucket:tenant-two-another-bucket Key:testfile00
> {code}
> h3. Root cause
> HDDS-16560 added a {{Wait Until Keyword Succeeds}} on the first {{Get Object
> Should Succeed}} in {{Verify Role-Scoped Token Accesses}}, meant to wait for
> all role policies to reach the Ranger plugin cache. But that read
> ({{${TENANT_ONE}/${TENANT_ONE_ICEBERG_BUCKET}}}) only depends on an early
> policy. The linked bucket read depends on the {{${TENANT_TWO}
> ${TENANT_TWO_ANOTHER_BUCKET} table access}} policy, which is created last.
> In the failing run, the leader OM (om1) loaded Ranger policy version 70 at
> 08:43:48 and version 72 at 08:44:18. The wait passed on version 70, and the
> linked bucket read at 08:44:16 was denied because the last policy was not in
> the cache yet.
> h3. Fix
> Wait on the linked bucket read instead. It needs the last created policy, so
> once it succeeds every earlier policy is in the plugin cache too.
> The tenant two and tenant three {{AssumeRole}} calls in {{Create Role-Scoped
> Tokens for All Three Tenants}} have the same race (only the first call
> waits), so they get the same wait.
> - https://github.com/apache/ozone/actions/runs/39016022509/job/117350675536
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
