[
https://issues.apache.org/jira/browse/HDDS-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huang Kuan Hao updated HDDS-16673:
----------------------------------
Description:
RootEndpoint calls hasNext() again to determine whether a continuation token is
needed after the listing loop has already reached the end. The bucket iterator
fetches the same empty page again, adding a redundant ListBuckets RPC to
non-empty final pages in paginated ListBuckets and ListDirectoryBuckets
requests.
Avoid probing the exhausted iterator again, while retaining the lookahead
needed for full pages and the existing listing behavior.
AWS API references:
[ListBuckets|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html],
[ListDirectoryBuckets|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListDirectoryBuckets.html].
was:
RootEndpoint checks the bucket iterator again for a continuation token after
the listing loop has already observed an empty page. OzoneVolume's bucket
iterator does not retain the exhausted state, so this sends a duplicate
ListBuckets RPC for non-empty final pages in paginated ListBuckets and
ListDirectoryBuckets requests.
Check the response limit before calling hasNext() in the loop, and perform the
post-loop lookahead only when the limit is reached. Keep that lookahead for
full pages, since reaching the requested limit alone does not establish whether
more buckets remain. This saves one ListBuckets RPC for affected requests while
retaining the initial listing request, existing authorization, bucket
filtering, and continuation-token construction.
Regression tests should use the real OzoneVolume iterator and verify:
* The terminal empty page is fetched only once for both underfilled and
exact-limit final pages.
* Full pages with more results retain a continuation token, and continuation
requests return the remaining results correctly.
* Empty lists, bucket results, and owner information retain their existing
behavior.
* ListDirectoryBuckets retains its FSO filtering with mixed bucket layouts and
its existing max-directory-buckets=0 behavior.
AWS API references:
*
[ListBuckets|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html]:
max-buckets ranges from 1 to 10000; ContinuationToken indicates more buckets
can be listed.
*
[ListDirectoryBuckets|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListDirectoryBuckets.html]:
max-directory-buckets ranges from 0 to 1000; ContinuationToken is used for
pagination.
> Avoid redundant ListBuckets RPCs when S3 bucket listing reaches the end
> -----------------------------------------------------------------------
>
> Key: HDDS-16673
> URL: https://issues.apache.org/jira/browse/HDDS-16673
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Huang Kuan Hao
> Assignee: Huang Kuan Hao
> Priority: Major
>
> RootEndpoint calls hasNext() again to determine whether a continuation token
> is needed after the listing loop has already reached the end. The bucket
> iterator fetches the same empty page again, adding a redundant ListBuckets
> RPC to non-empty final pages in paginated ListBuckets and
> ListDirectoryBuckets requests.
> Avoid probing the exhausted iterator again, while retaining the lookahead
> needed for full pages and the existing listing behavior.
> AWS API references:
> [ListBuckets|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html],
>
> [ListDirectoryBuckets|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListDirectoryBuckets.html].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
