Huang Kuan Hao created HDDS-16673:
-------------------------------------
Summary: Avoid redundant ListBuckets RPCs when S3 bucket listing
reaches the end
Key: HDDS-16673
URL: https://issues.apache.org/jira/browse/HDDS-16673
Project: Apache Ozone
Issue Type: Sub-task
Reporter: Huang Kuan Hao
Assignee: Huang Kuan Hao
RootEndpoint checks the bucket iterator again for a continuation token after
the listing loop has already observed an empty page. OzoneVolume's bucket
iterator does not retain the exhausted state, so this sends a duplicate
ListBuckets RPC for non-empty final pages in paginated ListBuckets and
ListDirectoryBuckets requests.
Check the response limit before calling hasNext() in the loop, and perform the
post-loop lookahead only when the limit is reached. Keep that lookahead for
full pages, since reaching the requested limit alone does not establish whether
more buckets remain. This saves one ListBuckets RPC for affected requests while
retaining the initial listing request, existing authorization, bucket
filtering, and continuation-token construction.
Regression tests should use the real OzoneVolume iterator and verify:
* The terminal empty page is fetched only once for both underfilled and
exact-limit final pages.
* Full pages with more results retain a continuation token, and continuation
requests return the remaining results correctly.
* Empty lists, bucket results, and owner information retain their existing
behavior.
* ListDirectoryBuckets retains its FSO filtering with mixed bucket layouts and
its existing max-directory-buckets=0 behavior.
AWS API references:
*
[ListBuckets|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListBuckets.html]:
max-buckets ranges from 1 to 10000; ContinuationToken indicates more buckets
can be listed.
*
[ListDirectoryBuckets|https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListDirectoryBuckets.html]:
max-directory-buckets ranges from 0 to 1000; ContinuationToken is used for
pagination.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
