[
https://issues.apache.org/jira/browse/HDDS-12882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ivan Andika reopened HDDS-12882:
--------------------------------
Reopening, we can use max-keys if there is no delimiter.
> S3G should consider S3 listObjects max-keys parameter
> -----------------------------------------------------
>
> Key: HDDS-12882
> URL: https://issues.apache.org/jira/browse/HDDS-12882
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
>
> We have a situation where OM crash with heap space OOM because S3 user keeps
> sending a lot listObjectsV2 with max-keys=1 parameter as a way to check the
> object's owner (although GetObjectAcl should be a better alternative).
> We see that the KeyIterator implementation always send listStatus with
> maxKeys specified by ozone.client.list.cache (default is 1000). The default
> ozone.client.list.cache of 1000 is also the same as the "max-keys" specified
> by S3G, so normally it will be ok. If there are 1000 files inside this
> directory, the S3 client will send 1000 S3 listObjectsV2 (with max-keys=1),
> and OM has to load 1000 * 1000 = 1,000,000 OmKeyInfo entries to its memory.
> This causes OM heap space to be OOM.
> We need to take into account this max-keys sent by S3. One possible idea is
> to use the S3 max-keys if it's smaller than the configured
> ozone.client.list.cache. This way, OM will only load the necessary OmKeyInfo
> to memory.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
