[
https://issues.apache.org/jira/browse/HDDS-12998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Cheng Li updated HDDS-12998:
--------------------------------
Description:
Generally, the tar zip file of container will have smaller size than it's real
size, so reserve a 2* max container size is conservative enough in most cases.
But there are container over-allocated case, where container size can be double
or triple the max container size as
[@siddhantsangwan|https://github.com/siddhantsangwan] saw in user's
environment, have a accurate container size can handle this case welly, and
that info could be brought in:
*
[SendContainerRequest|https://github.com/apache/ozone/blob/dd79fa15f9471e8b5a2cf721c6d4c92c6a4192d3/hadoop-hdds/interface-client/src/main/proto/DatanodeClientProtocol.proto#L546-L552)]
*
[ReplicateContainerCommandProto|https://github.com/apache/ozone/blob/dd79fa15f9471e8b5a2cf721c6d4c92c6a4192d3/hadoop-hdds/interface-server/src/main/proto/ScmServerDatanodeHeartbeatProtocol.proto#L415-L422]
For backward compatibility, if that field unset, fallback to get the container
size from config.
comments:
* [https://github.com/apache/ozone/pull/8360#issuecomment-2850023149]
* [https://github.com/apache/ozone/pull/8360#issuecomment-2857154196]
* [https://github.com/apache/ozone/pull/8360#issuecomment-2857213750]
was:
Generally, the tar zip file of container will have smaller size than it's real
size, so reserve a 2* max container size is conservative enough in most cases.
But there are container over-allocated case, where container size can be double
or triple the max container size as
[@siddhantsangwan|https://github.com/siddhantsangwan] saw in user's
environment, have a accurate container size can handle this case welly, and
that info could be brought in [proto
message|https://github.com/apache/ozone/blob/f52069b7aa757d39e4819bae7d06b12955660576/hadoop-hdds/interface-client/src/main/proto/DatanodeClientProtocol.proto#L529-L545].
For backward compatibility, if that field unset, fallback to get the container
size from config.
comments:
* [https://github.com/apache/ozone/pull/8360#issuecomment-2850023149]
* [https://github.com/apache/ozone/pull/8360#issuecomment-2857154196]
* [https://github.com/apache/ozone/pull/8360#issuecomment-2857213750]
> Bring real container size in pb message when exporting/importing containers
> ---------------------------------------------------------------------------
>
> Key: HDDS-12998
> URL: https://issues.apache.org/jira/browse/HDDS-12998
> Project: Apache Ozone
> Issue Type: Sub-task
> Components: Ozone Datanode
> Reporter: Chu Cheng Li
> Assignee: Gargi Jaiswal
> Priority: Major
> Labels: pull-request-available
>
> Generally, the tar zip file of container will have smaller size than it's
> real size, so reserve a 2* max container size is conservative enough in most
> cases. But there are container over-allocated case, where container size can
> be double or triple the max container size as
> [@siddhantsangwan|https://github.com/siddhantsangwan] saw in user's
> environment, have a accurate container size can handle this case welly, and
> that info could be brought in:
> *
> [SendContainerRequest|https://github.com/apache/ozone/blob/dd79fa15f9471e8b5a2cf721c6d4c92c6a4192d3/hadoop-hdds/interface-client/src/main/proto/DatanodeClientProtocol.proto#L546-L552)]
> *
> [ReplicateContainerCommandProto|https://github.com/apache/ozone/blob/dd79fa15f9471e8b5a2cf721c6d4c92c6a4192d3/hadoop-hdds/interface-server/src/main/proto/ScmServerDatanodeHeartbeatProtocol.proto#L415-L422]
>
> For backward compatibility, if that field unset, fallback to get the
> container size from config.
>
> comments:
> * [https://github.com/apache/ozone/pull/8360#issuecomment-2850023149]
> * [https://github.com/apache/ozone/pull/8360#issuecomment-2857154196]
> * [https://github.com/apache/ozone/pull/8360#issuecomment-2857213750]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
