[
https://issues.apache.org/jira/browse/KAFKA-7186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16582914#comment-16582914
]
ASF GitHub Bot commented on KAFKA-7186:
---------------------------------------
hzxa21 opened a new pull request #5519: KAFKA-7186: Avoid re-instantiating
UpdateMetadataReuqest and struct objects to reduce controller memory usage
URL: https://github.com/apache/kafka/pull/5519
During controller failover and broker changes, it sends out
UpdateMetadataRequest to all brokers in the cluster containing the states for
all partitions and live brokers. The current implementation will instantiate
the UpdateMetadataRequest object and its serialized form (Struct) for `# of
brokers` times, which causes OOM if the memory exceeds the configure JVM heap
size before GC kicks in. We have seen this issue in the production environment
for multiple times.
For example, if we have 100 brokers in the cluster and each broker is the
leader of 2k partitions, the extra memory usage introduced by controller trying
to send out UpdateMetadataRequest is around:
```
<memory used by UpdateMetadataRequest Structs> * <# of brokers> * <total #
of leader parittions>
= 250B * 100 * 200k = 5GB
```
This PR avoids the recreation of UpdateMetadataReuqest and struct objects if
the same builder object is used to build the request. This can significantly
reduce memory footprint in Controller based on the above calculation.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Controller uses too much memory when sending out UpdateMetadataRequest that
> can cause OutOfMemoryError
> ------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-7186
> URL: https://issues.apache.org/jira/browse/KAFKA-7186
> Project: Kafka
> Issue Type: Bug
> Components: controller
> Reporter: Zhanxiang (Patrick) Huang
> Assignee: Zhanxiang (Patrick) Huang
> Priority: Major
>
> During controller failover and broker changes, it sends out
> UpdateMetadataRequest to all brokers in the cluster containing the states for
> all partitions and live brokers. The current implementation will instantiate
> the UpdateMetadataRequest object and its serialized form (Struct) for <# of
> brokers> times, which causes OOM if the memory exceeds the configure JVM heap
> size. We have seen this issue in the production environment for multiple
> times.
>
> For example, if we have 100 brokers in the cluster and each broker is the
> leader of 2k partitions, the extra memory usage introduced by controller
> trying to send out UpdateMetadataRequest is around:
> <memory used by UpdateMetadataRequest Structs> * <# of brokers> * <total # of
> leader parittions>
> = 250B * 100 * 200k = 5GB
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
