[
https://issues.apache.org/jira/browse/KAFKA-17578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Kim updated KAFKA-17578:
-----------------------------
Description:
The ModernGroup#subscribedTopicMetadata takes too much memory due to
partitionRacks:
```
/**
* Map of every partition Id to a set of its rack Ids, if they exist.
* If rack information is unavailable for all partitions, this is an empty map.
*/
private final Map<Integer, Set<String>> partitionRacks;
```
This is not being used at the moment as the consumer protocol does not support
rack aware assignments.
A heap dump from a group with 500 members, 2K subscribed topic partitions shows
654,400 bytes used for partitionRacks. The rest of the ConsumerGroup object
holds 822,860 bytes.
was:
The ModernGroup#subscribedTopicMetadata takes too much memory due to
partitionRacks:
/**
* Map of every partition Id to a set of its rack Ids, if they exist.
* If rack information is unavailable for all partitions, this is an empty map.
*/
private final Map<Integer, Set<String>> partitionRacks;
This is not being used at the moment as the consumer protocol does not support
rack aware assignments.
A heap dump from a group with 500 members, 2K subscribed topic partitions shows
654,400 bytes used for partitionRacks. The rest of the ConsumerGroup object
holds 822,860 bytes.
> Remove partitionRacks from TopicMetadata
> ----------------------------------------
>
> Key: KAFKA-17578
> URL: https://issues.apache.org/jira/browse/KAFKA-17578
> Project: Kafka
> Issue Type: Sub-task
> Reporter: Jeff Kim
> Priority: Major
> Labels: kip-848
>
> The ModernGroup#subscribedTopicMetadata takes too much memory due to
> partitionRacks:
> ```
> /**
> * Map of every partition Id to a set of its rack Ids, if they exist.
> * If rack information is unavailable for all partitions, this is an empty
> map.
> */
> private final Map<Integer, Set<String>> partitionRacks;
>
> ```
> This is not being used at the moment as the consumer protocol does not
> support rack aware assignments.
>
> A heap dump from a group with 500 members, 2K subscribed topic partitions
> shows
> 654,400 bytes used for partitionRacks. The rest of the ConsumerGroup object
> holds 822,860 bytes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
