[
https://issues.apache.org/jira/browse/KAFKA-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15345164#comment-15345164
]
ASF GitHub Bot commented on KAFKA-3892:
---------------------------------------
GitHub user iamnoah opened a pull request:
https://github.com/apache/kafka/pull/1542
KAFKA-3892 prune metadata response to subscribed topics
Rebased from PR #1541
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/spredfast/kafka-1 remove-extra-metadata-trunk
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/1542.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1542
----
commit b1af18d06a18080f8f8fd1535ea99807c55cbf50
Author: Noah Sloan <[email protected]>
Date: 2016-06-22T20:10:35Z
KAFKA-3892 prune metadata response to subscribed topics
----
> Clients retain metadata for non-subscribed topics
> -------------------------------------------------
>
> Key: KAFKA-3892
> URL: https://issues.apache.org/jira/browse/KAFKA-3892
> Project: Kafka
> Issue Type: Bug
> Components: clients
> Affects Versions: 0.9.0.1
> Reporter: Noah Sloan
>
> After upgrading to 0.9.0.1 from 0.8.2 (and adopting the new consumer and
> producer classes,) we noticed services with small heap crashing due to
> OutOfMemoryErrors. These services contained many producers and consumers (~20
> total) and were connected to brokers with >2000 topics and over 10k
> partitions. Heap dumps revealed that each client had 3.3MB of Metadata
> retained in their Cluster, with references to topics that were not being
> produced or subscribed to. While the services were running with 128MB of heap
> prior to the upgrade, we to had increased max heap to 200MB to accommodate
> all the extra data.
> While this is not technically a memory leak, it does impose a significant
> overhead on clients when connected to a large cluster.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
