Rajini Sivaram created KAFKA-2948:
-------------------------------------
Summary: Kafka producer does not cope well with topic deletions
Key: KAFKA-2948
URL: https://issues.apache.org/jira/browse/KAFKA-2948
Project: Kafka
Issue Type: Bug
Components: producer
Affects Versions: 0.9.0.0
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
Kafka producer gets metadata for topics when send is invoked and thereafter it
attempts to keep the metadata up-to-date without any explicit requests from the
client. This works well in static environments, but when topics are added or
deleted, list of topics in Metadata grows but never shrinks. Apart from being a
memory leak, this results in constant requests for metadata for deleted topics.
We are running into this issue with the Confluent REST server where topic
deletion from tests are filling up logs with warnings about unknown topics.
Auto-create is turned off in our Kafka cluster.
I am happy to provide a fix, but am not sure what the right fix is. Does it
make sense to remove topics from the metadata list when
UNKNOWN_TOPIC_OR_PARTITION response is received if there are no outstanding
sends? It doesn't look very straightforward to do this, so any alternative
suggestions are welcome.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
