[
https://issues.apache.org/jira/browse/KAFKA-2948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15046724#comment-15046724
]
Rajini Sivaram commented on KAFKA-2948:
---------------------------------------
[~becket_qin] Thank you for your feedback. The fix that we are testing at the
moment removes topics with `UNKNOWN_TOPIC_OR_PARTITION` error from the metadata
set when the error is received in a response, but re-adds it when metadata is
requested for the topic (eg. producer waiting for metadata to send a message).
This ensures that the request is retried when required, but not when the topic
is no longer in use.
TTL sounds like a better option to remove not just deleted topics, but also any
topic that is no longer being used. My only concern is that deleted topics
would remain in the list for a longer period of time with a lot of warnings in
the logs as metadata requests are retried. I could combine the current fix and
TTL if required to avoid this, but I will try out TTL on its own first with the
REST service and see how that goes.
> Kafka producer does not cope well with topic deletions
> ------------------------------------------------------
>
> Key: KAFKA-2948
> URL: https://issues.apache.org/jira/browse/KAFKA-2948
> Project: Kafka
> Issue Type: Bug
> Components: producer
> Affects Versions: 0.9.0.0
> Reporter: Rajini Sivaram
> Assignee: Rajini Sivaram
>
> Kafka producer gets metadata for topics when send is invoked and thereafter
> it attempts to keep the metadata up-to-date without any explicit requests
> from the client. This works well in static environments, but when topics are
> added or deleted, list of topics in Metadata grows but never shrinks. Apart
> from being a memory leak, this results in constant requests for metadata for
> deleted topics.
> We are running into this issue with the Confluent REST server where topic
> deletion from tests are filling up logs with warnings about unknown topics.
> Auto-create is turned off in our Kafka cluster.
> I am happy to provide a fix, but am not sure what the right fix is. Does it
> make sense to remove topics from the metadata list when
> UNKNOWN_TOPIC_OR_PARTITION response is received if there are no outstanding
> sends? It doesn't look very straightforward to do this, so any alternative
> suggestions are welcome.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
