[
https://issues.apache.org/jira/browse/KAFKA-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13896775#comment-13896775
]
Jay Kreps commented on KAFKA-1238:
----------------------------------
The way metadata works is the following:
1. When someone requests metadata using Metadata.java that we don't have we we
flag that we need to update metadata.
2. Once that flag is set the sender will update metadata until the flag is
unset. The flag is unset whenever the cluster is updated. There is a metadata
fetch backoff to prevent polling too quickly.
3. The metadata update described in (1) and (2) will repeat until we get
metadata for the topic we requested or we time out.
So if we don't update the cluster metadata then the metadata request just
repeats as soon as the backoff expires.
So that is why it works. But why is this the right logic? Why don't we check
the error code? The reason is simple, if we ever remove ALL the nodes from the
cluster metadata then we lose the ability to continue to update metadata
because we update metadata using a random node in our metadata. So if we update
our metadata to have no nodes, then we have no one to update metadata with.
The server's behavior is that it will always include whatever brokers are
referenced by the metadata in the response. So if you give a single topic and
that topic doesn't exist you will get no brokers back and an error code for
that topic. But regardless of the reason if the cluster metadata comes back
with no brokers we should not use it (the only reason I know this can happen is
an error, but it basically doesn't matter what error and perhaps there are
other conditions).
> New producer hangs in a loop detecting metadata for auto created topics
> -----------------------------------------------------------------------
>
> Key: KAFKA-1238
> URL: https://issues.apache.org/jira/browse/KAFKA-1238
> Project: Kafka
> Issue Type: Bug
> Components: producer
> Reporter: Neha Narkhede
> Assignee: Jay Kreps
> Priority: Critical
> Attachments: KAFKA-1238-v1.patch, KAFKA-1238.patch
>
>
> New producer hangs in a loop detecting metadata for auto created topics -
> java.lang.IllegalStateException: No known nodes.
> at kafka.common.Cluster.nextNode(Cluster.java:97)
> at
> kafka.clients.producer.internals.Sender.maybeMetadataRequest(Sender.java:154)
> at kafka.clients.producer.internals.Sender.run(Sender.java:120)
> at kafka.clients.producer.internals.Sender.run(Sender.java:84)
> at java.lang.Thread.run(Thread.java:695)
> java.lang.IllegalStateException: No known nodes.
> at kafka.common.Cluster.nextNode(Cluster.java:97)
> at
> kafka.clients.producer.internals.Sender.maybeMetadataRequest(Sender.java:154)
> at kafka.clients.producer.internals.Sender.run(Sender.java:120)
> at kafka.clients.producer.internals.Sender.run(Sender.java:84)
> at java.lang.Thread.run(Thread.java:695)
> java.lang.IllegalStateException: No known nodes.
> at kafka.common.Cluster.nextNode(Cluster.java:97)
> at
> kafka.clients.producer.internals.Sender.maybeMetadataRequest(Sender.java:154)
> at kafka.clients.producer.internals.Sender.run(Sender.java:120)
> at kafka.clients.producer.internals.Sender.run(Sender.java:84)
> at java.lang.Thread.run(Thread.java:695)
> java.lang.IllegalStateException: No known nodes.
> at kafka.common.Cluster.nextNode(Cluster.java:97)
> at
> kafka.clients.producer.internals.Sender.maybeMetadataRequest(Sender.java:154)
> at kafka.clients.producer.internals.Sender.run(Sender.java:120)
> at kafka.clients.producer.internals.Sender.run(Sender.java:84)
> at java.lang.Thread.run(Thread.java:695)
> java.lang.IllegalStateException: No known nodes.
> at kafka.common.Cluster.nextNode(Cluster.java:97)
> at
> kafka.clients.producer.internals.Sender.maybeMetadataRequest(Sender.java:154)
> at kafka.clients.producer.internals.Sender.run(Sender.java:120)
> at kafka.clients.producer.internals.Sender.run(Sender.java:84)
> at java.lang.Thread.run(Thread.java:695)
> java.lang.IllegalStateException: No known nodes.
> at kafka.common.Cluster.nextNode(Cluster.java:97)
> at
> kafka.clients.producer.internals.Sender.maybeMetadataRequest(Sender.java:154)
> at kafka.clients.producer.internals.Sender.run(Sender.java:120)
> at kafka.clients.producer.internals.Sender.run(Sender.java:84)
> at java.lang.Thread.run(Thread.java:695)
> The producer needs to be restarted to start sending data to the auto created
> topic
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
