Thank you for looking into this Rajini.
A few questions related to this:
1) Looking at that patch, it looks like that WARN message will still be
logged for a "few" times till that topic is removed from the Set
maintained in the Metadata. Is my understanding correct? If yes, would
it be possible to log that message at DEBUG (or lower level) if the
producer send is initiated for a different topic and not for the topic
which isn't existent? That was the application where this gets logged
doesn't really have to be bothered by it. It does make sense to error
out or WARN about the missing topic only if the send was issued against
that topic.
2) I see that the JIRA https://issues.apache.org/jira/browse/KAFKA-2948
is marked for 0.10.1.0. I haven't been following the Kafka release
plans. Does this mean that there won't be a 0.9.x release any more? I
see a 0.10.0.0 release candidate being voted up. Is 0.10.0.0 the next
planned release? If yes, is there any chance this one can make into that
release? Or will we have to wait for 0.10.1.0 for this to be available
in a release? If that's the case, our application might have to change
the log level setting in our logging configuration for NetworkClient to
use ERROR level since this message completely messes up the entire
application logs. I had a quick look at the NetworkClient code on 0.9.0
branch
https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java
and I think we should be fine if we bumped up the log level of that
class in our logging config to be ERROR, since it doesn't log anything
real useful below that level from that class.
-Jaikiran
On Monday 16 May 2016 02:30 PM, Rajini Sivaram wrote:
Sorry, that was the wrong JIRA.
https://issues.apache.org/jira/browse/KAFKA-2948 is the one which addresses
this issue.
On Mon, May 16, 2016 at 7:52 AM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:
There is an open JIRA for this issue (
https://issues.apache.org/jira/browse/KAFKA-3065). The PR is quite old
and needs rebasing. I will take a look at it today.
On Sun, May 15, 2016 at 6:14 AM, Jaikiran Pai <jai.forums2...@gmail.com>
wrote:
Hello Kafka team,
We are using 0.9.0.1 (latest stable) of Kafka server and client
libraries. We use Java client for communicating with the Kafka
installation. Our simple application uses a single instance of
KafkaProducer (since the javadoc of that class states it's thread safe and
recommended to be shared across the threads) for the lifetime of our
application runtime. We seem to be running into a potential issue in the
Kafka producer and the way it expects metadata for topics which it had
communicated before but are no longer around.
The use case we have, where we run into this issue is as follows:
1. Our application is sent the name of the topic to which the application
sends a message using the KafkaProducer
2. The topics is short lived and after a while the topic is deleted via
Kafka tools externally
3. Our application continues to run and the next time it receives a
request to send a message to _some other topic_, it ends up running into an
issue where it endlessly floods the logs with messages:
10:17:53,245 WARN [NetworkClient] - Error while fetching metadata with
correlation id 122 :
{name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}
10:17:53,347 WARN [NetworkClient] - Error while fetching metadata with
correlation id 123 :
{name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}
10:17:53,449 WARN [NetworkClient] - Error while fetching metadata with
correlation id 124 :
{name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}
It appears that the KafkaProducer wants to get hold of the metadata for
this topic which we deleted externally and which we have no intention to
communicate to anymore. These logs never stop, till we bring down the
application or close that producer instance.
This looks like an issue to me since the producer should either be aware
that the topic got deleted and no longer request for metadata (unless there
is an explicit call to send some message to it from the user application)
or it should just ignore the fact that the metadata for this topic isn't
there and move on without logging these logs (unless, again, there is an
explicit call to send some message to the deleted topic, from the user
application).
Looking at the code in the Kafka, it appears that the "topics" set gets
added with the topic name of the topic to which a communication is
established by the KafkaProducer. Once added, that topic name, never gets
removed from that set for the lifetime of that producer, even in cases like
these where the topic is deleted and never again used with that producer.
Do you think this is a bug in the Kafka code? I have a simple application
which reproduces this easily on a 0.9.0.1 setup here
https://gist.github.com/jaikiran/45e9ce510c259267b28821b84105a25a.
Let me know if you need more details about this.
-Jaikiran
--
Regards,
Rajini
