Hi Jaikiran,
1) If you delete a topic with no outstanding sends, you will see one WARN
log entry and the topic is immediately removed from the metadata. If you
delete a topic with outstanding sends, you will see a sequence of WARN logs
until the outstanding send requests expire ("max.block.ms" property of the
producer).
2) The 0.10.0.0 release is imminent and I think it would be too late to
include this.
On Mon, May 16, 2016 at 11:59 AM, Jaikiran Pai <jai.forums2...@gmail.com>
wrote:
> Thank you for looking into this Rajini.
>
> A few questions related to this:
>
> 1) Looking at that patch, it looks like that WARN message will still be
> logged for a "few" times till that topic is removed from the Set maintained
> in the Metadata. Is my understanding correct? If yes, would it be possible
> to log that message at DEBUG (or lower level) if the producer send is
> initiated for a different topic and not for the topic which isn't existent?
> That was the application where this gets logged doesn't really have to be
> bothered by it. It does make sense to error out or WARN about the missing
> topic only if the send was issued against that topic.
>
> 2) I see that the JIRA https://issues.apache.org/jira/browse/KAFKA-2948
> is marked for 0.10.1.0. I haven't been following the Kafka release plans.
> Does this mean that there won't be a 0.9.x release any more? I see a
> 0.10.0.0 release candidate being voted up. Is 0.10.0.0 the next planned
> release? If yes, is there any chance this one can make into that release?
> Or will we have to wait for 0.10.1.0 for this to be available in a release?
> If that's the case, our application might have to change the log level
> setting in our logging configuration for NetworkClient to use ERROR level
> since this message completely messes up the entire application logs. I had
> a quick look at the NetworkClient code on 0.9.0 branch
> https://github.com/apache/kafka/blob/0.9.0/clients/src/main/java/org/apache/kafka/clients/NetworkClient.java
> and I think we should be fine if we bumped up the log level of that class
> in our logging config to be ERROR, since it doesn't log anything real
> useful below that level from that class.
>
>
> -Jaikiran
>
>
> On Monday 16 May 2016 02:30 PM, Rajini Sivaram wrote:
>
>> Sorry, that was the wrong JIRA.
>> https://issues.apache.org/jira/browse/KAFKA-2948 is the one which
>> addresses
>> this issue.
>>
>> On Mon, May 16, 2016 at 7:52 AM, Rajini Sivaram <
>> rajinisiva...@googlemail.com> wrote:
>>
>> There is an open JIRA for this issue (
>>> https://issues.apache.org/jira/browse/KAFKA-3065). The PR is quite old
>>> and needs rebasing. I will take a look at it today.
>>>
>>> On Sun, May 15, 2016 at 6:14 AM, Jaikiran Pai <jai.forums2...@gmail.com>
>>> wrote:
>>>
>>> Hello Kafka team,
>>>>
>>>>
>>>> We are using 0.9.0.1 (latest stable) of Kafka server and client
>>>> libraries. We use Java client for communicating with the Kafka
>>>> installation. Our simple application uses a single instance of
>>>> KafkaProducer (since the javadoc of that class states it's thread safe
>>>> and
>>>> recommended to be shared across the threads) for the lifetime of our
>>>> application runtime. We seem to be running into a potential issue in the
>>>> Kafka producer and the way it expects metadata for topics which it had
>>>> communicated before but are no longer around.
>>>>
>>>> The use case we have, where we run into this issue is as follows:
>>>>
>>>> 1. Our application is sent the name of the topic to which the
>>>> application
>>>> sends a message using the KafkaProducer
>>>> 2. The topics is short lived and after a while the topic is deleted via
>>>> Kafka tools externally
>>>> 3. Our application continues to run and the next time it receives a
>>>> request to send a message to _some other topic_, it ends up running
>>>> into an
>>>> issue where it endlessly floods the logs with messages:
>>>>
>>>> 10:17:53,245 WARN [NetworkClient] - Error while fetching metadata with
>>>> correlation id 122 :
>>>> {name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}
>>>> 10:17:53,347 WARN [NetworkClient] - Error while fetching metadata with
>>>> correlation id 123 :
>>>> {name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}
>>>> 10:17:53,449 WARN [NetworkClient] - Error while fetching metadata with
>>>> correlation id 124 :
>>>> {name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}
>>>>
>>>> It appears that the KafkaProducer wants to get hold of the metadata for
>>>> this topic which we deleted externally and which we have no intention to
>>>> communicate to anymore. These logs never stop, till we bring down the
>>>> application or close that producer instance.
>>>>
>>>> This looks like an issue to me since the producer should either be aware
>>>> that the topic got deleted and no longer request for metadata (unless
>>>> there
>>>> is an explicit call to send some message to it from the user
>>>> application)
>>>> or it should just ignore the fact that the metadata for this topic isn't
>>>> there and move on without logging these logs (unless, again, there is an
>>>> explicit call to send some message to the deleted topic, from the user
>>>> application).
>>>>
>>>> Looking at the code in the Kafka, it appears that the "topics" set gets
>>>> added with the topic name of the topic to which a communication is
>>>> established by the KafkaProducer. Once added, that topic name, never
>>>> gets
>>>> removed from that set for the lifetime of that producer, even in cases
>>>> like
>>>> these where the topic is deleted and never again used with that
>>>> producer.
>>>>
>>>> Do you think this is a bug in the Kafka code? I have a simple
>>>> application
>>>> which reproduces this easily on a 0.9.0.1 setup here
>>>> https://gist.github.com/jaikiran/45e9ce510c259267b28821b84105a25a.
>>>>
>>>> Let me know if you need more details about this.
>>>>
>>>> -Jaikiran
>>>>
>>>>
>>>
>>> --
>>> Regards,
>>>
>>> Rajini
>>>
>>>
>>
>>
>
--
Regards,
Rajini
