[
https://issues.apache.org/jira/browse/KAFKA-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15214916#comment-15214916
]
Jiangjie Qin commented on KAFKA-3334:
-------------------------------------
Not sure if I understand the issue. When sending message to a newly
auto-created topic, the producer will:
1. Send a TopicMetadataRequest regarding the topic
2. Wait for the topic metadata (producer will retry metadata refresh, hopefully
the topic will be created and metadata will be available before times out)
3. The producer will append the message to the accumulator
4. The sender thread will send the message to the leader in the metadata which
is acquired in step 2. The message will be successfully sent at this point, or,
5. There is a slight chance that the leader hasn't received the
LeaderAndIsrRequest from the controller so it might think the partition does
not exist and return a UnknownTopicOrPartitionException.
6. Producer will retry sending the message and the message will eventually be
successfully sent to the leader.
It seems although there are a few exceptions seen by the producer, the message
will eventually be sent. If the message did not got sent out, user will either
receive a TimeoutException from producer.send() or some other exceptions in the
callback.
I am not sure why we would silently fail the first message?
> First message on new topic not actually being sent, no exception thrown
> -----------------------------------------------------------------------
>
> Key: KAFKA-3334
> URL: https://issues.apache.org/jira/browse/KAFKA-3334
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.9.0.0
> Environment: Linux, Java
> Reporter: Aleksandar Stojadinovic
> Assignee: Ashish K Singh
> Fix For: 0.10.1.0
>
>
> Although I've seen this issue pop around the internet in a few forms, I'm not
> sure it is yet properly fixed.
> When publishing to a new topic, with auto create-enabled, the java client
> (0.9.0) shows this WARN message in the log, and the message is not sent
> obviously:
> org.apache.kafka.clients.NetworkClient - Error while fetching metadata with
> correlation id 0 : {file.topic=LEADER_NOT_AVAILABLE}
> In the meantime I see in the console the message that a log for partition is
> created. The next messages are patched through normally, but the first one is
> never sent. No exception is ever thrown, either by calling get on the future,
> or with the async usage, like everything is perfect.
> I notice when I leave my application blocked on the get call, in the
> debugger, then the message may be processed, but with significant delay. This
> is consistent with another issue I found for the python client. Also, if I
> call partitionsFor previously, the topic is created and the message is sent.
> But it seems silly to call it every time, just to mitigate this issue.
> {code}
> Future<RecordMetadata> recordMetadataFuture = producer.send(new
> ProducerRecord<>(topic, key, file));
> RecordMetadata recordMetadata = recordMetadataFuture.get(30,
> TimeUnit.SECONDS);
> {code}
> I hope I'm clear enough.
> Related similar (but not same) issues:
> https://issues.apache.org/jira/browse/KAFKA-1124
> https://github.com/dpkp/kafka-python/issues/150
> http://stackoverflow.com/questions/35187933/how-to-resolve-leader-not-available-kafka-error-when-trying-to-consume
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
