Send() will only block if the metadata is *not available* for the topic. It won’t block if metadata there is stale. The metadata refresh is async to send(). However, if you send the message to a topic for the first time, send() will trigger a metadata refresh and block until it has metadata for that topic.
Jiangjie (Becket) Qin On 5/12/15, 12:58 PM, "Magnus Edenhill" <mag...@edenhill.se> wrote: >I completely agree with Mohit, an application should not have to know or >care about >producer implementation internals. >Given a message and its delivery constraints (produce retry count and >timeout) the producer >should hide any temporal failures until the message is succesfully >delivered, a permanent >error is encountered or the constraints are hit. >This should also include internal start up sequencing, such as metadata >retrieval. > > > >2015-05-12 21:22 GMT+02:00 Mohit Gupta <success.mohit.gu...@gmail.com>: > >> I could not follow the reasoning behind blocking the send method if the >> metadata is not up-to-date. Though, I see that it as per design, it >> requires the metadata to batch the message into appropriate >>topicPartition >> queue. Also, if the metadata could not be updated in the specified >> interval, it throws an exception and the message is not queued to be >> retried once the brokers are up. >> >> Should it not be that messages are buffered in another queue ( up-to a >> limit ) if the brokers are down and retried later? >> Is it not a general use case to require producer to be asynchronous in >>all >> the scenarios? >> >> >> On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat < >> gharatmayures...@gmail.com> wrote: >> >> > The way it works I suppose is that, the producer will do >>fetchMetadata, >> if >> > the last fetched metadata is stale (the refresh interval has expired) >>or >> if >> > it is not able to send data to a particular broker in its current >> metadata >> > (This might happen in some cases like if the leader moves). >> > >> > It cannot produce without having the right metadata. >> > >> > Thanks, >> > >> > Mayuresh >> > >> > On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin >><j...@linkedin.com.invalid >> > >> > wrote: >> > >> > > That¹s right. Send() will first try to get metadata of a topic, that >> is a >> > > blocking operation. >> > > >> > > On 5/12/15, 2:48 AM, "Rendy Bambang Junior" >><rendy.b.jun...@gmail.com> >> > > wrote: >> > > >> > > >Hi, sorry if my understanding is incorrect. >> > > > >> > > >I am integrating kafka producer with application, when i try to >> shutdown >> > > >all kafka broker (preparing for prod env) I notice that 'send' >>method >> is >> > > >blocking. >> > > > >> > > >Is new producer fetch metadata not async? >> > > > >> > > >Rendy >> > > >> > > >> > >> > >> > -- >> > -Regards, >> > Mayuresh R. Gharat >> > (862) 250-7125 >> > >> >> >> >> -- >> Best Regards, >> >> Mohit Gupta >>
