Application will not block on each metadata refresh or metadata is expired. Application will only be blocked when 1. It sends the first message to a topic (only for that single message), or 2. The topic has been deleted from broker thus refreshed metadata loses the topic info (which is pretty rare).
So I think the async here might mean a little bit different. It means when you send first message to a topic, you wait till you know the topic exist, after that point it is async. It is very low chance that your application will block on send. If it is then something probably really went wrong and needs immediate attention. Thanks. Jiangjie (Becket) Qin On 5/12/15, 5:08 PM, "Rendy Bambang Junior" <rendy.b.jun...@gmail.com> wrote: >Thank you for the clarification. > >I think I agree with Mohit. Sometime blocking on logging is not acceptable >by nature of application who uses kafka. > >Yes it is not blocking when metadata is still available. But application >will be blocked once metada is expired. > >It might be handled by application, by implementing async call when do >send() and manage buffer and async timeout internally, but it makes async >feature in kafka producer has less meaning. > >Sorry if my understanding is incorrect. > >Rendy >On May 13, 2015 6:59 AM, "Jiangjie Qin" <j...@linkedin.com.invalid> wrote: > >> Send() will only block if the metadata is *not available* for the topic. >> It won’t block if metadata there is stale. The metadata refresh is async >> to send(). However, if you send the message to a topic for the first >>time, >> send() will trigger a metadata refresh and block until it has metadata >>for >> that topic. >> >> Jiangjie (Becket) Qin >> >> On 5/12/15, 12:58 PM, "Magnus Edenhill" <mag...@edenhill.se> wrote: >> >> >I completely agree with Mohit, an application should not have to know >>or >> >care about >> >producer implementation internals. >> >Given a message and its delivery constraints (produce retry count and >> >timeout) the producer >> >should hide any temporal failures until the message is succesfully >> >delivered, a permanent >> >error is encountered or the constraints are hit. >> >This should also include internal start up sequencing, such as metadata >> >retrieval. >> > >> > >> > >> >2015-05-12 21:22 GMT+02:00 Mohit Gupta <success.mohit.gu...@gmail.com>: >> > >> >> I could not follow the reasoning behind blocking the send method if >>the >> >> metadata is not up-to-date. Though, I see that it as per design, it >> >> requires the metadata to batch the message into appropriate >> >>topicPartition >> >> queue. Also, if the metadata could not be updated in the specified >> >> interval, it throws an exception and the message is not queued to be >> >> retried once the brokers are up. >> >> >> >> Should it not be that messages are buffered in another queue ( up-to >>a >> >> limit ) if the brokers are down and retried later? >> >> Is it not a general use case to require producer to be asynchronous >>in >> >>all >> >> the scenarios? >> >> >> >> >> >> On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat < >> >> gharatmayures...@gmail.com> wrote: >> >> >> >> > The way it works I suppose is that, the producer will do >> >>fetchMetadata, >> >> if >> >> > the last fetched metadata is stale (the refresh interval has >>expired) >> >>or >> >> if >> >> > it is not able to send data to a particular broker in its current >> >> metadata >> >> > (This might happen in some cases like if the leader moves). >> >> > >> >> > It cannot produce without having the right metadata. >> >> > >> >> > Thanks, >> >> > >> >> > Mayuresh >> >> > >> >> > On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin >> >><j...@linkedin.com.invalid >> >> > >> >> > wrote: >> >> > >> >> > > That¹s right. Send() will first try to get metadata of a topic, >>that >> >> is a >> >> > > blocking operation. >> >> > > >> >> > > On 5/12/15, 2:48 AM, "Rendy Bambang Junior" >> >><rendy.b.jun...@gmail.com> >> >> > > wrote: >> >> > > >> >> > > >Hi, sorry if my understanding is incorrect. >> >> > > > >> >> > > >I am integrating kafka producer with application, when i try to >> >> shutdown >> >> > > >all kafka broker (preparing for prod env) I notice that 'send' >> >>method >> >> is >> >> > > >blocking. >> >> > > > >> >> > > >Is new producer fetch metadata not async? >> >> > > > >> >> > > >Rendy >> >> > > >> >> > > >> >> > >> >> > >> >> > -- >> >> > -Regards, >> >> > Mayuresh R. Gharat >> >> > (862) 250-7125 >> >> > >> >> >> >> >> >> >> >> -- >> >> Best Regards, >> >> >> >> Mohit Gupta >> >> >> >>
