Thanks Jiangjie. This is helpful. Adding to what you have mentioned, I can think of one more scenario which may not be very rare. Say, the application is rebooted and the Kafka brokers registered in the producer are not reachable ( could be due to network issues or those brokers are actually down ). Since, no metadata is available the send will block. Right?
On Wed, May 13, 2015 at 10:51 AM, Jiangjie Qin <j...@linkedin.com.invalid> wrote: > > Application will not block on each metadata refresh or metadata is > expired. > Application will only be blocked when > 1. It sends the first message to a topic (only for that single message), or > 2. The topic has been deleted from broker thus refreshed metadata loses > the topic info (which is pretty rare). > > So I think the async here might mean a little bit different. It means when > you send first message to a topic, you wait till you know the topic exist, > after that point it is async. > It is very low chance that your application will block on send. If it is > then something probably really went wrong and needs immediate attention. > > Thanks. > > Jiangjie (Becket) Qin > > On 5/12/15, 5:08 PM, "Rendy Bambang Junior" <rendy.b.jun...@gmail.com> > wrote: > > >Thank you for the clarification. > > > >I think I agree with Mohit. Sometime blocking on logging is not acceptable > >by nature of application who uses kafka. > > > >Yes it is not blocking when metadata is still available. But application > >will be blocked once metada is expired. > > > >It might be handled by application, by implementing async call when do > >send() and manage buffer and async timeout internally, but it makes async > >feature in kafka producer has less meaning. > > > >Sorry if my understanding is incorrect. > > > >Rendy > >On May 13, 2015 6:59 AM, "Jiangjie Qin" <j...@linkedin.com.invalid> > wrote: > > > >> Send() will only block if the metadata is *not available* for the topic. > >> It won't block if metadata there is stale. The metadata refresh is async > >> to send(). However, if you send the message to a topic for the first > >>time, > >> send() will trigger a metadata refresh and block until it has metadata > >>for > >> that topic. > >> > >> Jiangjie (Becket) Qin > >> > >> On 5/12/15, 12:58 PM, "Magnus Edenhill" <mag...@edenhill.se> wrote: > >> > >> >I completely agree with Mohit, an application should not have to know > >>or > >> >care about > >> >producer implementation internals. > >> >Given a message and its delivery constraints (produce retry count and > >> >timeout) the producer > >> >should hide any temporal failures until the message is succesfully > >> >delivered, a permanent > >> >error is encountered or the constraints are hit. > >> >This should also include internal start up sequencing, such as metadata > >> >retrieval. > >> > > >> > > >> > > >> >2015-05-12 21:22 GMT+02:00 Mohit Gupta <success.mohit.gu...@gmail.com > >: > >> > > >> >> I could not follow the reasoning behind blocking the send method if > >>the > >> >> metadata is not up-to-date. Though, I see that it as per design, it > >> >> requires the metadata to batch the message into appropriate > >> >>topicPartition > >> >> queue. Also, if the metadata could not be updated in the specified > >> >> interval, it throws an exception and the message is not queued to be > >> >> retried once the brokers are up. > >> >> > >> >> Should it not be that messages are buffered in another queue ( up-to > >>a > >> >> limit ) if the brokers are down and retried later? > >> >> Is it not a general use case to require producer to be asynchronous > >>in > >> >>all > >> >> the scenarios? > >> >> > >> >> > >> >> On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat < > >> >> gharatmayures...@gmail.com> wrote: > >> >> > >> >> > The way it works I suppose is that, the producer will do > >> >>fetchMetadata, > >> >> if > >> >> > the last fetched metadata is stale (the refresh interval has > >>expired) > >> >>or > >> >> if > >> >> > it is not able to send data to a particular broker in its current > >> >> metadata > >> >> > (This might happen in some cases like if the leader moves). > >> >> > > >> >> > It cannot produce without having the right metadata. > >> >> > > >> >> > Thanks, > >> >> > > >> >> > Mayuresh > >> >> > > >> >> > On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin > >> >><j...@linkedin.com.invalid > >> >> > > >> >> > wrote: > >> >> > > >> >> > > That¹s right. Send() will first try to get metadata of a topic, > >>that > >> >> is a > >> >> > > blocking operation. > >> >> > > > >> >> > > On 5/12/15, 2:48 AM, "Rendy Bambang Junior" > >> >><rendy.b.jun...@gmail.com> > >> >> > > wrote: > >> >> > > > >> >> > > >Hi, sorry if my understanding is incorrect. > >> >> > > > > >> >> > > >I am integrating kafka producer with application, when i try to > >> >> shutdown > >> >> > > >all kafka broker (preparing for prod env) I notice that 'send' > >> >>method > >> >> is > >> >> > > >blocking. > >> >> > > > > >> >> > > >Is new producer fetch metadata not async? > >> >> > > > > >> >> > > >Rendy > >> >> > > > >> >> > > > >> >> > > >> >> > > >> >> > -- > >> >> > -Regards, > >> >> > Mayuresh R. Gharat > >> >> > (862) 250-7125 > >> >> > > >> >> > >> >> > >> >> > >> >> -- > >> >> Best Regards, > >> >> > >> >> Mohit Gupta > >> >> > >> > >> > > -- Best Regards, Mohit Gupta
