Isn’t the producer part of the application? The metadata is stored in memory. If the application rebooted (process restarted), all the metadata will be gone.
Jiangjie (Becket) Qin On 5/13/15, 9:54 AM, "Mohit Gupta" <success.mohit.gu...@gmail.com> wrote: >I meant the producer. ( i.e. application using the producer api to push >messages into kafka ) . > >On Wed, May 13, 2015 at 10:20 PM, Mayuresh Gharat < >gharatmayures...@gmail.com> wrote: > >> By application rebooting, do you mean you bounce the brokers? >> >> Thanks, >> >> Mayuresh >> >> On Wed, May 13, 2015 at 4:06 AM, Mohit Gupta < >> success.mohit.gu...@gmail.com> >> wrote: >> >> > Thanks Jiangjie. This is helpful. >> > >> > Adding to what you have mentioned, I can think of one more scenario >>which >> > may not be very rare. >> > Say, the application is rebooted and the Kafka brokers registered in >>the >> > producer are not reachable ( could be due to network issues or those >> > brokers are actually down ). Since, no metadata is available the send >> will >> > block. Right? >> > >> > On Wed, May 13, 2015 at 10:51 AM, Jiangjie Qin >><j...@linkedin.com.invalid >> > >> > wrote: >> > >> > > >> > > Application will not block on each metadata refresh or metadata is >> > > expired. >> > > Application will only be blocked when >> > > 1. It sends the first message to a topic (only for that single >> message), >> > or >> > > 2. The topic has been deleted from broker thus refreshed metadata >>loses >> > > the topic info (which is pretty rare). >> > > >> > > So I think the async here might mean a little bit different. It >>means >> > when >> > > you send first message to a topic, you wait till you know the topic >> > exist, >> > > after that point it is async. >> > > It is very low chance that your application will block on send. If >>it >> is >> > > then something probably really went wrong and needs immediate >> attention. >> > > >> > > Thanks. >> > > >> > > Jiangjie (Becket) Qin >> > > >> > > On 5/12/15, 5:08 PM, "Rendy Bambang Junior" >><rendy.b.jun...@gmail.com> >> > > wrote: >> > > >> > > >Thank you for the clarification. >> > > > >> > > >I think I agree with Mohit. Sometime blocking on logging is not >> > acceptable >> > > >by nature of application who uses kafka. >> > > > >> > > >Yes it is not blocking when metadata is still available. But >> application >> > > >will be blocked once metada is expired. >> > > > >> > > >It might be handled by application, by implementing async call >>when do >> > > >send() and manage buffer and async timeout internally, but it makes >> > async >> > > >feature in kafka producer has less meaning. >> > > > >> > > >Sorry if my understanding is incorrect. >> > > > >> > > >Rendy >> > > >On May 13, 2015 6:59 AM, "Jiangjie Qin" <j...@linkedin.com.invalid> >> > > wrote: >> > > > >> > > >> Send() will only block if the metadata is *not available* for the >> > topic. >> > > >> It won't block if metadata there is stale. The metadata refresh >>is >> > async >> > > >> to send(). However, if you send the message to a topic for the >>first >> > > >>time, >> > > >> send() will trigger a metadata refresh and block until it has >> metadata >> > > >>for >> > > >> that topic. >> > > >> >> > > >> Jiangjie (Becket) Qin >> > > >> >> > > >> On 5/12/15, 12:58 PM, "Magnus Edenhill" <mag...@edenhill.se> >>wrote: >> > > >> >> > > >> >I completely agree with Mohit, an application should not have to >> know >> > > >>or >> > > >> >care about >> > > >> >producer implementation internals. >> > > >> >Given a message and its delivery constraints (produce retry >>count >> and >> > > >> >timeout) the producer >> > > >> >should hide any temporal failures until the message is >>succesfully >> > > >> >delivered, a permanent >> > > >> >error is encountered or the constraints are hit. >> > > >> >This should also include internal start up sequencing, such as >> > metadata >> > > >> >retrieval. >> > > >> > >> > > >> > >> > > >> > >> > > >> >2015-05-12 21:22 GMT+02:00 Mohit Gupta < >> > success.mohit.gu...@gmail.com >> > > >: >> > > >> > >> > > >> >> I could not follow the reasoning behind blocking the send >>method >> if >> > > >>the >> > > >> >> metadata is not up-to-date. Though, I see that it as per >>design, >> it >> > > >> >> requires the metadata to batch the message into appropriate >> > > >> >>topicPartition >> > > >> >> queue. Also, if the metadata could not be updated in the >> specified >> > > >> >> interval, it throws an exception and the message is not >>queued to >> > be >> > > >> >> retried once the brokers are up. >> > > >> >> >> > > >> >> Should it not be that messages are buffered in another queue ( >> > up-to >> > > >>a >> > > >> >> limit ) if the brokers are down and retried later? >> > > >> >> Is it not a general use case to require producer to be >> asynchronous >> > > >>in >> > > >> >>all >> > > >> >> the scenarios? >> > > >> >> >> > > >> >> >> > > >> >> On Tue, May 12, 2015 at 10:54 PM, Mayuresh Gharat < >> > > >> >> gharatmayures...@gmail.com> wrote: >> > > >> >> >> > > >> >> > The way it works I suppose is that, the producer will do >> > > >> >>fetchMetadata, >> > > >> >> if >> > > >> >> > the last fetched metadata is stale (the refresh interval has >> > > >>expired) >> > > >> >>or >> > > >> >> if >> > > >> >> > it is not able to send data to a particular broker in its >> current >> > > >> >> metadata >> > > >> >> > (This might happen in some cases like if the leader moves). >> > > >> >> > >> > > >> >> > It cannot produce without having the right metadata. >> > > >> >> > >> > > >> >> > Thanks, >> > > >> >> > >> > > >> >> > Mayuresh >> > > >> >> > >> > > >> >> > On Tue, May 12, 2015 at 10:09 AM, Jiangjie Qin >> > > >> >><j...@linkedin.com.invalid >> > > >> >> > >> > > >> >> > wrote: >> > > >> >> > >> > > >> >> > > That¹s right. Send() will first try to get metadata of a >> topic, >> > > >>that >> > > >> >> is a >> > > >> >> > > blocking operation. >> > > >> >> > > >> > > >> >> > > On 5/12/15, 2:48 AM, "Rendy Bambang Junior" >> > > >> >><rendy.b.jun...@gmail.com> >> > > >> >> > > wrote: >> > > >> >> > > >> > > >> >> > > >Hi, sorry if my understanding is incorrect. >> > > >> >> > > > >> > > >> >> > > >I am integrating kafka producer with application, when i >>try >> > to >> > > >> >> shutdown >> > > >> >> > > >all kafka broker (preparing for prod env) I notice that >> 'send' >> > > >> >>method >> > > >> >> is >> > > >> >> > > >blocking. >> > > >> >> > > > >> > > >> >> > > >Is new producer fetch metadata not async? >> > > >> >> > > > >> > > >> >> > > >Rendy >> > > >> >> > > >> > > >> >> > > >> > > >> >> > >> > > >> >> > >> > > >> >> > -- >> > > >> >> > -Regards, >> > > >> >> > Mayuresh R. Gharat >> > > >> >> > (862) 250-7125 >> > > >> >> > >> > > >> >> >> > > >> >> >> > > >> >> >> > > >> >> -- >> > > >> >> Best Regards, >> > > >> >> >> > > >> >> Mohit Gupta >> > > >> >> >> > > >> >> > > >> >> > > >> > > >> > >> > >> > -- >> > Best Regards, >> > >> > Mohit Gupta >> > >> >> >> >> -- >> -Regards, >> Mayuresh R. Gharat >> (862) 250-7125 >> > > > >-- >Best Regards, > >Mohit Gupta
