Oleg, I believe 0.9 producer gave you the control "max.block.ms" now
On Wed, Mar 30, 2016 at 5:31 AM, Oleg Zhurakousky < ozhurakou...@hortonworks.com> wrote: > I'll buy both 'back pressure' and 'block' argument, but what does it have > to do with the Future? Isn't that the main point of the Future - a > reference to an invocation that may or may not occur some time in the > future? Isn't that the purpose of the Future.get(..) to give user a choice > and chance to wait for completion, and that is where blocking and back > pressure is handled. > > The scary part is that not only send() can block indefinitely, if used > within users code it creates a deadlock in users code unless user does > something to avoid that. And what some of us do is invoke send() as > Callable returning yet another Future. Doesn't that seem awkward? Please > let me know if there is another way to deal with infinite block. > > Furthermore, the problem is similar to the one in consumer API where > Iterator was mapped to network service and hasNext() is also a blocking > call, but at least unlike send() there is a workaround property for > hasNext(). Could at least the same workaround be introduced to send()? > > Cheers > Oleg > > > On Mar 29, 2016, at 23:12, Dana Powers <dana.pow...@gmail.com> wrote: > > > > Somewhat of an aside, but you might note that the kafka producer is > > intended to block during send() as backpressure on memory allocation. > > This is admittedly different than blocking on metadata, but it is worth > > considering that the premise that send() should *never* block because > > it returns a Future seems fundamentally at odds with the current design. > > > > In any event, there is a configuration that you can tweak to set the max > > time the producer will spend blocking in send(): max.block.ms > > > > -Dana > > > > > >> On Tue, Mar 29, 2016 at 7:26 PM, Steven Wu <stevenz...@gmail.com> > wrote: > >> > >> I also agree that returning a Future should never block. I have brought > >> this up when 0.8.2 was first released for new Java producer. > >> > >> As Oleg said, KafkaProducer can also block if metadata is not fetched. > This > >> is probably more often than offline broker, because metadata is loaded > >> lazily when there is a first send attempt for the topic. In another > word, > >> first send to a topic will always block until metadata is fetched or > timed > >> out. > >> > >> We had to handle this in our wrapper code. Basically, we check if > metadata > >> not available, we put msg into a queue and drain the queue from a diff > >> thread. > >> > >> Thanks, > >> Steven > >> > >> > >> On Tue, Mar 29, 2016 at 4:59 AM, Oleg Zhurakousky < > >> ozhurakou...@hortonworks.com> wrote: > >> > >>> I agree and considering that send(..) method returns Future one would > >>> argue it must never block, otherwise what’s the point of returning > Future > >>> if you remove user’s ability to control how long are they willing to > wait > >>> and what to do when certain types of exception arise. Nevertheless it > >> does > >>> and it happens in the very first line of code: > >>> // first make sure the metadata for the topic is available > >>> waitOnMetadata(record.topic(), this.metadataFetchTimeoutMs); > >>> So I am curious as well as to what is the motivation for it as we’ve > seen > >>> the same symptoms multiple times. > >>> Cheers > >>> Oleg > >>> On Mar 29, 2016, at 4:31 AM, Paolo Patierno <ppatie...@live.com > <mailto: > >>> ppatie...@live.com>> wrote: > >>> > >>> Hello, > >>> as documentation says, the KafkaProducer.send() method is asynchronous > >> and > >>> it just returns immediately.I found out that it's not so true when the > >>> Kafka server it's trying to connect isn't online.Of course it happens > >> only > >>> on the first send() method invocation. It means that if the Kafka > server > >>> isn't reachable when my application starts for the first time, the > send() > >>> method isn't so asynchronous but it blocks.I know that it's trying to > >>> connect to the Kafka server but why it doesn't save the message into > the > >>> buffer and returns immediately ?Is it a behavior or a bug ? > >>> Thanks,Paolo > >>> > >>> Paolo PatiernoSenior Software Engineer > >>> > >>> > >>> Windows Embedded & IoTMicrosoft Azure Advisor > >>> Twitter : @ppatierno > >>> Linkedin : paolopatierno > >>> Blog : DevExperienceBlog : Embedded101 > >> >
