I'll buy both 'back pressure' and 'block' argument, but what does it have to do with the Future? Isn't that the main point of the Future - a reference to an invocation that may or may not occur some time in the future? Isn't that the purpose of the Future.get(..) to give user a choice and chance to wait for completion, and that is where blocking and back pressure is handled.
The scary part is that not only send() can block indefinitely, if used within users code it creates a deadlock in users code unless user does something to avoid that. And what some of us do is invoke send() as Callable returning yet another Future. Doesn't that seem awkward? Please let me know if there is another way to deal with infinite block. Furthermore, the problem is similar to the one in consumer API where Iterator was mapped to network service and hasNext() is also a blocking call, but at least unlike send() there is a workaround property for hasNext(). Could at least the same workaround be introduced to send()? Cheers Oleg > On Mar 29, 2016, at 23:12, Dana Powers <dana.pow...@gmail.com> wrote: > > Somewhat of an aside, but you might note that the kafka producer is > intended to block during send() as backpressure on memory allocation. > This is admittedly different than blocking on metadata, but it is worth > considering that the premise that send() should *never* block because > it returns a Future seems fundamentally at odds with the current design. > > In any event, there is a configuration that you can tweak to set the max > time the producer will spend blocking in send(): max.block.ms > > -Dana > > >> On Tue, Mar 29, 2016 at 7:26 PM, Steven Wu <stevenz...@gmail.com> wrote: >> >> I also agree that returning a Future should never block. I have brought >> this up when 0.8.2 was first released for new Java producer. >> >> As Oleg said, KafkaProducer can also block if metadata is not fetched. This >> is probably more often than offline broker, because metadata is loaded >> lazily when there is a first send attempt for the topic. In another word, >> first send to a topic will always block until metadata is fetched or timed >> out. >> >> We had to handle this in our wrapper code. Basically, we check if metadata >> not available, we put msg into a queue and drain the queue from a diff >> thread. >> >> Thanks, >> Steven >> >> >> On Tue, Mar 29, 2016 at 4:59 AM, Oleg Zhurakousky < >> ozhurakou...@hortonworks.com> wrote: >> >>> I agree and considering that send(..) method returns Future one would >>> argue it must never block, otherwise what’s the point of returning Future >>> if you remove user’s ability to control how long are they willing to wait >>> and what to do when certain types of exception arise. Nevertheless it >> does >>> and it happens in the very first line of code: >>> // first make sure the metadata for the topic is available >>> waitOnMetadata(record.topic(), this.metadataFetchTimeoutMs); >>> So I am curious as well as to what is the motivation for it as we’ve seen >>> the same symptoms multiple times. >>> Cheers >>> Oleg >>> On Mar 29, 2016, at 4:31 AM, Paolo Patierno <ppatie...@live.com<mailto: >>> ppatie...@live.com>> wrote: >>> >>> Hello, >>> as documentation says, the KafkaProducer.send() method is asynchronous >> and >>> it just returns immediately.I found out that it's not so true when the >>> Kafka server it's trying to connect isn't online.Of course it happens >> only >>> on the first send() method invocation. It means that if the Kafka server >>> isn't reachable when my application starts for the first time, the send() >>> method isn't so asynchronous but it blocks.I know that it's trying to >>> connect to the Kafka server but why it doesn't save the message into the >>> buffer and returns immediately ?Is it a behavior or a bug ? >>> Thanks,Paolo >>> >>> Paolo PatiernoSenior Software Engineer >>> >>> >>> Windows Embedded & IoTMicrosoft Azure Advisor >>> Twitter : @ppatierno >>> Linkedin : paolopatierno >>> Blog : DevExperienceBlog : Embedded101 >>
