Can Y have a callback that will handle the notification to X? In this case, perhaps Y can be async and X can buffer the data until the callback triggers and says "all good" (or resend if the callback indicates an error)
On Mon, Feb 2, 2015 at 12:56 PM, Otis Gospodnetic <otis.gospodne...@gmail.com> wrote: > Hi, > > Thanks for the info. Here's the use case. We have something up stream > sending data, say a log shipper called X. It sends it to some remote > component Y. Y is the Kafka Producer and it puts data into Kafka. But Y > needs to send a reply to X and tell it whether it successfully put all its > data into Kafka. If it did not, Y wants to tell X to buffer data locally > and resend it later. > > If producer is ONLY async, Y can't easily do that. Or maybe Y would just > need to wait for the Future to come back and only then send the response > back to X? If so, I'm guessing the delay would be more or less the same as > if the Producer was using SYNC mode? > > Thanks, > Otis > -- > Monitoring * Alerting * Anomaly Detection * Centralized Log Management > Solr & Elasticsearch Support * http://sematext.com/ > > > On Mon, Feb 2, 2015 at 3:13 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > >> Yeah as Gwen says there is no sync/async mode anymore. There is a new >> configuration which does a lot of what async did in terms of allowing >> batching: >> >> batch.size - This is the target amount of data per partition the server >> will attempt to batch together. >> linger.ms - This is the time the producer will wait for more data to be >> sent to better batch up writes. The default is 0 (send immediately). So if >> you set this to 50 ms the client will send immediately if it has already >> filled up its batch, otherwise it will wait to accumulate the number of >> bytes given by batch.size. >> >> To send asynchronously you do >> producer.send(record) >> whereas to block on a response you do >> producer.send(record).get(); >> which will wait for acknowledgement from the server. >> >> One advantage of this model is that the client will do it's best to batch >> under the covers even if linger.ms=0. It will do this by batching any data >> that arrives while another send is in progress into a single >> request--giving a kind of "group commit" effect. >> >> The hope is that this will be both simpler to understand (a single api that >> always works the same) and more powerful (you always get a response with >> error and offset information whether or not you choose to use it). >> >> -Jay >> >> >> On Mon, Feb 2, 2015 at 11:15 AM, Gwen Shapira <gshap...@cloudera.com> >> wrote: >> >> > If you want to emulate the old sync producer behavior, you need to set >> > the batch size to 1 (in producer config) and wait on the future you >> > get from Send (i.e. future.get) >> > >> > I can't think of good reasons to do so, though. >> > >> > Gwen >> > >> > >> > On Mon, Feb 2, 2015 at 11:08 AM, Otis Gospodnetic >> > <otis.gospodne...@gmail.com> wrote: >> > > Hi, >> > > >> > > Is the plan for New Producer to have ONLY async mode? I'm asking >> because >> > > of this info from the Wiki: >> > > >> > > >> > > - The producer will always attempt to batch data and will always >> > > immediately return a SendResponse which acts as a Future to allow >> the >> > > client to await the completion of the request. >> > > >> > > >> > > The word "always" makes me think there will be no sync mode. >> > > >> > > Thanks, >> > > Otis >> > > -- >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management >> > > Solr & Elasticsearch Support * http://sematext.com/ >> > >>
