If you have enough memory for batch size then the scenario is no different
than your original question because block on full memory is true by default
(and if it is false you actually get a notification via exception). So when
you said reduce the buffer memory to "minimum", I assumed you are referring
to reducing the number of messages buffered in memory so that the potential
lose of messages is less (in case the producer totally dies)
On Tue, Aug 18, 2015 at 9:07 AM sunil kalva <kalva.ka...@gmail.com> wrote:
> kishore
> How is going to reduce throughput if we have enough memory for batch size
> ?, could you please explain
>
> On Tue, Aug 18, 2015 at 11:47 AM, Kishore Senji <kse...@gmail.com> wrote:
>
> > But this will reduce the throughput in a good scenario. May be we need to
> > enhance the Callback interface appropriately.
> >
> > On Mon, Aug 17, 2015 at 7:15 PM, sunil kalva <kalva.ka...@gmail.com>
> > wrote:
> >
> > > tx jeff,
> > > Actually we need to set "buffer.memory" to minimum (default is ~35 MB)
> > and
> > > "block.on.buffer.full" to "true" so that the sender will block as soon
> > as
> > > these conditions met. And then release once the cluster is healthy.
> > >
> > > --
> > > SunilKalva
> > >
> > > On Mon, Aug 17, 2015 at 11:20 PM, Jeff Holoman <jholo...@cloudera.com>
> > > wrote:
> > >
> > > > Actually this won't work. The problem is if the producer loses
> > > connectivity
> > > > to the broker, then messages will continue to queue up until
> batch.size
> > > is
> > > > exhausted. Then the send will block. At this point, if you gain
> > > > connectivity again, then the messages will be resent.
> > > >
> > > > If all brokers die, you should get Not Enough Replicas if you have
> the
> > > > min.isr.set first. This should be an indication that you are having a
> > > > problem, before you get to the error in the actual prodcuer thread.
> > Keep
> > > in
> > > > mind that as of right now you need to manually set the min.isr's with
> > > alter
> > > > topic due to KAFKA-2114
> > > > Either way, you will eventually get an error in the producer thread
> > that
> > > > isn't passed to the callback.
> > > >
> > > > And I think that's the point.
> > > >
> > > > Thanks
> > > >
> > > > Jeff
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Mon, Aug 17, 2015 at 12:39 PM, Madhukar Bharti <
> > > > bhartimadhu...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Sunil,
> > > > >
> > > > > Producer will throw an Exception in callback if there is problem
> > while
> > > > > sending data. You can check like:
> > > > >
> > > > > public void onCompletion(RecordMetadata arg0, Exception arg1) {
> > > > > if (arg1 != null) {
> > > > > System.out.println("exception occured");
> > > > > }
> > > > > System.out.println("sent")
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Aug 17, 2015 at 9:20 PM, sunil kalva <
> kalva.ka...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi all
> > > > > > I am using new java producer in async mode, when my entire
> cluster
> > is
> > > > > down
> > > > > > i am loosing all my messages. How do we get notification when the
> > > > cluster
> > > > > > is down so that i can send messages to another cluster. The
> > callback
> > > is
> > > > > > only triggered when the cluster is reachable .
> > > > > >
> > > > > > --SK
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks and Regards,
> > > > > Madhukar Bharti
> > > > > Mob: 7845755539
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Jeff Holoman
> > > > Systems Engineer
> > > >
> > >
> >
>
