We can't afford to block since we don't want the pressure to percolate upstream (even if we did, scribe would drop the messages rather than the producer).
aim is to not block while queueing as long as there are enough brokers that can handle the load. with infinite timeout (or even with smaller time out like 1 second), when a thread invokes producer.send(ProducerData) it does not know whether it is going to block or not. I am trying to write a partitioner that can avoid asyncProducers with higher backlog their queue. Of course, if the brokers in aggregate can't handle that load, then there is no choice but to drop the messages. But currently we end up dropping messages even with one single slow broker. Raghu. On Fri, Jan 4, 2013 at 10:49 AM, Neha Narkhede <neha.narkh...@gmail.com>wrote: > queue.enqueueTimeout.ms = -1 will block the producer instead of dropping > messages. This might be useful if you have the producer being fed by scribe > aggregators. > > Thanks, > Neha > > > On Fri, Jan 4, 2013 at 9:04 AM, Raghu Angadi <raghu.ang...@gmail.com> > wrote: > > > On Fri, Jan 4, 2013 at 8:39 AM, Jun Rao <jun...@gmail.com> wrote: > > > > > Do you know why some of the brokers are much slower than others? > > > > > > We are currently running these in a shared environment, to make things > > worse these machines have single spindle. We have to put up with that > until > > we move the brokers to dedicated hardware with multiple spindles. The > > problem is a bit exaggerated in current setup. > > > > Even with dedicated hardware, I am expecting some variation. One slightly > > degraded disk out of 12 could reduce effective b/w on all the spindles. > > Unfortunately there will be occasional rack level network slowdowns that > > take many hours to get fixed. > > > > In our case, we cannot let the back pressure from slow brokers propagate > > upstream. Producers receive messages from scribe aggregators and just > have > > to drop the messages it they can't write fast enough. > > >
