Basically, requirement is to support message dropping policy in event when queue is full. When you get storm of data (in our case logging due to buggy application code), we would like to retain current message instead of first one in queue. We will mitigate this by rate limiting on producer side. Only thing is if Kafka allows the flexibility to inject implementation, then developer have control over what to drop and retain and also what to priorities.
We would like to change tunable parameters (such as batch size and queue size etc or non intrusive parameter that does not impact the life cycle of the producer instance ) at runtime after producer instance is created. Thanks, Bhavesh On Mon, Aug 4, 2014 at 7:05 PM, Joe Stein <joe.st...@stealth.ly> wrote: > Is it possible there is another solution to the problem? I think if you > could better describe the problem(s) you are facing and how you are > architected some then you may get responses from others that perhaps have > faced the same problem with similar architectures ... or maybe folks can > chime in with solution(s) to the problem(s). When only being presented > with solutions it is hard to say much about if it is problem that folks > will have and if this solution will work for them. > > /******************************************* > Joe Stein > Founder, Principal Consultant > Big Data Open Source Security LLC > http://www.stealth.ly > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > ********************************************/ > > > On Mon, Aug 4, 2014 at 8:52 PM, Bhavesh Mistry <mistry.p.bhav...@gmail.com > > > wrote: > > > Kafka Version: 0.8.x > > > > 1) Ability to define which messages get drooped (least recently instead > of > > most recent in queue) > > 2) Try Unbounded Queue to find out the Upper Limit without drooping any > > messages for application (use case Stress test) > > 3) Priority Blocking Queue ( meaning a single Producer can send messages > to > > multiple topic and I would like to give Delivery Priority to message for > > particular message for topic) > > > > We have use case to support #3 and #1 since we would like to deliver the > > Application Heartbeat first then any other event within the queue for any > > topics. To lower TCP connections, we only use one producer for 4 topics > but > > one of the topics has priority for delivery. > > > > Please let me know if this is useful feature to have or not. > > > > Thanks in advance for great support !! > > > > Thanks, > > > > Bhavesh > > > > P.S. Sorry for asking this question again, but last time there was no > > conclusion. > > >
