Hey Guozhang, Thanks for the heads up!
Best On Thu, May 7, 2015 at 1:26 AM, Guozhang Wang <wangg...@gmail.com> wrote: > The metrics for checking that would better be "buffer-available-bytes" > instead of "bufferpool-wait-ratio", checking on its value approaching 0. > > Guozhang > > On Wed, May 6, 2015 at 3:02 AM, mete <efk...@gmail.com> wrote: > > > Hey Guozhang, > > > > I could go with both of the options, eventually i want to detect if there > > is a problem and isolate it from the rest of the system. And i am trying > to > > decide what would be the appropriate metrics to do that. > > > > Best > > > > > > > > On Wed, May 6, 2015 at 6:35 AM, Guozhang Wang <wangg...@gmail.com> > wrote: > > > > > 1. KAFKA-1955 <https://issues.apache.org/jira/browse/KAFKA-1955>, I > > > think Jay has a WIP patch for it. > > > 2. > > > 3. > > > > > > > > > On Tue, May 5, 2015 at 5:10 PM, Jason Rosenberg <j...@squareup.com> > > wrote: > > > > > > > Guozhang, > > > > > > > > Do you have the ticket number for possibly adding in local log file > > > > failover? Is it actively being worked on? > > > > > > > > Thanks, > > > > > > > > Jason > > > > > > > > On Tue, May 5, 2015 at 6:11 PM, Guozhang Wang <wangg...@gmail.com> > > > wrote: > > > > > > > > > Does this "log file" acts as a temporary disk buffer when broker > > slows > > > > > down, whose data will be re-send to broker later, or do you plan to > > use > > > > it > > > > > as a separate persistent storage as Kafka brokers? > > > > > > > > > > For the former use case, I think there is an open ticket for > > > integrating > > > > > this kind of functionality into producer; for the latter use case, > > you > > > > may > > > > > want to do this traffic control out of Kafka producer, i.e. upon > > > > detecting > > > > > producer buffer full, do not call send() on it for a while but > write > > > to a > > > > > different file, etc. > > > > > > > > > > Guozhang > > > > > > > > > > On Tue, May 5, 2015 at 11:28 AM, mete <efk...@gmail.com> wrote: > > > > > > > > > > > Sure, i kind of count on that actually, i guess with this setting > > the > > > > > > sender blocks on allocate method and this bufferpool-wait-ratio > > > > > increases. > > > > > > > > > > > > I want to fully compartmentalize the kafka producer from the rest > > of > > > > the > > > > > > system. Ex: writing to a log file instead of trying to send to > > kafka > > > > when > > > > > > some metric in the producer indicates that there is a performance > > > > > > degradation or some other problem. > > > > > > I was wondering what would be the ideal way of deciding that? > > > > > > > > > > > > > > > > > > > > > > > > On Tue, May 5, 2015 at 6:32 PM, Jay Kreps <jay.kr...@gmail.com> > > > wrote: > > > > > > > > > > > > > Does block.on.buffer.full=false do what you want? > > > > > > > > > > > > > > -Jay > > > > > > > > > > > > > > On Tue, May 5, 2015 at 1:59 AM, mete <efk...@gmail.com> wrote: > > > > > > > > > > > > > > > Hello Folks, > > > > > > > > > > > > > > > > I was looking through the kafka.producer metrics on the JMX > > > > > interface, > > > > > > to > > > > > > > > find a good indicator when to "trip" the circuit. So far it > > seems > > > > > like > > > > > > > the > > > > > > > > "bufferpool-wait-ratio" metric is a useful decision mechanism > > > when > > > > to > > > > > > cut > > > > > > > > off the production to kafka. > > > > > > > > > > > > > > > > As far as i experienced, when kafka server slow for some > > reason, > > > > > > requests > > > > > > > > start piling up on the producer queue and if you are not > > willing > > > to > > > > > > drop > > > > > > > > any messages on the producer, send method starts blocking > > because > > > > of > > > > > > the > > > > > > > > slow responsiveness. > > > > > > > > > > > > > > > > So this buffer pool wait ratio starts going up from 0.x up to > > > 1.0. > > > > > And > > > > > > i > > > > > > > am > > > > > > > > thinking about tripping the circuit breaker using this > metric, > > > ex: > > > > if > > > > > > > > wait-ratio > 0.90 etc... > > > > > > > > > > > > > > > > What do you think? Do you think there would be a better > > indicator > > > > to > > > > > > > check > > > > > > > > the health overall? > > > > > > > > > > > > > > > > Best > > > > > > > > Mete > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > -- Guozhang > > > > > > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > -- > -- Guozhang >
