Does this "log file" acts as a temporary disk buffer when broker slows down, whose data will be re-send to broker later, or do you plan to use it as a separate persistent storage as Kafka brokers?
For the former use case, I think there is an open ticket for integrating this kind of functionality into producer; for the latter use case, you may want to do this traffic control out of Kafka producer, i.e. upon detecting producer buffer full, do not call send() on it for a while but write to a different file, etc. Guozhang On Tue, May 5, 2015 at 11:28 AM, mete <efk...@gmail.com> wrote: > Sure, i kind of count on that actually, i guess with this setting the > sender blocks on allocate method and this bufferpool-wait-ratio increases. > > I want to fully compartmentalize the kafka producer from the rest of the > system. Ex: writing to a log file instead of trying to send to kafka when > some metric in the producer indicates that there is a performance > degradation or some other problem. > I was wondering what would be the ideal way of deciding that? > > > > On Tue, May 5, 2015 at 6:32 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > Does block.on.buffer.full=false do what you want? > > > > -Jay > > > > On Tue, May 5, 2015 at 1:59 AM, mete <efk...@gmail.com> wrote: > > > > > Hello Folks, > > > > > > I was looking through the kafka.producer metrics on the JMX interface, > to > > > find a good indicator when to "trip" the circuit. So far it seems like > > the > > > "bufferpool-wait-ratio" metric is a useful decision mechanism when to > cut > > > off the production to kafka. > > > > > > As far as i experienced, when kafka server slow for some reason, > requests > > > start piling up on the producer queue and if you are not willing to > drop > > > any messages on the producer, send method starts blocking because of > the > > > slow responsiveness. > > > > > > So this buffer pool wait ratio starts going up from 0.x up to 1.0. And > i > > am > > > thinking about tripping the circuit breaker using this metric, ex: if > > > wait-ratio > 0.90 etc... > > > > > > What do you think? Do you think there would be a better indicator to > > check > > > the health overall? > > > > > > Best > > > Mete > > > > > > -- -- Guozhang
