Hemanth, Specifically, you'd want to monitor kafka:type=kafka.SocketServerStats:getMaxProduceRequestMs and kafka:type=kafka.LogFlushStats:getMaxFlushMs. If the broker is under load due to frequent flushes, it will almost certainly show up as spikes in the flush latency and consequently the produce request latency. A side effect of that is that your producer queue will back up and your producer will eventually lose data.
Thanks, Neha On Thu, Sep 11, 2014 at 5:48 PM, Hemanth Yamijala <yhema...@gmail.com> wrote: > Neha, > > Thanks. We are on 0.7.2. I have written on another thread on the list here > about one of the reasons we are stuck - the absence of a PHP client for our > front end producer systems. (On a side note, would appreciate if any inputs > can be given on that thread as well) > > When you mean performance, do you mean throughput ? We did measure > throughput with our default configuration of 1000 ms for the flush interval > value, and the much lower 100 ms value I proposed on this thread. Our > numbers were identical - for a single broker we were clocking at around > 20,000 messages read per second on the consumer side. Using a small 'n' > brokers we can easily exceed our target numbers. (The load was > synthetically generated - using a likely message size and at a rate that > seems reasonable for our producing side). > > Given this observation, do you suggest any further tests / measurements for > us to be sure ? Would appreciate any inputs. > > Thanks > Hemanth > > On Fri, Sep 12, 2014 at 1:32 AM, Neha Narkhede <neha.narkh...@gmail.com> > wrote: > > > I should mention that the impact of doing so is much higher wrt to > taking a > > hit on performance, on versions < 0.8.1. As long as you're on 0.8.1 or > > later, it should mostly be fine. You might want to keep a close tab on > how > > your iostat numbers are doing, to be sure. > > > > On Wed, Sep 10, 2014 at 5:46 PM, Hemanth Yamijala <yhema...@gmail.com> > > wrote: > > > > > Thanks Jun. > > > > > > On Thu, Sep 11, 2014 at 4:13 AM, Jun Rao <jun...@gmail.com> wrote: > > > > > > > As long as the I/O load is reasonable, this is probably ok. > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > On Wed, Sep 10, 2014 at 4:59 AM, Hemanth Yamijala < > yhema...@gmail.com> > > > > wrote: > > > > > > > > > Hi folks, > > > > > > > > > > In order to meet latency requirements for a system we are building, > > we > > > > > tested with different values of the above two parameters and found > > that > > > > > settings as low as 100 work best for us, balancing the required > > > > throughput > > > > > and latencies. > > > > > > > > > > I just wanted to check if 100 is a sane value, notwithstanding we > are > > > > > getting good results in our tests, anything we need to be aware of > > > while > > > > > setting to low values like this (apart from the throughput, which > we > > > see > > > > is > > > > > OK for us) ? > > > > > > > > > > Any experience reports will help. > > > > > > > > > > Thanks > > > > > Hemanth > > > > > > > > > > > > > > >
