Hello Neil, Did you use any compression on your producers?
Guozhang On Tue, Aug 5, 2014 at 12:23 PM, Neil Harkins <nhark...@gmail.com> wrote: > Hi. > I'm seeing some odd numbers from kafka.SocketServerStats. > > Ideally, I'd like to have stats broken down per-topic, e.g. > what's our most written/read topics? For write rates, I've got > a separate process iterating topics every minute, doing this: > (head_offset_now - head_offset_last) / (time_now - time_last) > and inserting that into graphite (was previously inserting the > head_offset, but graphite's derivative() on 2000+ topics > was a bit much). > > a graphite sumSeries() across them shows 20~25 MB/s > on a single-instance kafka 0.7.2, which sounds correct, > but when I compare that to the kafka.SocketServerStats > we're collecting in graphite, derivative(TotalBytesWritten) > shows 3.0GB/s, and BytesWrittenPerSecond shows > 200KB/s, neither of which jive with my offset rate. > > Is anyone aware of any weirdness with those stats? > > thanks in advance for any insight, > -neil > -- -- Guozhang
