Hey Chris, I tried setting producer.batch.size to 256KB (the Kafka docs say the default is 16KB) and the throughput is much better. That job is running a bit faster than 0.8 now.
Gian On Tue, Apr 7, 2015 at 2:02 PM, Chris Riccomini <criccom...@apache.org> wrote: > Hey Gian, > > Hmm, this is strange. We ran some tests, and found that the new producer to > be faster than the old producer default (sync), and almost as fast as the > old producer's async producer. > > Could you paste all of your configs? > > Cheers, > Chris > > On Tue, Apr 7, 2015 at 10:40 AM, Gian Merlino <g...@metamarkets.com> > wrote: > > > Has anyone else seen issues with producer performance in 0.9.0? > > > > I updated a few of our jobs recently and ended up rolling one back to 0.8 > > since it was being really sluggish. I profiled it for a bit and a lot of > > time was being spent in BufferPool.allocate and the busy-loop in > > KafkaSystemProducer's flush. The flush-ms metric said that flushes were > > taking on the order of tens of seconds. > > > > The slow topic was a state changelog for a stream-stream join buffer. I > > tried setting producer.linger.ms to 1, 5, and 1000, but that didn't > change > > the behavior much. > > > > Gian > > >
