You are reaching 10gb * 1000 / 64 = 156 MB / s which probably saturated your hard drive bandwidth ? So you can take a look at your iostats
-- Sent from my iPhone On Aug 22, 2018, at 8:20 PM, Nan Xu <nanxu1...@gmail.com> wrote: I setup a local single node test. producer and broker are sitting at the same VM. broker only has a single node(localhost) and a single partition. producer produce message as fast as it could in a single thread. all update to a SINGLE key(String). the kafka broker data directory is memory based directory(RAM). in this setup, because I am not access network( everything is localhost), and not access disk( memory based dir). I should avoid all the IO operations, so the pub should be really, really fast, maybe close to the memory speed. But I publish 1,000,000 messges (every message is 10K string, so total 10G), that takes 64s, which I think it's very slow. And from htop, I don't see any cpu thread get pushed to close to 100%. is that mean kafka has some internal lock/wait which can not fully utilize the hardware? producer and broker setting are out of box default, I am using 0.10.1.0. Thanks, Nan
