the data directory is memory basd, no hard drive involved. mount -t tmpfs -o size=25G tmpfs /mnt/ramdisk and use this as data folder. iostat show 0 write too.
On Wed, Aug 22, 2018 at 11:06 PM Ken Chen <zlchen....@gmail.com> wrote: > You are reaching 10gb * 1000 / 64 = 156 MB / s which probably saturated > your hard drive bandwidth ? So you can take a look at your iostats > > -- > Sent from my iPhone > > On Aug 22, 2018, at 8:20 PM, Nan Xu <nanxu1...@gmail.com> wrote: > > I setup a local single node test. producer and broker are sitting at the > same VM. broker only has a single node(localhost) and a single partition. > producer produce message as fast as it could in a single thread. all update > to a SINGLE key(String). the kafka broker data directory is memory based > directory(RAM). in this setup, because I am not access network( everything > is localhost), and not access disk( memory based dir). I should avoid all > the IO operations, so the pub should be really, really fast, maybe close to > the memory speed. But I publish 1,000,000 messges (every message is 10K > string, so total 10G), that takes 64s, which I think it's very slow. And > from htop, I don't see any cpu thread get pushed to close to 100%. is that > mean kafka has some internal lock/wait which can not fully utilize the > hardware? producer and broker setting are out of box default, I am using > 0.10.1.0. > > Thanks, > Nan >
