There is RAID10 on 10 SSD Intel 1.9TB with raid cache 2GB. Topic size is 290GB it nearly fully in cache. Broker CPU usage near 2-3%. CPU is 2 x Intel(R) Xeon(R) Gold 6244 CPU @ 3.60GHz.
I didn't find any bottlenecks via JMX graphs. On 2020/04/16 15:59:23, Seva Feldman <sev...@ironsrc.com> wrote: > Hi, > > Can you look at the disk IO while you are approaching end of the topic and > your performance raises? End of the topic may reside in FS cache in memory > and that may explain. > > BR > > On Thu, Apr 16, 2020 at 6:56 PM Ilya R <mrgreen...@gmail.com> wrote: > > > Hello, everyone > > > > I'm researching problem when reading a topic from the beginning max speed > > per partition/per consumer not more than 80MB/s. But near the end of the > > topic speed rise up to 250-300MB/s unexpectedly, not only on the last log > > segment on several latest log segments. I've already change network buffers > > in a linux, change socket settings on server/client side, but there are not > > results. For testing I'm using kafka-console-consimer.sh script from > > standart installation. Kafka version is 2.3.0. > > > > Does anyone else encountered this? > > > > Server settings > > broker.id=1 > > log.message.timestamp.type=LogAppendTime > > num.network.threads=24 > > num.io.threads=10 > > offset.metadata.max.bytes=8096 > > offsets.load.buffer.size=10485760 > > offsets.topic.num.partitions=1 > > offsets.topic.replication.factor=1 > > socket.request.max.bytes=4194304 > > socket.send.buffer.bytes=4194304 > > leader.imbalance.check.interval.seconds = 30 > > log.dir=/local/kafka/data > > log.dirs=/local/kafka/data > > zookeeper.connect=localhost:2181 > > message.max.bytes = 50000000 > > log.segment.bytes = 104857600 > > > > Topic settings: > > compression.type uncompressed > > segment.bytes 104857600 > > max.message.bytes 50000000 > > min.compaction.lag.ms 43200000 > > message.timestamp.type LogAppendTime > > min.cleanable.dirty.ratio 0.1 > > > > Consumer settings: > > max.partition.fetch.bytes=8048576 > > fetch.max.bytes=128777216 > > fetch.min.bytes=8048576 > > receive.buffer.bytes=8048577 > > > > > -- > Seva Feldman > VP R&D Mobile Delivery > [image: ironSource] <http://www.ironsrc.com/> > > email sev...@ironsrc.com > mobile +972544346089 > > ironSource HQ - 121 Derech Menachem Begin st. Tel Aviv >
