Hi, Can you look at the disk IO while you are approaching end of the topic and your performance raises? End of the topic may reside in FS cache in memory and that may explain.
BR On Thu, Apr 16, 2020 at 6:56 PM Ilya R <mrgreen...@gmail.com> wrote: > Hello, everyone > > I'm researching problem when reading a topic from the beginning max speed > per partition/per consumer not more than 80MB/s. But near the end of the > topic speed rise up to 250-300MB/s unexpectedly, not only on the last log > segment on several latest log segments. I've already change network buffers > in a linux, change socket settings on server/client side, but there are not > results. For testing I'm using kafka-console-consimer.sh script from > standart installation. Kafka version is 2.3.0. > > Does anyone else encountered this? > > Server settings > broker.id=1 > log.message.timestamp.type=LogAppendTime > num.network.threads=24 > num.io.threads=10 > offset.metadata.max.bytes=8096 > offsets.load.buffer.size=10485760 > offsets.topic.num.partitions=1 > offsets.topic.replication.factor=1 > socket.request.max.bytes=4194304 > socket.send.buffer.bytes=4194304 > leader.imbalance.check.interval.seconds = 30 > log.dir=/local/kafka/data > log.dirs=/local/kafka/data > zookeeper.connect=localhost:2181 > message.max.bytes = 50000000 > log.segment.bytes = 104857600 > > Topic settings: > compression.type uncompressed > segment.bytes 104857600 > max.message.bytes 50000000 > min.compaction.lag.ms 43200000 > message.timestamp.type LogAppendTime > min.cleanable.dirty.ratio 0.1 > > Consumer settings: > max.partition.fetch.bytes=8048576 > fetch.max.bytes=128777216 > fetch.min.bytes=8048576 > receive.buffer.bytes=8048577 > -- Seva Feldman VP R&D Mobile Delivery [image: ironSource] <http://www.ironsrc.com/> email sev...@ironsrc.com mobile +972544346089 ironSource HQ - 121 Derech Menachem Begin st. Tel Aviv
