Hello, everyone I'm researching problem when reading a topic from the beginning max speed per partition/per consumer not more than 80MB/s. But near the end of the topic speed rise up to 250-300MB/s unexpectedly, not only on the last log segment on several latest log segments. I've already change network buffers in a linux, change socket settings on server/client side, but there are not results. For testing I'm using kafka-console-consimer.sh script from standart installation. Kafka version is 2.3.0.
Does anyone else encountered this? Server settings broker.id=1 log.message.timestamp.type=LogAppendTime num.network.threads=24 num.io.threads=10 offset.metadata.max.bytes=8096 offsets.load.buffer.size=10485760 offsets.topic.num.partitions=1 offsets.topic.replication.factor=1 socket.request.max.bytes=4194304 socket.send.buffer.bytes=4194304 leader.imbalance.check.interval.seconds = 30 log.dir=/local/kafka/data log.dirs=/local/kafka/data zookeeper.connect=localhost:2181 message.max.bytes = 50000000 log.segment.bytes = 104857600 Topic settings: compression.type uncompressed segment.bytes 104857600 max.message.bytes 50000000 min.compaction.lag.ms 43200000 message.timestamp.type LogAppendTime min.cleanable.dirty.ratio 0.1 Consumer settings: max.partition.fetch.bytes=8048576 fetch.max.bytes=128777216 fetch.min.bytes=8048576 receive.buffer.bytes=8048577
