Hi All, I also have noticed that the number of log/index files are too high and log roll is happening more frequently than expected. The log.roll.hours is default (168) and log.segment.bytes is 1g and log files size in the topic partition folders are usually smaller than 1g.
Regards, Vitalii. On Wed, Jul 22, 2020 at 8:15 PM Vitalii Stoianov < vitalii.stoianov...@gmail.com> wrote: > Hi All, > > According to this: https://docs.confluent.io/current/kafka/deployment.html > vm.max_map_count is depend on number of index file: > *find /tmp/kafka_logs -name '*index' | wc -l* > > In our test lab we have next setup: > > *Topic:test PartitionCount:256 ReplicationFactor:2 > Configs:segment.bytes=1073741824,retention.ms > <http://retention.ms>=86400000,message.format.version=2.3-IV1,max.message.bytes=4194304,unclean.leader.election.enable=true* > > No cleanup.policy set explicitly for topic or in server.properties so I > assume default: delete according to > https://kafka.apache.org/23/documentation.html#brokerconfigs > > I did a small script that counted the number of index files and for this > topic it is: > ~638000. > Also if I check kafka log/data dir it contain some old log/index files > create date for which is older than 10 days.(retention for topic is one day) > Note: When i checked log-cleaner.log it contains info only about cleanup > for compacted logs. > > In order to set: vm.max_map_count value correctly, I need to > understand the following: > Why do such old index/log files exist and not cleaned? > How properly set vm.max_map_count if index/logs is not freed on time ?? > > Regards, > Vitalii. >
