Re: kafka tuning(vm.max_map_count) and logs retention.

Hi All, I was checking it more and found this (we use librdkafka to put data into kafka topics): https://docs.confluent.io/5.0.0/clients/librdkafka/classRdKafka_1_1Producer.html#ab90a30c5e5fb006a3b4004dc4c9a7923 As docs say they use microseconds: virtual ErrorCode produce

Re: kafka tuning(vm.max_map_count) and logs retention.

Hi Alexandre, According to kafka broker logs it happens even faster each 5-30 sec. Regards, Vitalii. On Thu, Jul 23, 2020 at 11:15 AM Alexandre Dupriez < alexandre.dupr...@gmail.com> wrote: > Hi Vitalii, > > The timestamps provided by your producers are in microseconds, whereas > Kafka expects

Re: kafka tuning(vm.max_map_count) and logs retention.

Hi Vitalii, The timestamps provided by your producers are in microseconds, whereas Kafka expects milliseconds epochs. This could be the reason for over-rolling. When you had the default roll time value of a week, did you experience segment rolls every 15 minutes or so? Thanks, Alexandre Le jeu.

Re: kafka tuning(vm.max_map_count) and logs retention.

Hi Vitali, When I ran into it it was latest time being very large. Until we could get the messages set right we set segment.ms to maxint so it only rolled based on size. Cheers William On Thu, 23 Jul 2020 at 4:46 pm, Vitalii Stoianov < vitalii.stoianov...@gmail.com> wrote: > Hi William, > > > ./

Re: kafka tuning(vm.max_map_count) and logs retention.

Hi William, ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --property print.timestamp=true --topic test One of the messages TS output: CreateTime:1595485571406707 1595485026.850 1595485571.406 216301538579718 {msg data} So which one of these is used to roll over a log segment? I

Re: kafka tuning(vm.max_map_count) and logs retention.

Hi Vitali, What are the timestamps in your message? I have seen this before where you have timestamps well into the future so every few messages causes a log roll and you end up with a very large amount of log files. *William* On Thu, 23 Jul 2020 at 16:22, Vitalii Stoianov < vitalii.stoianov...@g

Re: kafka tuning(vm.max_map_count) and logs retention.

Hi All, I also have noticed that the number of log/index files are too high and log roll is happening more frequently than expected. The log.roll.hours is default (168) and log.segment.bytes is 1g and log files size in the topic partition folders are usually smaller than 1g. Regards, Vitalii. On