Hi Alexandre,
According to kafka broker logs it happens even faster each 5-30 sec.
Regards,
Vitalii.
On Thu, Jul 23, 2020 at 11:15 AM Alexandre Dupriez <
alexandre.dupr...@gmail.com> wrote:
> Hi Vitalii,
>
> The timestamps provided by your producers are in microseconds, whereas
> Kafka expects milliseconds epochs. This could be the reason for
> over-rolling. When you had the default roll time value of a week, did
> you experience segment rolls every 15 minutes or so?
>
> Thanks,
> Alexandre
>
> Le jeu. 23 juil. 2020 à 08:31, William Reynolds
> <william.reyno...@instaclustr.com> a écrit :
> >
> > Hi Vitali,
> > When I ran into it it was latest time being very large. Until we could
> get
> > the messages set right we set segment.ms to maxint so it only rolled
> based
> > on size.
> > Cheers
> > William
> >
> > On Thu, 23 Jul 2020 at 4:46 pm, Vitalii Stoianov <
> > vitalii.stoianov...@gmail.com> wrote:
> >
> > > Hi William,
> > >
> > >
> > > ./kafka-console-consumer.sh --bootstrap-server localhost:9092
> --property
> > > print.timestamp=true --topic test
> > > One of the messages TS output:
> > > CreateTime:1595485571406707 1595485026.850 1595485571.406
> 216301538579718
> > > {msg data}
> > >
> > > So which one of these is used to roll over a log segment?
> > > I was trying to find some explanation on the web but with no luck.
> > >
> > > Regards,
> > > Vitalii.
> > >
> > > On Thu, Jul 23, 2020 at 9:25 AM William Reynolds <
> > > william.reyno...@instaclustr.com> wrote:
> > >
> > > > Hi Vitali,
> > > > What are the timestamps in your message? I have seen this before
> where
> > > you
> > > > have timestamps well into the future so every few messages causes a
> log
> > > > roll and you end up with a very large amount of log files.
> > > >
> > > > *William*
> > > >
> > > > On Thu, 23 Jul 2020 at 16:22, Vitalii Stoianov <
> > > > vitalii.stoianov...@gmail.com> wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I also have noticed that the number of log/index files are too
> high and
> > > > log
> > > > > roll is happening more frequently than expected.
> > > > > The log.roll.hours is default (168) and log.segment.bytes is 1g
> and log
> > > > > files size in the topic partition folders are usually smaller than
> 1g.
> > > > >
> > > > > Regards,
> > > > > Vitalii.
> > > > >
> > > > > On Wed, Jul 22, 2020 at 8:15 PM Vitalii Stoianov <
> > > > > vitalii.stoianov...@gmail.com> wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > According to this:
> > > > > https://docs.confluent.io/current/kafka/deployment.html
> > > > > > vm.max_map_count is depend on number of index file:
> > > > > > *find /tmp/kafka_logs -name '*index' | wc -l*
> > > > > >
> > > > > > In our test lab we have next setup:
> > > > > >
> > > > > > *Topic:test PartitionCount:256 ReplicationFactor:2
> > > > > > Configs:segment.bytes=1073741824,retention.ms
> > > > > > <http://retention.ms
> > > > >
> > > >
> > >
> >=86400000,message.format.version=2.3-IV1,max.message.bytes=4194304,unclean.leader.election.enable=true*
> > > > > >
> > > > > > No cleanup.policy set explicitly for topic or in
> server.properties
> > > so I
> > > > > > assume default: delete according to
> > > > > > https://kafka.apache.org/23/documentation.html#brokerconfigs
> > > > > >
> > > > > > I did a small script that counted the number of index files and
> for
> > > > this
> > > > > > topic it is:
> > > > > > ~638000.
> > > > > > Also if I check kafka log/data dir it contain some old log/index
> > > files
> > > > > > create date for which is older than 10 days.(retention for topic
> is
> > > one
> > > > > day)
> > > > > > Note: When i checked log-cleaner.log it contains info only about
> > > > cleanup
> > > > > > for compacted logs.
> > > > > >
> > > > > > In order to set: vm.max_map_count value correctly, I need to
> > > > > > understand the following:
> > > > > > Why do such old index/log files exist and not cleaned?
> > > > > > How properly set vm.max_map_count if index/logs is not freed on
> time
> > > ??
> > > > > >
> > > > > > Regards,
> > > > > > Vitalii.
> > > > > >
> > > > >
> > > >
> > >
> > --
> >
> >
> > *William Reynolds**Technical Operations Engineer*
> >
> >
> > <https://www.facebook.com/instaclustr> <
> https://twitter.com/instaclustr>
> > <https://www.linkedin.com/company/instaclustr>
> >
> > Read our latest technical blog posts here
> > <https://www.instaclustr.com/blog/>.
> >
> > This email has been sent on behalf of Instaclustr Pty. Limited
> (Australia)
> > and Instaclustr Inc (USA).
> >
> > This email and any attachments may contain confidential and legally
> > privileged information. If you are not the intended recipient, do not
> copy
> > or disclose its content, but please reply to this email immediately and
> > highlight the error to the sender and then immediately delete the
> message.
> >
> > Instaclustr values your privacy. Our privacy policy can be found at
> > https://www.instaclustr.com/company/policies/privacy-policy
>
