Hi Vitali,
When I ran into it it was latest time being very large. Until we could get
the messages set right we set segment.ms to maxint so it only rolled based
on size.
Cheers
William
On Thu, 23 Jul 2020 at 4:46 pm, Vitalii Stoianov <
vitalii.stoianov...@gmail.com> wrote:
> Hi William,
>
>
> ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --property
> print.timestamp=true --topic test
> One of the messages TS output:
> CreateTime:1595485571406707 1595485026.850 1595485571.406 216301538579718
> {msg data}
>
> So which one of these is used to roll over a log segment?
> I was trying to find some explanation on the web but with no luck.
>
> Regards,
> Vitalii.
>
> On Thu, Jul 23, 2020 at 9:25 AM William Reynolds <
> william.reyno...@instaclustr.com> wrote:
>
> > Hi Vitali,
> > What are the timestamps in your message? I have seen this before where
> you
> > have timestamps well into the future so every few messages causes a log
> > roll and you end up with a very large amount of log files.
> >
> > *William*
> >
> > On Thu, 23 Jul 2020 at 16:22, Vitalii Stoianov <
> > vitalii.stoianov...@gmail.com> wrote:
> >
> > > Hi All,
> > >
> > > I also have noticed that the number of log/index files are too high and
> > log
> > > roll is happening more frequently than expected.
> > > The log.roll.hours is default (168) and log.segment.bytes is 1g and log
> > > files size in the topic partition folders are usually smaller than 1g.
> > >
> > > Regards,
> > > Vitalii.
> > >
> > > On Wed, Jul 22, 2020 at 8:15 PM Vitalii Stoianov <
> > > vitalii.stoianov...@gmail.com> wrote:
> > >
> > > > Hi All,
> > > >
> > > > According to this:
> > > https://docs.confluent.io/current/kafka/deployment.html
> > > > vm.max_map_count is depend on number of index file:
> > > > *find /tmp/kafka_logs -name '*index' | wc -l*
> > > >
> > > > In our test lab we have next setup:
> > > >
> > > > *Topic:test PartitionCount:256 ReplicationFactor:2
> > > > Configs:segment.bytes=1073741824,retention.ms
> > > > <http://retention.ms
> > >
> >
> >=86400000,message.format.version=2.3-IV1,max.message.bytes=4194304,unclean.leader.election.enable=true*
> > > >
> > > > No cleanup.policy set explicitly for topic or in server.properties
> so I
> > > > assume default: delete according to
> > > > https://kafka.apache.org/23/documentation.html#brokerconfigs
> > > >
> > > > I did a small script that counted the number of index files and for
> > this
> > > > topic it is:
> > > > ~638000.
> > > > Also if I check kafka log/data dir it contain some old log/index
> files
> > > > create date for which is older than 10 days.(retention for topic is
> one
> > > day)
> > > > Note: When i checked log-cleaner.log it contains info only about
> > cleanup
> > > > for compacted logs.
> > > >
> > > > In order to set: vm.max_map_count value correctly, I need to
> > > > understand the following:
> > > > Why do such old index/log files exist and not cleaned?
> > > > How properly set vm.max_map_count if index/logs is not freed on time
> ??
> > > >
> > > > Regards,
> > > > Vitalii.
> > > >
> > >
> >
>
--
*William Reynolds**Technical Operations Engineer*
<https://www.facebook.com/instaclustr> <https://twitter.com/instaclustr>
<https://www.linkedin.com/company/instaclustr>
Read our latest technical blog posts here
<https://www.instaclustr.com/blog/>.
This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).
This email and any attachments may contain confidential and legally
privileged information. If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.
Instaclustr values your privacy. Our privacy policy can be found at
https://www.instaclustr.com/company/policies/privacy-policy
