Hi Nicolas,
> Any Idea some may have to limit log compaction disk usage ? That's a
pretty
big issue we're having, compaction of a few topics is using far more IO
than everything else on the broker...
I don't know any other better solution except make the
`log.cleaner.io.max.bytes.per.second` lower than you expected, maybe 512?
As I explained earlier, based on how throttler works currently, there will
be a 300ms time slot of full speed read/write.
And it will make the final IO speed higher than you expected.
However, Apache Kafka is an open source project, so, welcome to make a
contribution to make it better!
Let me know if you're interested.
Thank you.
Luke
On Wed, Dec 28, 2022 at 6:25 PM Nicolas Carlot
<nicolas.car...@chronopost.fr.invalid> wrote:
> Okay, thx for the explaination Luc.
> When looking for logs on a bigger topic, I can see the IO limitation kinda
> works.
> I have like15MB/s when having configured
> log.cleaner.io.max.bytes.per.second=1024.0
> I'm not very comfortable with this. Setting the limit rate to 1Kb/s, ending
> with 15Mb/s, seems a bit akward.
> Any Idea some may have to limit log compaction disk usage ? That's a pretty
> big issue we're having, compaction of a few topics is using far more IO
> than everything else on the broker...
>
> Log cleaner thread 0 cleaned log
>
> kafka-stream-parcel360.lt_event360.multiprofiles-lteventmulti2-KSTREAM.RETAINER.kafka-stream-parcel360.lt_event360.multiprofiles-lteventmulti2-changelog-8
> (dirty section = [34819497, 44148502])
> 1 372,1 MB of log processed in 182,2 seconds (7,5 MB/sec).
> Indexed 1 143,2 MB in 84,9 seconds (13,5 Mb/sec, 46,6% of total
> time)
> Buffer utilization: 9,2%
> Cleaned 1 372,1 MB in 97,3 seconds (14,1 Mb/sec, 53,4% of total
> time)
> Start size: 1 372,1 MB (10 329 108 messages)
> End size: 230,0 MB (1 000 105 messages)
> 83,2% size reduction (90,3% fewer messages)
>
> Le mer. 28 déc. 2022 à 09:41, Luke Chen <show...@gmail.com> a écrit :
>
> > Hi Nicolas,
> >
> > The throttler in log cleaner is just a simple throttler to control the IO
> > speed by periodically (every 300ms) check the IO rate so far, and sleep
> > some time to slow it down.
> > That is, in your case, it could be:
> > log cleaner runs for 300ms with full IO speed -> check -> throttle (sleep
> > to reach desired rate) -> run 300ms with full speed -> done
> >
> > So, when printing the log, you will see the rate is higher than you
> > expected.
> > At least I ran in kafka v3.3.1 and it works well.
> >
> > Thank you.
> > Luke
> >
> > On Thu, Dec 22, 2022 at 11:31 PM Nicolas Carlot
> > <nicolas.car...@chronopost.fr.invalid> wrote:
> >
> > > Hello users,
> > >
> > > I'm trying to use the log.cleaner.io.max.bytes.per.second configuration
> > to
> > > limit disk usage when the log cleaner runs, but it doesn't seem to work
> > as
> > > expected.
> > > When looking at the logs, I can see it still uses far more than
> > configured.
> > >
> > > This is the output with log.cleaner.io.max.bytes.per.second=1024
> > >
> > > [2022-12-22 16:14:49,462] INFO [kafka-log-cleaner-thread-0]:
> > > Log cleaner thread 0 cleaned log __consumer_offsets-9 (dirty
> > > section = [13509895330, 13510879425])
> > > 100,2 MB of log processed in 1,7 seconds (60,1 MB/sec).
> > > Indexed 99,7 MB in 0,8 seconds (117,5 Mb/sec, 50,8% of total
> > time)
> > > Buffer utilization: 0,1%
> > > Cleaned 100,2 MB in 0,8 seconds (122,2 Mb/sec, 49,2% of total
> > time)
> > > Start size: 100,2 MB (988642 messages)
> > > End size: 0,5 MB (4547 messages)
> > > 99,5% size reduction (99,5% fewer messages)
> > >
> > > I did update the value trough kafak-config this way:
> > >
> > > kafka-configs.sh --bootstrap-server localhost:59092 --entity-type
> brokers
> > > --command-config ~/server.admin.properties --entity-default --alter
> > > --add-config log.cleaner.io.max.bytes.per.second=1024
> > >
> > > Which shows up correctly in broker config:
> > > log.cleaner.io.max.bytes.per.second=1024.0 sensitive=false
> > > synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:log.cleaner.io
> > > .max.bytes.per.second=1024,
> > > DEFAULT_CONFIG:log.cleaner.io
> > .max.bytes.per.second=1.7976931348623157E308}
> > >
> > > Any idea ? Something I'm missing ?
> > >
> > > --
> > > [image: Chronopost] <https://www.chronopost.fr/fr?xtatc=INT-149>
> > >
> > >
> > > *Nicolas Carlot*
> > > *Lead dev*Direction des Systèmes d'Information
> > >
> > >
> > > 3 boulevard Romain Rolland
> > > 75014 Paris
> > > [image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ>
> > > [image:
> > > img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image:
> > > img]
> > > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
> > > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
> > >
> > > [image: img]
> > >
> > > [image: img]
> > >
> >
>
>
> --
> [image: Chronopost] <https://www.chronopost.fr/fr?xtatc=INT-149>
>
>
> *Nicolas Carlot*
> *Lead dev*Direction des Systèmes d'Information
>
>
> 3 boulevard Romain Rolland
> 75014 Paris
> [image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ>
> [image:
> img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image:
> img]
> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
>
> [image: img]
>
> [image: img]
>
