Okay, thx for the explaination Luc.
When looking for logs on a bigger topic, I can see the IO limitation kinda
works.
I have like15MB/s when having configured
log.cleaner.io.max.bytes.per.second=1024.0
I'm not very comfortable with this. Setting the limit rate to 1Kb/s, ending
with 15Mb/s, seems a bit akward.
Any Idea some may have to limit log compaction disk usage ? That's a pretty
big issue we're having, compaction of a few topics is using far more IO
than everything else on the broker...
Log cleaner thread 0 cleaned log
kafka-stream-parcel360.lt_event360.multiprofiles-lteventmulti2-KSTREAM.RETAINER.kafka-stream-parcel360.lt_event360.multiprofiles-lteventmulti2-changelog-8
(dirty section = [34819497, 44148502])
1 372,1 MB of log processed in 182,2 seconds (7,5 MB/sec).
Indexed 1 143,2 MB in 84,9 seconds (13,5 Mb/sec, 46,6% of total
time)
Buffer utilization: 9,2%
Cleaned 1 372,1 MB in 97,3 seconds (14,1 Mb/sec, 53,4% of total
time)
Start size: 1 372,1 MB (10 329 108 messages)
End size: 230,0 MB (1 000 105 messages)
83,2% size reduction (90,3% fewer messages)
Le mer. 28 déc. 2022 à 09:41, Luke Chen <show...@gmail.com> a écrit :
> Hi Nicolas,
>
> The throttler in log cleaner is just a simple throttler to control the IO
> speed by periodically (every 300ms) check the IO rate so far, and sleep
> some time to slow it down.
> That is, in your case, it could be:
> log cleaner runs for 300ms with full IO speed -> check -> throttle (sleep
> to reach desired rate) -> run 300ms with full speed -> done
>
> So, when printing the log, you will see the rate is higher than you
> expected.
> At least I ran in kafka v3.3.1 and it works well.
>
> Thank you.
> Luke
>
> On Thu, Dec 22, 2022 at 11:31 PM Nicolas Carlot
> <nicolas.car...@chronopost.fr.invalid> wrote:
>
> > Hello users,
> >
> > I'm trying to use the log.cleaner.io.max.bytes.per.second configuration
> to
> > limit disk usage when the log cleaner runs, but it doesn't seem to work
> as
> > expected.
> > When looking at the logs, I can see it still uses far more than
> configured.
> >
> > This is the output with log.cleaner.io.max.bytes.per.second=1024
> >
> > [2022-12-22 16:14:49,462] INFO [kafka-log-cleaner-thread-0]:
> > Log cleaner thread 0 cleaned log __consumer_offsets-9 (dirty
> > section = [13509895330, 13510879425])
> > 100,2 MB of log processed in 1,7 seconds (60,1 MB/sec).
> > Indexed 99,7 MB in 0,8 seconds (117,5 Mb/sec, 50,8% of total
> time)
> > Buffer utilization: 0,1%
> > Cleaned 100,2 MB in 0,8 seconds (122,2 Mb/sec, 49,2% of total
> time)
> > Start size: 100,2 MB (988642 messages)
> > End size: 0,5 MB (4547 messages)
> > 99,5% size reduction (99,5% fewer messages)
> >
> > I did update the value trough kafak-config this way:
> >
> > kafka-configs.sh --bootstrap-server localhost:59092 --entity-type brokers
> > --command-config ~/server.admin.properties --entity-default --alter
> > --add-config log.cleaner.io.max.bytes.per.second=1024
> >
> > Which shows up correctly in broker config:
> > log.cleaner.io.max.bytes.per.second=1024.0 sensitive=false
> > synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:log.cleaner.io
> > .max.bytes.per.second=1024,
> > DEFAULT_CONFIG:log.cleaner.io
> .max.bytes.per.second=1.7976931348623157E308}
> >
> > Any idea ? Something I'm missing ?
> >
> > --
> > [image: Chronopost] <https://www.chronopost.fr/fr?xtatc=INT-149>
> >
> >
> > *Nicolas Carlot*
> > *Lead dev*Direction des Systèmes d'Information
> >
> >
> > 3 boulevard Romain Rolland
> > 75014 Paris
> > [image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ>
> > [image:
> > img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image:
> > img]
> > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
> > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
> >
> > [image: img]
> >
> > [image: img]
> >
>
--
[image: Chronopost] <https://www.chronopost.fr/fr?xtatc=INT-149>
*Nicolas Carlot*
*Lead dev*Direction des Systèmes d'Information
3 boulevard Romain Rolland
75014 Paris
[image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ> [image:
img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image: img]
<https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
<https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
[image: img]
[image: img]
