Hi Nicolas,
The throttler in log cleaner is just a simple throttler to control the IO
speed by periodically (every 300ms) check the IO rate so far, and sleep
some time to slow it down.
That is, in your case, it could be:
log cleaner runs for 300ms with full IO speed -> check -> throttle (sleep
to reach desired rate) -> run 300ms with full speed -> done
So, when printing the log, you will see the rate is higher than you
expected.
At least I ran in kafka v3.3.1 and it works well.
Thank you.
Luke
On Thu, Dec 22, 2022 at 11:31 PM Nicolas Carlot
<nicolas.car...@chronopost.fr.invalid> wrote:
> Hello users,
>
> I'm trying to use the log.cleaner.io.max.bytes.per.second configuration to
> limit disk usage when the log cleaner runs, but it doesn't seem to work as
> expected.
> When looking at the logs, I can see it still uses far more than configured.
>
> This is the output with log.cleaner.io.max.bytes.per.second=1024
>
> [2022-12-22 16:14:49,462] INFO [kafka-log-cleaner-thread-0]:
> Log cleaner thread 0 cleaned log __consumer_offsets-9 (dirty
> section = [13509895330, 13510879425])
> 100,2 MB of log processed in 1,7 seconds (60,1 MB/sec).
> Indexed 99,7 MB in 0,8 seconds (117,5 Mb/sec, 50,8% of total time)
> Buffer utilization: 0,1%
> Cleaned 100,2 MB in 0,8 seconds (122,2 Mb/sec, 49,2% of total time)
> Start size: 100,2 MB (988642 messages)
> End size: 0,5 MB (4547 messages)
> 99,5% size reduction (99,5% fewer messages)
>
> I did update the value trough kafak-config this way:
>
> kafka-configs.sh --bootstrap-server localhost:59092 --entity-type brokers
> --command-config ~/server.admin.properties --entity-default --alter
> --add-config log.cleaner.io.max.bytes.per.second=1024
>
> Which shows up correctly in broker config:
> log.cleaner.io.max.bytes.per.second=1024.0 sensitive=false
> synonyms={DYNAMIC_DEFAULT_BROKER_CONFIG:log.cleaner.io
> .max.bytes.per.second=1024,
> DEFAULT_CONFIG:log.cleaner.io.max.bytes.per.second=1.7976931348623157E308}
>
> Any idea ? Something I'm missing ?
>
> --
> [image: Chronopost] <https://www.chronopost.fr/fr?xtatc=INT-149>
>
>
> *Nicolas Carlot*
> *Lead dev*Direction des Systèmes d'Information
>
>
> 3 boulevard Romain Rolland
> 75014 Paris
> [image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ>
> [image:
> img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image:
> img]
> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
>
> [image: img]
>
> [image: img]
>
