On 10/25/2017 12:03 PM, Manikumar wrote:
any errors in log cleaner logs?
Not as far as I can see
root@kafka-1:/opt/kafka/logs# cat log-cleaner.log* | grep -i error
(empty)
However, I've seen that it actually did cleaning of the whole topic
(excerpts below), but it didn't seem to find anything worth cleaning
(size stayed the same).
Are there any global settings that could affect this?
I'm running a default config from Kafka, the only things changed are
message size, topic creation/deletion and the defaults for retention.
Which are all overwritten for this topic (see original post)
I'm writing a simple script to read and confirm the key distributions to
debug further, but as the same data (without duplicates) is also written
to a DB I'm pretty sure that the size is too big for not having gotten
compacted.
root@kafka-1:/opt/kafka/logs# cat log-cleaner.log* | grep
events.lg.aggregated
[2017-10-24 11:08:43,462] INFO Cleaner 0: Beginning cleaning of log
events.lg.aggregated-0. (kafka.log.LogCleaner)
[2017-10-24 11:08:43,462] INFO Cleaner 0: Building offset map for
events.lg.aggregated-0... (kafka.log.LogCleaner)
[2017-10-24 11:08:43,688] INFO Cleaner 0: Building offset map for log
events.lg.aggregated-0 for 1 segments in offset range [0, 81308).
(kafka.log.LogCleaner)
[2017-10-24 11:08:44,163] INFO Cleaner 0: Offset map for log
events.lg.aggregated-0 complete. (kafka.log.LogCleaner)
[2017-10-24 11:08:44,165] INFO Cleaner 0: Cleaning log
events.lg.aggregated-0 (cleaning prior to Tue Oct 24 11:08:30 UTC 2017,
discarding tombstones prior to Thu Jan 01 00:00:00 UTC 1970)...
(kafka.log.LogCleaner)
[2017-10-24 11:08:44,166] INFO Cleaner 0: Cleaning segment 0 in log
events.lg.aggregated-0 (largest timestamp Tue Oct 24 11:08:30 UTC 2017)
into 0, retaining deletes. (kafka.log.LogCleaner)
[2017-10-24 11:08:47,865] INFO Cleaner 0: Swapping in cleaned segment 0
for segment(s) 0 in log events.lg.aggregated-0. (kafka.log.LogCleaner)
Log cleaner thread 0 cleaned log events.lg.aggregated-0 (dirty
section = [0, 0])
[2017-10-24 11:10:47,875] INFO Cleaner 0: Beginning cleaning of log
events.lg.aggregated-0. (kafka.log.LogCleaner)
[2017-10-24 11:10:47,875] INFO Cleaner 0: Building offset map for
events.lg.aggregated-0... (kafka.log.LogCleaner)
[2017-10-24 11:10:47,910] INFO Cleaner 0: Building offset map for log
events.lg.aggregated-0 for 1 segments in offset range [81308, 154902).
(kafka.log.LogCleaner)
[2017-10-24 11:10:48,410] INFO Cleaner 0: Offset map for log
events.lg.aggregated-0 complete. (kafka.log.LogCleaner)
[2017-10-24 11:10:48,411] INFO Cleaner 0: Cleaning log
events.lg.aggregated-0 (cleaning prior to Tue Oct 24 11:10:32 UTC 2017,
discarding tombstones prior to Mon Oct 23 11:08:30 UTC 2017)...
(kafka.log.LogCleaner)
[2017-10-24 11:10:48,411] INFO Cleaner 0: Cleaning segment 0 in log
events.lg.aggregated-0 (largest timestamp Tue Oct 24 11:08:30 UTC 2017)
into 0, retaining deletes. (kafka.log.LogCleaner)
[2017-10-24 11:10:50,308] INFO Cleaner 0: Swapping in cleaned segment 0
for segment(s) 0 in log events.lg.aggregated-0. (kafka.log.LogCleaner)
[2017-10-24 11:10:50,309] INFO Cleaner 0: Cleaning segment 81308 in log
events.lg.aggregated-0 (largest timestamp Tue Oct 24 11:10:32 UTC 2017)
into 81308, retaining deletes. (kafka.log.LogCleaner)
[2017-10-24 11:10:53,389] INFO Cleaner 0: Swapping in cleaned segment
81308 for segment(s) 81308 in log events.lg.aggregated-0.
(kafka.log.LogCleaner)
Log cleaner thread 0 cleaned log events.lg.aggregated-0 (dirty
section = [81308, 81308])
