Hi All,
I have a kafka cluster with 3 brokers.
My kafka version is 2.5.0
I have a topic named "order" with 3 partitions and replication factor of 3.
cleanup policy of the topic is "compact,delete"
topic is evenly distributed across all brokers.
broker 0 => partition 0,1,2
broker 1 => partition 0,1,2
broker 2 => partition 0,1,2
What i have noticed is log cleaner on broker 1 is not cleaning the log of
partition 0.
it is cleaning all other partitions of the topic on that broker and all
partitions of that topic on other brokers.
i have checked the kafka log carefully and did not find any log cleaning
message for partition 0. neither any error logged.
This is causing the log of the partition to grow continuously and
eventually filling the disk space.
This happen randomly for a partition on one of the broker. after the broker
restart log cleaner start cleaning the partition again but some other
partition get affected.
Any hint what causing the log cleaner to skip cleaning the log of a
partition ?
topic is created with following configuration.
[2021-09-22 05:06:18,544] INFO Created log for partition order-0 in
/home/kafka/data/order-0 with properties {compression.type -> producer,
min.insync.replicas -> 2, message.downconversion.enable -> true,
segment.jitter.ms -> 0, cleanup.policy -> [delete], flush.ms ->
9223372036854775807, retention.ms -> 86400000, segment.bytes -> 1073741824,
flush.messages -> 9223372036854775807, message.format.version -> 2.5-IV0,
max.compaction.lag.ms -> 9223372036854775807, file.delete.delay.ms ->
60000, max.message.bytes -> 1000000, min.compaction.lag.ms -> 0,
message.timestamp.type -> LogAppendTime, preallocate -> false,
index.interval.bytes -> 4096, min.cleanable.dirty.ratio -> 0.5,
unclean.leader.election.enable -> false, retention.bytes -> -1,
delete.retention.ms -> 86400000, segment.ms -> 3600000,
message.timestamp.difference.max.ms -> 9223372036854775807,
segment.index.bytes -> 10485760}. (kafka.log.LogManager)
[2021-09-22 05:06:19,466] INFO Processing notification(s) to
/config/changes (kafka.common.ZkNodeChangeNotificationListener)
[2021-09-22 05:06:19,469] INFO Processing override for entityPath:
topics/order with config: HashMap(cleanup.policy -> compact,delete,
retention.ms -> 72000000) (kafka.server.DynamicConfigManager)
Thanks,
Dhirendra.
