Kafka's retention policy is a little bit tricky, because it determines
if data is subject to deletion by comparing the embedded record
timestamps to the broker wall-clock time.
Can you verify your record timestamps as well as broker clocks?
If you write "future data", ie, data with timestamps larger than broker
wall-clock time, it might stay in the topic longer than retention time.
If you write "very old data", ie, data with timestamps smaller than
broker wall-clock minus retention time, data might get purged aggressively.
-Matthias
On 5/25/21 5:07 PM, Marina Popova wrote:
> Thank you, Ran,
> yes, I understand about the segment size impact - however, if you look at the
> configuration of those topics (see below) - you will see they have exactly
> the same config, with the same segment.bytes setting (to 1G, default), and
> all other settings are also the same - except for the retention:
> 1 topic has retention 4 hrs:
>
> segment.bytes=1073741824 sensitive=false
> synonyms={STATIC_BROKER_CONFIG:log.segment.bytes=1073741824,
> DEFAULT_CONFIG:log.segment.bytes=1073741824}
> retention.ms=14400000 sensitive=false
> synonyms={DYNAMIC_TOPIC_CONFIG:retention.ms=14400000}
>
> 2nd topic has retention 720 hrs (30 days), default:
> segment.bytes=1073741824 sensitive=false
> synonyms={STATIC_BROKER_CONFIG:log.segment.bytes=1073741824,
> DEFAULT_CONFIG:log.segment.bytes=1073741824}
> retention.ms=2592000000 sensitive=false synonyms={}
>
> yet, 1st topic is not being cleaned up, but the 2-nd topic is being cleaned
> up every 5 min, due to "
>
> Found deletable segments with base offsets [11755700] due to retention time
> 2592000000ms breach "
>
> And I can see that the size of the segments never reaches 1G for the 2-nd
> topic either ...
>
> thank you,
>
> Marina
>
> Sent with [ProtonMail](https://protonmail.com) Secure Email.
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Tuesday, May 25, 2021 4:11 PM, Ran Lupovich <ranlupov...@gmail.com> wrote:
>
>> Sorry I did not see all the info at first, what do you mean by topic getting
>> cleaned, you have setting to check retention every 5 minutes, the data that
>> getting "cleaned" is the older data which is 30 days old... am I missing
>> something?
>>
>> בתאריך יום ג׳, 25 במאי 2021, 23:04, מאת Ran Lupovich
>> <ranlupov...@gmail.com>:
>>
>>> By the segment size you are "delete" after 1 giga bytes is full , per
>>> partition, you need to remmber the retention is done when segments closed ,
>>> per partition
>>>
>>> בתאריך יום ג׳, 25 במאי 2021, 22:59, מאת Ran Lupovich
>>> <ranlupov...@gmail.com>:
>>>
>>>> Have you checked the segment size? Did you decribe the topic
>>>> configuration?maybe you created it with some settings you dont remember
>>>>
>>>> בתאריך יום ג׳, 25 במאי 2021, 19:51, מאת Marina Popova
>>>> <ppine7...@protonmail.com.invalid>:
>>>>
>>>>> Any idea what is wrong here? I have restarted Kafka brokers a few times,
>>>>> and all other Confluent services like KSQL - but I see exactly the same
>>>>> behavior - one topic gets its logs cleaned up every 5 minutes, while the
>>>>> other one - does not get cleaned up at all ....
>>>>>
>>>>> Is there anything else I could check, apart from what I already did - see
>>>>> the post below - to troubleshoot this?
>>>>>
>>>>> thank you!
>>>>> Marina
>>>>>
>>>>> Sent with ProtonMail Secure Email.
>>>>>
>>>>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>>>>> On Thursday, May 20, 2021 2:10 PM, Marina Popova
>>>>> <ppin...@protonmail.com.INVALID> wrote:
>>>>>
>>>>>> Hi, I have posted this question on SO:
>>>>>> https://stackoverflow.com/questions/67625641/kafka-segments-are-deleted-too-often-or-not-at-all
>>>>>> but wanted to re-post here as well in case someone spots the issue right
>>>>>> away ....
>>>>>>
>>>>>> Thank you for your help!
>>>>>>
>>>>>>>>>>>
>>>>>>
>>>>>> We have two topics on our Kafka cluster that exhibit weird (wrong)
>>>>>> behavior related to retention configuration.
>>>>>>
>>>>>> One topic, tracking.ap.client.traffic, has retention set explicitly to
>>>>>> "retention.ms=14400000" (4 hrs) - but it is not cleaned up , grows in
>>>>>> size, and caused 2 out of 3 kafka brokers to run out of disk space.
>>>>>> (details about config and log below)
>>>>>>
>>>>>> Second topic, tracking.ap.client.traffic.keyed, is created in KSQL as a
>>>>>> stream topic:
>>>>>>
>>>>>> CREATE STREAM AP_CLIENT_TRAFFIC_KEYED
>>>>>> WITH (KAFKA_TOPIC='tracking.ap.client.traffic.keyed',
>>>>>> TIMESTAMP='ACTIVITY_DATE', VALUE_FORMAT='JSON', PARTITIONS=6)
>>>>>> AS SELECT
>>>>>> ...
>>>>>>
>>>>>>
>>>>>>
>>>>>> its retention is set to the default broker value, which is 720 hrs :
>>>>>>
>>>>>> cat /etc/kafka/server.properties | grep retention
>>>>>> log.retention.hours=720
>>>>>> # A size-based retention policy for logs. Segments are pruned from the
>>>>>> log unless the remaining
>>>>>> # segments drop below log.retention.bytes. Functions independently of
>>>>>> log.retention.hours.
>>>>>> #log.retention.bytes=1073741824
>>>>>> # to the retention policies
>>>>>> log.retention.check.interval.ms=300000
>>>>>>
>>>>>>
>>>>>> This topic, though, gets cleaned up every 5 min or so - according to the
>>>>>> logs....
>>>>>> The log entry says the segment is marked for deletion "due to retention
>>>>>> time 2592000000ms breach (kafka.log.Log)" - but how can that be true if
>>>>>> this is happening every 5 min??
>>>>>>
>>>>>> No size-based retention is set for any of the topics.
>>>>>>
>>>>>> Two questions:
>>>>>>
>>>>>> 1. why is the first topic not being cleaned p?
>>>>>> 2. why is the second topic being cleaned up so often?
>>>>>>
>>>>>> Below are the details about logs and full config of both topics:
>>>>>>
>>>>>> log entries for tracking.ap.client.traffic.keyed-2 topic/partition -
>>>>>> show that this partition is getting cleaned too often:
>>>>>>
>>>>>>
>>>>>> [2021-05-19 21:35:05,822] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Incrementing log start offset to 11755700 (kafka.log.Log)
>>>>>> [2021-05-19 21:36:05,822] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Deleting segment 11753910 (kafka.log.Log)
>>>>>> [2021-05-19 21:36:05,825] INFO Deleted log
>>>>>> /apps/kafka-data/tracking.ap.client.traffic.keyed-2/00000000000011753910.log.deleted.
>>>>>> (kafka.log.LogSegment)
>>>>>> [2021-05-19 21:36:05,827] INFO Deleted offset index
>>>>>> /apps/kafka-data/tracking.ap.client.traffic.keyed-2/00000000000011753910.index.deleted.
>>>>>> (kafka.log.LogSegment)
>>>>>> [2021-05-19 21:36:05,829] INFO Deleted time index
>>>>>> /apps/kafka-data/tracking.ap.client.traffic.keyed-2/00000000000011753910.timeindex.deleted.
>>>>>> (kafka.log.LogSegment)
>>>>>> [2021-05-19 21:40:05,838] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Found deletable segments with base offsets [11755700] due to retention
>>>>>> time 2592000000ms breach (kafka.log.Log)
>>>>>> [2021-05-19 21:40:05,843] INFO [ProducerStateManager
>>>>>> partition=tracking.ap.client.traffic.keyed-2] Writing producer snapshot
>>>>>> at offset 11757417 (kafka.log.ProducerStateManager)
>>>>>> [2021-05-19 21:40:05,845] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Rolled new log segment at offset 11757417 in 7 ms. (kafka.log.Log)
>>>>>> [2021-05-19 21:40:05,845] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Scheduling log segment [baseOffset 11755700, size 936249] for deletion.
>>>>>> (kafka.log.Log)
>>>>>> [2021-05-19 21:40:05,847] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Incrementing log start offset to 11757417 (kafka.log.Log)
>>>>>> [2021-05-19 21:41:05,848] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Deleting segment 11755700 (kafka.log.Log)
>>>>>> [2021-05-19 21:41:05,850] INFO Deleted log
>>>>>> /apps/kafka-data/tracking.ap.client.traffic.keyed-2/00000000000011755700.log.deleted.
>>>>>> (kafka.log.LogSegment)
>>>>>> [2021-05-19 21:41:05,852] INFO Deleted offset index
>>>>>> /apps/kafka-data/tracking.ap.client.traffic.keyed-2/00000000000011755700.index.deleted.
>>>>>> (kafka.log.LogSegment)
>>>>>> [2021-05-19 21:41:05,853] INFO Deleted time index
>>>>>> /apps/kafka-data/tracking.ap.client.traffic.keyed-2/00000000000011755700.timeindex.deleted.
>>>>>> (kafka.log.LogSegment)
>>>>>> [2021-05-19 21:45:05,834] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Found deletable segments with base offsets [11757417] due to retention
>>>>>> time 2592000000ms breach (kafka.log.Log)
>>>>>> [2021-05-19 21:45:05,837] INFO [ProducerStateManager
>>>>>> partition=tracking.ap.client.traffic.keyed-2] Writing producer snapshot
>>>>>> at offset 11759201 (kafka.log.ProducerStateManager)
>>>>>> [2021-05-19 21:45:05,839] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Rolled new log segment at offset 11759201 in 6 ms. (kafka.log.Log)
>>>>>> [2021-05-19 21:45:05,839] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Scheduling log segment [baseOffset 11757417, size 977444] for deletion.
>>>>>> (kafka.log.Log)
>>>>>> [2021-05-19 21:45:05,842] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Incrementing log start offset to 11759201 (kafka.log.Log)
>>>>>> [2021-05-19 21:46:05,842] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Deleting segment 11757417 (kafka.log.Log)
>>>>>> [2021-05-19 21:46:05,844] INFO Deleted log
>>>>>> /apps/kafka-data/tracking.ap.client.traffic.keyed-2/00000000000011757417.log.deleted.
>>>>>> (kafka.log.LogSegment)
>>>>>> [2021-05-19 21:46:05,845] INFO Deleted offset index
>>>>>> /apps/kafka-data/tracking.ap.client.traffic.keyed-2/00000000000011757417.index.deleted.
>>>>>> (kafka.log.LogSegment)
>>>>>> [2021-05-19 21:46:05,847] INFO Deleted time index
>>>>>> /apps/kafka-data/tracking.ap.client.traffic.keyed-2/00000000000011757417.timeindex.deleted.
>>>>>> (kafka.log.LogSegment)
>>>>>> [2021-05-19 21:50:05,832] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Found deletable segments with base offsets [11759201] due to retention
>>>>>> time 2592000000ms breach (kafka.log.Log)
>>>>>> [2021-05-19 21:50:05,837] INFO [ProducerStateManager
>>>>>> partition=tracking.ap.client.traffic.keyed-2] Writing producer snapshot
>>>>>> at offset 11760980 (kafka.log.ProducerStateManager)
>>>>>> [2021-05-19 21:50:05,839] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Rolled new log segment at offset 11760980 in 7 ms. (kafka.log.Log)
>>>>>> [2021-05-19 21:50:05,839] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Scheduling log segment [baseOffset 11759201, size 991447] for deletion.
>>>>>> (kafka.log.Log)
>>>>>> [2021-05-19 21:50:05,844] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Incrementing log start offset to 11760980 (kafka.log.Log)
>>>>>> [2021-05-19 21:51:05,844] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Deleting segment 11759201 (kafka.log.Log)
>>>>>> [2021-05-19 21:51:05,846] INFO Deleted log
>>>>>> /apps/kafka-data/tracking.ap.client.traffic.keyed-2/00000000000011759201.log.deleted.
>>>>>> (kafka.log.LogSegment)
>>>>>> [2021-05-19 21:51:05,848] INFO Deleted offset index
>>>>>> /apps/kafka-data/tracking.ap.client.traffic.keyed-2/00000000000011759201.index.deleted.
>>>>>> (kafka.log.LogSegment)
>>>>>> [2021-05-19 21:51:05,849] INFO Deleted time index
>>>>>> /apps/kafka-data/tracking.ap.client.traffic.keyed-2/00000000000011759201.timeindex.deleted.
>>>>>> (kafka.log.LogSegment)
>>>>>> [2021-05-19 21:55:05,828] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Found deletable segments with base offsets [11760980] due to retention
>>>>>> time 2592000000ms breach (kafka.log.Log)
>>>>>> [2021-05-19 21:55:05,833] INFO [ProducerStateManager
>>>>>> partition=tracking.ap.client.traffic.keyed-2] Writing producer snapshot
>>>>>> at offset 11762941 (kafka.log.ProducerStateManager)
>>>>>> [2021-05-19 21:55:05,835] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Rolled new log segment at offset 11762941 in 7 ms. (kafka.log.Log)
>>>>>> [2021-05-19 21:55:05,835] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Scheduling log segment [baseOffset 11760980, size 1079204] for deletion.
>>>>>> (kafka.log.Log)
>>>>>> [2021-05-19 21:55:05,838] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Incrementing log start offset to 11762941 (kafka.log.Log)
>>>>>> [2021-05-19 21:56:05,838] INFO [Log
>>>>>> partition=tracking.ap.client.traffic.keyed-2, dir=/apps/kafka-data]
>>>>>> Deleting segment 11760980 (kafka.log.Log)
>>>>>> [2021-05-19 21:56:05,840] INFO Deleted log
>>>>>> /apps/kafka-data/tracking.ap.client.traffic.keyed-2/00000000000011760980.log.deleted.
>>>>>> (kafka.log.LogSegment)
>>>>>> [2021-05-19 21:56:05,842] INFO Deleted offset index
>>>>>> /apps/kafka-data/tracking.ap.client.traffic.keyed-2/00000000000011760980.index.deleted.
>>>>>> (kafka.log.LogSegment)
>>>>>> [2021-05-19 21:56:05,843] INFO Deleted time index
>>>>>> /apps/kafka-data/tracking.ap.client.traffic.keyed-2/00000000000011760980.timeindex.deleted.
>>>>>> (kafka.log.LogSegment)
>>>>>>
>>>>>>
>>>>>> There areno such entries for log cleaning for the first topic,
>>>>>> tracking.ap.client.traffic, which indicates it is never even checked for
>>>>>> hitting retention ??
>>>>>>
>>>>>> Full config for tracking.ap.client.traffic.keyed topic: (that is cleaned
>>>>>> up too often):
>>>>>>
>>>>>> mac-lt-mpopova:confluent-6.1.0 mpopova$ ./bin/kafka-configs
>>>>>> --bootstrap-server mybroker1:9092 --entity-type topics --entity-name
>>>>>> tracking.ap.client.traffic.keyed --describe --all
>>>>>>
>>>>>> All configs for topic tracking.ap.client.traffic.keyed are:
>>>>>> compression.type=producer sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:compression.type=producer}
>>>>>> leader.replication.throttled.replicas= sensitive=false synonyms={}
>>>>>> message.downconversion.enable=true sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.message.downconversion.enable=true}
>>>>>> min.insync.replicas=2 sensitive=false
>>>>>> synonyms={STATIC_BROKER_CONFIG:min.insync.replicas=2,
>>>>>> DEFAULT_CONFIG:min.insync.replicas=1}
>>>>>> segment.jitter.ms=0 sensitive=false synonyms={}
>>>>>> cleanup.policy=delete sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.cleanup.policy=delete}
>>>>>> flush.ms=9223372036854775807 sensitive=false synonyms={}
>>>>>> follower.replication.throttled.replicas= sensitive=false synonyms={}
>>>>>> segment.bytes=1073741824 sensitive=false
>>>>>> synonyms={STATIC_BROKER_CONFIG:log.segment.bytes=1073741824,
>>>>>> DEFAULT_CONFIG:log.segment.bytes=1073741824}
>>>>>> retention.ms=2592000000 sensitive=false synonyms={}
>>>>>> flush.messages=9223372036854775807 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.flush.interval.messages=9223372036854775807}
>>>>>> message.format.version=2.1-IV2 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.message.format.version=2.1-IV2}
>>>>>> file.delete.delay.ms=60000 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.segment.delete.delay.ms=60000}
>>>>>> max.message.bytes=1000012 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:message.max.bytes=1000012}
>>>>>> min.compaction.lag.ms=0 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.cleaner.min.compaction.lag.ms=0}
>>>>>> message.timestamp.type=CreateTime sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.message.timestamp.type=CreateTime}
>>>>>> preallocate=false sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.preallocate=false}
>>>>>> min.cleanable.dirty.ratio=0.5 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.cleaner.min.cleanable.ratio=0.5}
>>>>>> index.interval.bytes=4096 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.index.interval.bytes=4096}
>>>>>> unclean.leader.election.enable=false sensitive=false
>>>>>> synonyms={STATIC_BROKER_CONFIG:unclean.leader.election.enable=false,
>>>>>> DEFAULT_CONFIG:unclean.leader.election.enable=false}
>>>>>> retention.bytes=-1 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.retention.bytes=-1}
>>>>>> delete.retention.ms=86400000 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.cleaner.delete.retention.ms=86400000}
>>>>>> segment.ms=604800000 sensitive=false synonyms={}
>>>>>> message.timestamp.difference.max.ms=9223372036854775807 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.message.timestamp.difference.max.ms=9223372036854775807}
>>>>>> segment.index.bytes=10485760 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.index.size.max.bytes=10485760}
>>>>>>
>>>>>>
>>>>>> Full config fortracking.ap.client.traffic topic: (not being cleaned up
>>>>>> at all):
>>>>>>
>>>>>> mac-lt-mpopova:confluent-6.1.0 mpopova$ ./bin/kafka-configs
>>>>>> --bootstrap-server mybroker1:9092 --entity-type topics --entity-name
>>>>>> tracking.ap.client.traffic --describe --all
>>>>>>
>>>>>> All configs for topic tracking.ap.client.traffic are:
>>>>>> compression.type=producer sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:compression.type=producer}
>>>>>> leader.replication.throttled.replicas= sensitive=false synonyms={}
>>>>>> message.downconversion.enable=true sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.message.downconversion.enable=true}
>>>>>> min.insync.replicas=2 sensitive=false
>>>>>> synonyms={STATIC_BROKER_CONFIG:min.insync.replicas=2,
>>>>>> DEFAULT_CONFIG:min.insync.replicas=1}
>>>>>> segment.jitter.ms=0 sensitive=false synonyms={}
>>>>>> cleanup.policy=delete sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.cleanup.policy=delete}
>>>>>> flush.ms=9223372036854775807 sensitive=false synonyms={}
>>>>>> follower.replication.throttled.replicas= sensitive=false synonyms={}
>>>>>> segment.bytes=1073741824 sensitive=false
>>>>>> synonyms={STATIC_BROKER_CONFIG:log.segment.bytes=1073741824,
>>>>>> DEFAULT_CONFIG:log.segment.bytes=1073741824}
>>>>>> retention.ms=14400000 sensitive=false
>>>>>> synonyms={DYNAMIC_TOPIC_CONFIG:retention.ms=14400000}
>>>>>> flush.messages=9223372036854775807 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.flush.interval.messages=9223372036854775807}
>>>>>> message.format.version=2.1-IV2 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.message.format.version=2.1-IV2}
>>>>>> file.delete.delay.ms=60000 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.segment.delete.delay.ms=60000}
>>>>>> max.message.bytes=1000012 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:message.max.bytes=1000012}
>>>>>> min.compaction.lag.ms=0 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.cleaner.min.compaction.lag.ms=0}
>>>>>> message.timestamp.type=CreateTime sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.message.timestamp.type=CreateTime}
>>>>>> preallocate=false sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.preallocate=false}
>>>>>> min.cleanable.dirty.ratio=0.5 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.cleaner.min.cleanable.ratio=0.5}
>>>>>> index.interval.bytes=4096 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.index.interval.bytes=4096}
>>>>>> unclean.leader.election.enable=false sensitive=false
>>>>>> synonyms={STATIC_BROKER_CONFIG:unclean.leader.election.enable=false,
>>>>>> DEFAULT_CONFIG:unclean.leader.election.enable=false}
>>>>>> retention.bytes=-1 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.retention.bytes=-1}
>>>>>> delete.retention.ms=86400000 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.cleaner.delete.retention.ms=86400000}
>>>>>> segment.ms=604800000 sensitive=false synonyms={}
>>>>>> message.timestamp.difference.max.ms=9223372036854775807 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.message.timestamp.difference.max.ms=9223372036854775807}
>>>>>> segment.index.bytes=10485760 sensitive=false
>>>>>> synonyms={DEFAULT_CONFIG:log.index.size.max.bytes=10485760}
>>>>>>
>>>>>>
>>>>>> Here is an example directory content of the logs for both topics:
>>>>>>
>>>>>> ls -la /apps/kafka-data/tracking.ap.client.traffic-*
>>>>>>
>>>>>> /apps/kafka-data/tracking.ap.client.traffic-1:
>>>>>> total 412640
>>>>>> drwxr-xr-x 2 cp-kafka confluent 141 May 17 15:01 .
>>>>>> drwxr-xr-x 290 cp-kafka confluent 20480 May 19 22:48 ..
>>>>>> -rw-r--r-- 1 cp-kafka confluent 10485760 May 19 22:48
>>>>>> 00000000000000000000.index
>>>>>> -rw-r--r-- 1 cp-kafka confluent 420599482 May 19 22:48
>>>>>> 00000000000000000000.log
>>>>>> -rw-r--r-- 1 cp-kafka confluent 10485756 May 19 22:48
>>>>>> 00000000000000000000.timeindex
>>>>>> -rw-r--r-- 1 cp-kafka confluent 8 May 17 15:01 leader-epoch-checkpoint
>>>>>>
>>>>>> /apps/kafka-data/tracking.ap.client.traffic-11:
>>>>>> total 412844
>>>>>> drwxr-xr-x 2 cp-kafka confluent 141 May 17 11:25 .
>>>>>> drwxr-xr-x 290 cp-kafka confluent 20480 May 19 22:48 ..
>>>>>> -rw-r--r-- 1 cp-kafka confluent 10485760 May 19 22:48
>>>>>> 00000000000000000000.index
>>>>>> -rw-r--r-- 1 cp-kafka confluent 420788365 May 19 22:48
>>>>>> 00000000000000000000.log
>>>>>> -rw-r--r-- 1 cp-kafka confluent 10485756 May 19 22:48
>>>>>> 00000000000000000000.timeindex
>>>>>> -rw-r--r-- 1 cp-kafka confluent 8 May 17 11:25 leader-epoch-checkpoint
>>>>>>
>>>>>> /apps/kafka-data/tracking.ap.client.traffic-13:
>>>>>> total 412660
>>>>>> drwxr-xr-x 2 cp-kafka confluent 141 May 17 15:01 .
>>>>>> drwxr-xr-x 290 cp-kafka confluent 20480 May 19 22:48 ..
>>>>>> -rw-r--r-- 1 cp-kafka confluent 10485760 May 19 22:48
>>>>>> 00000000000000000000.index
>>>>>> -rw-r--r-- 1 cp-kafka confluent 420606307 May 19 22:48
>>>>>> 00000000000000000000.log
>>>>>> -rw-r--r-- 1 cp-kafka confluent 10485756 May 19 22:48
>>>>>> 00000000000000000000.timeindex
>>>>>> -rw-r--r-- 1 cp-kafka confluent 8 May 17 15:01 leader-epoch-checkpoint
>>>>>>
>>>>>> /apps/kafka-data/tracking.ap.client.traffic-14:
>>>>>> total 412920
>>>>>> drwxr-xr-x 2 cp-kafka confluent 141 May 17 11:25 .
>>>>>> drwxr-xr-x 290 cp-kafka confluent 20480 May 19 22:48 ..
>>>>>> -rw-r--r-- 1 cp-kafka confluent 10485760 May 19 22:48
>>>>>> 00000000000000000000.index
>>>>>> -rw-r--r-- 1 cp-kafka confluent 420862000 May 19 22:48
>>>>>> 00000000000000000000.log
>>>>>> -rw-r--r-- 1 cp-kafka confluent 10485756 May 19 22:48
>>>>>> 00000000000000000000.timeindex
>>>>>> -rw-r--r-- 1 cp-kafka confluent 8 May 17 11:25 leader-epoch-checkpoint
>>>>>>
>>>>>>
>>>>>> ls -la /apps/kafka-data/tracking.ap.client.traffic.keyed*
>>>>>>
>>>>>> /apps/kafka-data/tracking.ap.client.traffic.keyed-1:
>>>>>> total 288
>>>>>> drwxr-xr-x 2 cp-kafka confluent 141 May 19 22:47 .
>>>>>> drwxr-xr-x 290 cp-kafka confluent 20480 May 19 22:48 ..
>>>>>> -rw-r--r-- 1 cp-kafka confluent 10485760 May 19 22:48
>>>>>> 00000000000004255431.index
>>>>>> -rw-r--r-- 1 cp-kafka confluent 254355 May 19 22:48
>>>>>> 00000000000004255431.log
>>>>>> -rw-r--r-- 1 cp-kafka confluent 10485756 May 19 22:48
>>>>>> 00000000000004255431.timeindex
>>>>>> -rw-r--r-- 1 cp-kafka confluent 14 May 19 22:47 leader-epoch-checkpoint
>>>>>>
>>>>>> /apps/kafka-data/tracking.ap.client.traffic.keyed-2:
>>>>>> total 816
>>>>>> drwxr-xr-x 2 cp-kafka confluent 178 May 19 22:46 .
>>>>>> drwxr-xr-x 290 cp-kafka confluent 20480 May 19 22:48 ..
>>>>>> -rw-r--r-- 1 cp-kafka confluent 10485760 May 19 22:48
>>>>>> 00000000000011782064.index
>>>>>> -rw-r--r-- 1 cp-kafka confluent 787175 May 19 22:48
>>>>>> 00000000000011782064.log
>>>>>> -rw-r--r-- 1 cp-kafka confluent 10 May 19 22:45
>>>>>> 00000000000011782064.snapshot
>>>>>> -rw-r--r-- 1 cp-kafka confluent 10485756 May 19 22:48
>>>>>> 00000000000011782064.timeindex
>>>>>> -rw-r--r-- 1 cp-kafka confluent 15 May 19 22:45 leader-epoch-checkpoint
>>>>>>
>>>>>> /apps/kafka-data/tracking.ap.client.traffic.keyed-3:
>>>>>> total 1812
>>>>>> drwxr-xr-x 2 cp-kafka confluent 141 May 19 22:47 .
>>>>>> drwxr-xr-x 290 cp-kafka confluent 20480 May 19 22:48 ..
>>>>>> -rw-r--r-- 1 cp-kafka confluent 10485760 May 19 22:48
>>>>>> 00000000000028393394.index
>>>>>> -rw-r--r-- 1 cp-kafka confluent 1814528 May 19 22:48
>>>>>> 00000000000028393394.log
>>>>>> -rw-r--r-- 1 cp-kafka confluent 10485756 May 19 22:48
>>>>>> 00000000000028393394.timeindex
>>>>>> -rw-r--r-- 1 cp-kafka confluent 15 May 19 22:47 leader-epoch-checkpoint
>>>>>>
>>>>>>
>>>>>> Thank you!
>>>>>>
>>>>>> Sent with ProtonMail Secure Email.
