What minor version of Kafka are you running? Might you be impacted by https://issues.apache.org/jira/browse/KAFKA-6030? -w
On Tue, 9 Jan 2018 at 19:02 Thunder Stumpges <thunder.stump...@gmail.com> wrote: > Hello, I posted this on StackOverflow > < > https://stackoverflow.com/questions/47948399/kafka-per-topic-retention-bytes-and-global-log-retention-bytes-not-working > >also > but haven't gotten any response. > > thanks in advance, > Thunder > __________ > > We are running a 6 node cluster of kafka 0.11.0. We have set a global as > well as a per-topic retention in bytes, neither of which is being applied. > There are no errors that I can see in the logs, just nothing being deleted > (by size; the time retention does seem to be working) > > See relevant configs below: > > *./config/server.properties* : > > # global retention 75GB or 60 days, segment size 512MB > log.retention.bytes=75000000000 > log.retention.check.interval.ms=60000 > > log.retention.hours=1440 > > log.cleanup.policy=delete > > log.segment.bytes=536870912 > > *topic configuration (30GB):* > > [tstumpges@kafka-02 kafka]$ bin/kafka-topics.sh --zookeeper > zk-01:2181/kafka --describe --topic stg_logtopic > Topic:stg_logtopic PartitionCount:12 ReplicationFactor:3 > Configs:retention.bytes=30000000000 > Topic: stg_logtopic Partition: 0 Leader: 4 > Replicas: 4,5,6 Isr: 4,5,6 > Topic: stg_logtopic Partition: 1 Leader: 5 > Replicas: 5,6,1 Isr: 5,1,6 > ... > > And, disk usage showing 910GB usage for one partition! > > [tstumpges@kafka-02 kafka]$ sudo du -s -h /data1/kafka-data/* > 82G /data1/kafka-data/stg_logother3-2 > 155G /data1/kafka-data/stg_logother2-9 > 169G /data1/kafka-data/stg_logother1-6 > 910G /data1/kafka-data/stg_logtopic-4 > > I can see there are plenty of segment log files (512MB each) in the > partition directory... what is going on?! > > Thanks in advance, Thunder >
