We are on Kafka 10.2.1 and facing similar issue. Upgrading to 1.0 is disruptive. Any other way this can be circumvented ?
Thanks. On Fri, Jan 12, 2018 at 1:24 AM, Wim Van Leuven < wim.vanleu...@highestpoint.biz> wrote: > awesome! > > On Thu, 11 Jan 2018 at 23:48 Thunder Stumpges <thunder.stump...@gmail.com> > wrote: > > > Thanks, yes we upgraded to 1.0.0 and that has indeed fixed the issue. > > Thanks for the pointer! > > -Thunder > > > > > > On Tue, Jan 9, 2018 at 9:50 PM Wim Van Leuven < > > wim.vanleu...@highestpoint.biz> wrote: > > > > > Upgrade? > > > > > > On Wed, Jan 10, 2018, 00:26 Thunder Stumpges < > thunder.stump...@gmail.com > > > > > > wrote: > > > > > > > How would I know if we are seeing that issue? We are running 0.11.0.0 > > so > > > we > > > > would not have this fix. > > > > > > > > On Tue, Jan 9, 2018 at 11:07 AM Wim Van Leuven < > > > > wim.vanleu...@highestpoint.biz> wrote: > > > > > > > > > What minor version of Kafka are you running? Might you be impacted > by > > > > > https://issues.apache.org/jira/browse/KAFKA-6030? > > > > > -w > > > > > > > > > > On Tue, 9 Jan 2018 at 19:02 Thunder Stumpges < > > > thunder.stump...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > Hello, I posted this on StackOverflow > > > > > > < > > > > > > > > > > > > > > > > > > > > https://stackoverflow.com/questions/47948399/kafka-per- > topic-retention-bytes-and-global-log-retention-bytes-not-working > > > > > > >also > > > > > > but haven't gotten any response. > > > > > > > > > > > > thanks in advance, > > > > > > Thunder > > > > > > __________ > > > > > > > > > > > > We are running a 6 node cluster of kafka 0.11.0. We have set a > > global > > > > as > > > > > > well as a per-topic retention in bytes, neither of which is being > > > > > applied. > > > > > > There are no errors that I can see in the logs, just nothing > being > > > > > deleted > > > > > > (by size; the time retention does seem to be working) > > > > > > > > > > > > See relevant configs below: > > > > > > > > > > > > *./config/server.properties* : > > > > > > > > > > > > # global retention 75GB or 60 days, segment size 512MB > > > > > > log.retention.bytes=75000000000 > > > > > > log.retention.check.interval.ms=60000 > > > > > > > > > > > > log.retention.hours=1440 > > > > > > > > > > > > log.cleanup.policy=delete > > > > > > > > > > > > log.segment.bytes=536870912 > > > > > > > > > > > > *topic configuration (30GB):* > > > > > > > > > > > > [tstumpges@kafka-02 kafka]$ bin/kafka-topics.sh --zookeeper > > > > > > zk-01:2181/kafka --describe --topic stg_logtopic > > > > > > Topic:stg_logtopic PartitionCount:12 ReplicationFactor:3 > > > > > > Configs:retention.bytes=30000000000 > > > > > > Topic: stg_logtopic Partition: 0 Leader: 4 > > > > > > Replicas: 4,5,6 Isr: 4,5,6 > > > > > > Topic: stg_logtopic Partition: 1 Leader: 5 > > > > > > Replicas: 5,6,1 Isr: 5,1,6 > > > > > > ... > > > > > > > > > > > > And, disk usage showing 910GB usage for one partition! > > > > > > > > > > > > [tstumpges@kafka-02 kafka]$ sudo du -s -h /data1/kafka-data/* > > > > > > 82G /data1/kafka-data/stg_logother3-2 > > > > > > 155G /data1/kafka-data/stg_logother2-9 > > > > > > 169G /data1/kafka-data/stg_logother1-6 > > > > > > 910G /data1/kafka-data/stg_logtopic-4 > > > > > > > > > > > > I can see there are plenty of segment log files (512MB each) in > the > > > > > > partition directory... what is going on?! > > > > > > > > > > > > Thanks in advance, Thunder > > > > > > > > > > > > > > > > > > > > > -- Raghav
