Re: Log Compaction Not Picking up Topic [solved]

2017-10-25 Thread Elmar Weber
Hello Xin, hello Jan, worked perfectly. I did a build of an image based on 0.11.0.1 and applied the missing patch, cleaning went through and resulted in the expected size. Thanks a lot for the quick help, Elmar On 10/25/2017 1:03 PM, Xin Li wrote: Hey Elmar, The only thing you need to do i

Re: Log Compaction Not Picking up Topic

2017-10-25 Thread Xin Li
Hey Elmar, The only thing you need to do is upgrade, Kafka track cleaned offset using cleaner-offset-checkpoint file. Best, Xin Xin Li Data EngineeringXin.Li@ trivago.com www.trivago.com

Re: Log Compaction Not Picking up Topic

2017-10-25 Thread Xin Li
Hey, Because of the overflow the calculation for dirty ratios is minus, and I guess upgrade is the one time for good fix. And we running that for quite a while, so far so good. Best, Xin Xin Li Data EngineeringXin.Li@ trivago.com <

Re: Log Compaction Not Picking up Topic

2017-10-25 Thread Elmar Weber
Hi, thanks, I'll give it a try, we run on Kubernetes so it's not a big issue to replicate the whole env including data. One question I'd have left: - How can I force a re-compaction over the whole topic? Because I guess the Log Cleaner market everything so far as not able to clean, how wil

Re: Log Compaction Not Picking up Topic

2017-10-25 Thread Jan Filipiak
Hi, unfortunatly there is nothing trivial you could do here. Without upgrading your kafkas you can only bounce the partition back and forth between brokers so they compact while its still small. With upgrading you could also just cherrypick this very commit or put a logstatement to verify.

Re: Log Compaction Not Picking up Topic

2017-10-25 Thread Elmar Weber
Hi, On 10/25/2017 12:15 PM, Xin Li wrote: > I think that is a bug, and should be fixed in this task https://issues.apache.org/jira/browse/KAFKA-6030. > We experience that in our kafka cluster, we just check out the 11.0.2 version, build it ourselves. thanks for the hint, as it looks like a c

Re: Log Compaction Not Picking up Topic

2017-10-25 Thread Xin Li
Hey, I think that is a bug, and should be fixed in this task https://issues.apache.org/jira/browse/KAFKA-6030. We experience that in our kafka cluster, we just check out the 11.0.2 version, build it ourselves. Best, Xin Xin Li Data EngineeringXin.Li@

Re: Log Compaction Not Picking up Topic

2017-10-25 Thread Elmar Weber
On 10/25/2017 12:03 PM, Manikumar wrote: any errors in log cleaner logs? Not as far as I can see root@kafka-1:/opt/kafka/logs# cat log-cleaner.log* | grep -i error (empty) However, I've seen that it actually did cleaning of the whole topic (excerpts below), but it didn't seem to find anythin

Re: Log Compaction Not Picking up Topic

2017-10-25 Thread Manikumar
any errors in log cleaner logs? On Wed, Oct 25, 2017 at 3:12 PM, Elmar Weber wrote: > Hello, > > I'm having trouble getting Kafka to compact a topic. It's over 300GB and > has enough segments to warrant cleaning. It should only be about 40 GB > (there is a copy in a db that is unique on the key)

Log Compaction Not Picking up Topic

2017-10-25 Thread Elmar Weber
Hello, I'm having trouble getting Kafka to compact a topic. It's over 300GB and has enough segments to warrant cleaning. It should only be about 40 GB (there is a copy in a db that is unique on the key). Below are the configs we have (default broker) and topic override. Is there something I