Re: LogCleaner is not removing Transaction Records

2019-05-10 Thread Guozhang Wang
For who interested in this thread, there's a ticket created for it and we believe it is a lurking bug and are trying to fix it before the 2.3 release: https://issues.apache.org/jira/browse/KAFKA-8335 Guozhang On Fri, May 10, 2019 at 10:39 AM Michael Jaschob wrote: > Weichu, > > while I don't

Re: LogCleaner is not removing Transaction Records

2019-05-10 Thread Michael Jaschob
Weichu, while I don't have a solution we are seeing the same thing in our world. I put out a query to the mailing list a week or two ago (no responses unfortunately): https://lists.apache.org/thread.html/04273f5cfe4f6c6ed9ab370399f208a5cd780576880650aae839de25@%3Cusers.kafka.apache.org%3E . We're

LogCleaner is not removing Transaction Records

2019-05-07 Thread Weichu Liu
Hi, We are using Kafka Streams with exactly-once enabled on a Kafka cluster for a while. Recently we found that the size of __consumer_offsets partitions grew huge. Some partition went over 30G. This caused Kafka to take quite long to load "__consumer_offsets" topic on startup (it loads the topic