Segments Not Compacted Due to Empty Transactional Record Batches

2019-04-26 Thread Michael Jaschob
We're seeing an issue with some __consumer_offsets partitions that host consumer groups being used in EOS-style read-process-write cycles. Although the brokers' log cleaners are running, we are finding more and more segment files for these partitions, for example: ``` $ ls -lh data/__consumer_offs

Re: LogCleaner is not removing Transaction Records

2019-05-10 Thread Michael Jaschob
Weichu, while I don't have a solution we are seeing the same thing in our world. I put out a query to the mailing list a week or two ago (no responses unfortunately): https://lists.apache.org/thread.html/04273f5cfe4f6c6ed9ab370399f208a5cd780576880650aae839de25@%3Cusers.kafka.apache.org%3E . We're

Re: Evergrowing __consumer_offsets topic

2019-05-14 Thread Michael Jaschob
Claudia, This sounds like an issue I and other users have reported recently: https://issues.apache.org/jira/browse/KAFKA-8335. Looks like a fix was merged yesterday. Michael On Tue, May 14, 2019 at 4:44 AM Claudia Wegmann wrote: > Dear kafka users, > > I run a kafka cluster (version 2.1.1) wit