Re: Spread log segment deletion over a couple hours

2018-05-03 Thread Vincent Rischmann
Hi Johnathan. Yes I decreased the retention on all topics simultaneously. I realized my mistake later when I saw the cluster overloaded :) I wasn't 100% sure so I looked it up, but it looks to me like log.cleaner.threads and log.cleaner.io.max.bytes.per.second only apply when a topic is using

Re: Spread log segment deletion over a couple hours

2018-05-02 Thread Jonathan Bethune
Howdy Vincent. Sounds like a painful situation! I have experienced similar drama with Kafka so maybe I can offer some advice. You said you decreased the retention time on 4 topics. I wonder, was this done on all 4 topics at the same time? Depending on broker and partition config, that can be ver

Spread log segment deletion over a couple hours

2018-05-02 Thread Vincent Rischmann
Hi, I'm wondering if there is a way to tell Kafka to spread the log file deletion when decreasing the retention time of a topic, and if not, if it would make sense. I'm asking because this afternoon, after decreasing the retention time from 2 months to 1 month on 4 of my topics, the whole cluster