“We use the default log retention of 7 *days*" :)*

On Wed, May 25, 2016 at 12:34 PM, Andrew Otto <o...@wikimedia.org> wrote:

> Hiya,
>
> We’ve recently upgraded to 0.9.  In 0.8, when we restarted a broker, data
> log file mtimes were not changed.  In 0.9, any data log file that was on
> disk before the broker has it’s mtime modified to the time of the broker
> restart.
>
> This causes problems with log retention, as all the files then look like
> they contain recent data to kafka.  We use the default log retention of 7
> weeks, but if all the files are touched at the same time, this can cause us
> to retain up to 2 weeks of log data, which can fill up our disks.
>
> We saw this during our initial upgrade, but I had just thought it had
> something to do with the change of inter.broker.protocol.version, and
> assumed it wouldn’t happen again.  We just did our first broker restart
> after the upgrade, and we are seeing this again.  We worked around this
> during our upgrade by temporarily setting a high volume topic’s retention
> very low, causing brokers to purge more recent data.  This allowed us to
> avoid filling up our disks, but we shouldn’t have to do this every time we
> bounce brokers.
>
> Has anyone else noticed this?
>
> -Ao
>

Reply via email to