I noticed the same issue too with 0.9. On Wed, 25 May 2016 at 09:49 Andrew Otto <o...@wikimedia.org> wrote:
> “We use the default log retention of 7 *days*" :)* > > On Wed, May 25, 2016 at 12:34 PM, Andrew Otto <o...@wikimedia.org> wrote: > > > Hiya, > > > > We’ve recently upgraded to 0.9. In 0.8, when we restarted a broker, data > > log file mtimes were not changed. In 0.9, any data log file that was on > > disk before the broker has it’s mtime modified to the time of the broker > > restart. > > > > This causes problems with log retention, as all the files then look like > > they contain recent data to kafka. We use the default log retention of 7 > > weeks, but if all the files are touched at the same time, this can cause > us > > to retain up to 2 weeks of log data, which can fill up our disks. > > > > We saw this during our initial upgrade, but I had just thought it had > > something to do with the change of inter.broker.protocol.version, and > > assumed it wouldn’t happen again. We just did our first broker restart > > after the upgrade, and we are seeing this again. We worked around this > > during our upgrade by temporarily setting a high volume topic’s retention > > very low, causing brokers to purge more recent data. This allowed us to > > avoid filling up our disks, but we shouldn’t have to do this every time > we > > bounce brokers. > > > > Has anyone else noticed this? > > > > -Ao > > >
