Jason, Kafka closes the handler of all delete files. Otherwise, the broker will run out of file handler quickly.
Thanks, Jun On Wed, May 22, 2013 at 10:17 PM, Jason Rosenberg <j...@squareup.com> wrote: > So, does this indicate kafka (or the jvm itself) is not aggressively > closing file handles of deleted files? Is there a fix for this? Or is > there not likely anything to be done? What happens if the disk fills up > with file handles for phantom deleted files? > > Jason > > > On Wed, May 22, 2013 at 9:50 PM, Jonathan Creasy <j...@box.com> wrote: > > > It isn't uncommon if a process has an open file handle on a file that is > > deleted, the space is not freed until the handle is closed. So restarting > > the process that has a handle on the file would cause the space to be > freed > > also. > > > > You can troubleshoot that with lsof. > > Normally, I see 2-4 log segments deleted every hour in my brokers. I see > > log lines like this: > > > > 2013-05-23 04:40:06,857 INFO [kafka-logcleaner-0] log.LogManager - > > Deleting log segment 00000000035434043157.kafka from <redacted topic> > > > > However, it seems like if I restart the broker, a massive amount of disk > > space is freed (without a corresponding flood of these log segment > deleted > > messages). Is there an explanation for this? Does kafka keep reference > to > > file segments around, and reuse them as needed or something? And then or > > restart, the references to those free segment files are dropped? > > > > Thoughts? > > > > This is with 0.7.2. > > > > Jason > > >
