Well, it sounds like files were deleted while Kafka still had them open. Or something else opened them while Kafka deleted them. I haven't noticed this on our systems but we haven't looked for it either.
Is anything outside of Kafka deleting or reading those files? On May 23, 2013 1:17 AM, "Jason Rosenberg" <j...@squareup.com> wrote: > So, does this indicate kafka (or the jvm itself) is not aggressively > closing file handles of deleted files? Is there a fix for this? Or is > there not likely anything to be done? What happens if the disk fills up > with file handles for phantom deleted files? > > Jason > > > On Wed, May 22, 2013 at 9:50 PM, Jonathan Creasy <j...@box.com> wrote: > >> It isn't uncommon if a process has an open file handle on a file that is >> deleted, the space is not freed until the handle is closed. So restarting >> the process that has a handle on the file would cause the space to be >> freed >> also. >> >> You can troubleshoot that with lsof. >> Normally, I see 2-4 log segments deleted every hour in my brokers. I see >> log lines like this: >> >> 2013-05-23 04:40:06,857 INFO [kafka-logcleaner-0] log.LogManager - >> Deleting log segment 00000000035434043157.kafka from <redacted topic> >> >> However, it seems like if I restart the broker, a massive amount of disk >> space is freed (without a corresponding flood of these log segment deleted >> messages). Is there an explanation for this? Does kafka keep reference >> to >> file segments around, and reuse them as needed or something? And then or >> restart, the references to those free segment files are dropped? >> >> Thoughts? >> >> This is with 0.7.2. >> >> Jason >> > >
