Did you check if log.delete.delay.ms is set to 60000? Guozhang
On Sun, Mar 1, 2015 at 9:00 PM, Guangle Fan <fanguan...@gmail.com> wrote: > Slightly different from what I observed. > > Broker box has 800GB disk space. By setting the appropriate log retention, > it's supposed to hold the log size. But then the usage of disk hits 90%, > and by doing nothing but restarting broker server. It free 40% disk space. > It's for sure the speed of the traffic won't be able to fill 40% disk space > within one minute period (log.delete.delay.ms). > > The obvious change before and after restart broker server is broker server > frees tons of file descriptors of .index. Most of those file descriptors > are very old. > > lsof -p <broker_pid> | grep .deleted > > Don't know how come kafka didn't release those file descriptors and what's > the dial of it. > > > > > > > > > On Sun, Mar 1, 2015 at 12:50 PM, Mayuresh Gharat < > gharatmayures...@gmail.com > > wrote: > > > Also I suppose when the broker starts up it will remove the files that > are > > marked with suffix .deleted and that's why you can see the free disk > space > > on restarting. Guozhang can correct me if I am wrong. > > > > Thanks, > > > > Mayuresh > > > > On Sat, Feb 28, 2015 at 9:27 PM, Guozhang Wang <wangg...@gmail.com> > wrote: > > > > > Guangle, > > > > > > The deletion of the segment log / index files are async, i.e. when > Kafka > > > decide to clean the logs, it only adds a suffix ".deleted" to the files > > > such that it will not be access any more by other Kafka threads. The > > actual > > > file deletion will be executed later, with period controlled by " > > > file.delete.delay.ms" (default 1 minute). > > > > > > On Fri, Feb 27, 2015 at 9:49 PM, Guangle Fan <fanguan...@gmail.com> > > wrote: > > > > > > > Hi, > > > > > > > > After Kafka cleaned .log / .index files based on topic retention. I > can > > > > still lsof a lot of .index.deleted files. And df shows usage on disk > > > space > > > > is accumulated to full. > > > > > > > > When this happened, just by restarting broker, it will immediately > free > > > > those disk space. I seems to me kafka after cleaning expired files > > still > > > > hold file descriptors which lead to disk space still being held. > > > > > > > > How do you config kafka to let kafka release file descriptors in this > > > case > > > > ? > > > > > > > > Using kafka 0.8.1.1 > > > > > > > > Regards, > > > > > > > > Guangle > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > > > -- > > -Regards, > > Mayuresh R. Gharat > > (862) 250-7125 > > > -- -- Guozhang
