Slightly different from what I observed. Broker box has 800GB disk space. By setting the appropriate log retention, it's supposed to hold the log size. But then the usage of disk hits 90%, and by doing nothing but restarting broker server. It free 40% disk space. It's for sure the speed of the traffic won't be able to fill 40% disk space within one minute period (log.delete.delay.ms).
The obvious change before and after restart broker server is broker server frees tons of file descriptors of .index. Most of those file descriptors are very old. lsof -p <broker_pid> | grep .deleted Don't know how come kafka didn't release those file descriptors and what's the dial of it. On Sun, Mar 1, 2015 at 12:50 PM, Mayuresh Gharat <gharatmayures...@gmail.com > wrote: > Also I suppose when the broker starts up it will remove the files that are > marked with suffix .deleted and that's why you can see the free disk space > on restarting. Guozhang can correct me if I am wrong. > > Thanks, > > Mayuresh > > On Sat, Feb 28, 2015 at 9:27 PM, Guozhang Wang <wangg...@gmail.com> wrote: > > > Guangle, > > > > The deletion of the segment log / index files are async, i.e. when Kafka > > decide to clean the logs, it only adds a suffix ".deleted" to the files > > such that it will not be access any more by other Kafka threads. The > actual > > file deletion will be executed later, with period controlled by " > > file.delete.delay.ms" (default 1 minute). > > > > On Fri, Feb 27, 2015 at 9:49 PM, Guangle Fan <fanguan...@gmail.com> > wrote: > > > > > Hi, > > > > > > After Kafka cleaned .log / .index files based on topic retention. I can > > > still lsof a lot of .index.deleted files. And df shows usage on disk > > space > > > is accumulated to full. > > > > > > When this happened, just by restarting broker, it will immediately free > > > those disk space. I seems to me kafka after cleaning expired files > still > > > hold file descriptors which lead to disk space still being held. > > > > > > How do you config kafka to let kafka release file descriptors in this > > case > > > ? > > > > > > Using kafka 0.8.1.1 > > > > > > Regards, > > > > > > Guangle > > > > > > > > > > > -- > > -- Guozhang > > > > > > -- > -Regards, > Mayuresh R. Gharat > (862) 250-7125 >
