I see DEL file descriptors as well.
java2978 root DELREG 202,16 1074192884
/mnt/data1/kafka/flirt_tasks-1/000143874814.index.deleted
Using java from Oracle
java version "1.7.0_72"
Java(TM) SE Runtime Environment (build 1.7.0_72-b14)
Java HotSpot(TM) 64-Bit
One thing to remember is that the .index files are memory-mapped [1]
which in Java means that the file descriptors may not be released even
when the program is done using it. A garbage collection is expected to
close such resources, but forcing a System.gc() is only a hint and thus
doesn't guar
Did you check if log.delete.delay.ms is set to 6?
Guozhang
On Sun, Mar 1, 2015 at 9:00 PM, Guangle Fan wrote:
> Slightly different from what I observed.
>
> Broker box has 800GB disk space. By setting the appropriate log retention,
> it's supposed to hold the log size. But then the usage of
Slightly different from what I observed.
Broker box has 800GB disk space. By setting the appropriate log retention,
it's supposed to hold the log size. But then the usage of disk hits 90%,
and by doing nothing but restarting broker server. It free 40% disk space.
It's for sure the speed of the tra
Also I suppose when the broker starts up it will remove the files that are
marked with suffix .deleted and that's why you can see the free disk space
on restarting. Guozhang can correct me if I am wrong.
Thanks,
Mayuresh
On Sat, Feb 28, 2015 at 9:27 PM, Guozhang Wang wrote:
> Guangle,
>
> The
Guangle,
The deletion of the segment log / index files are async, i.e. when Kafka
decide to clean the logs, it only adds a suffix ".deleted" to the files
such that it will not be access any more by other Kafka threads. The actual
file deletion will be executed later, with period controlled by "
fi
Hi,
After Kafka cleaned .log / .index files based on topic retention. I can
still lsof a lot of .index.deleted files. And df shows usage on disk space
is accumulated to full.
When this happened, just by restarting broker, it will immediately free
those disk space. I seems to me kafka after cleani