[
https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14963574#comment-14963574
]
Vinoth Chandar commented on KAFKA-2580:
---------------------------------------
[~jjkoshy] Good point. if I understand correctly, even if say all consumers
start bootstrapping with startTime=earliest, which can just force opening of
all file handles, an LRU based scheme would keep closing the file handles
internally from oldest to latest file, which still is good behaviour. In order
to lessen the impact of fs.close() on old file by delegating to a background
thread, which takes a config that caps the number of items in the file handle
cache.
I like the cache approach better since it will be one place thru which all
access go,so future feature transparently play nicely with overall system
limits.
> Kafka Broker keeps file handles open for all log files (even if its not
> written to/read from)
> ---------------------------------------------------------------------------------------------
>
> Key: KAFKA-2580
> URL: https://issues.apache.org/jira/browse/KAFKA-2580
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.2.1
> Reporter: Vinoth Chandar
> Assignee: Grant Henke
>
> We noticed this in one of our clusters where we stage logs for a longer
> amount of time. It appears that the Kafka broker keeps file handles open even
> for non active (not written to or read from) files. (in fact, there are some
> threads going back to 2013
> http://grokbase.com/t/kafka/users/132p65qwcn/keeping-logs-forever)
> Needless to say, this is a problem and forces us to either artificially bump
> up ulimit (its already at 100K) or expand the cluster (even if we have
> sufficient IO and everything).
> Filing this ticket, since I could find anything similar. Very interested to
> know if there are plans to address this (given how Samza's changelog topic is
> meant to be a persistent large state use case).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
