[
https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14984243#comment-14984243
]
Vinoth Chandar commented on KAFKA-2580:
---------------------------------------
Based on this, looks like we can close this?
>> So a lot of this comes down to the implementation. A naive 10k item LRU
>> cache could easily be far more memory hungry than having 50k open FDs, plus
>> being in heap this would add a huge number of objects to manage.
[~jkreps] I am a little confused. What I meant by LRU cache was simply limiting
the number of "java.io.File" objects (or equivalent in Kafka codebase) that
represents the handle to the segment. So, if there are 10K such objects in a
(properly sized) ConcurrentHashMap, how would that add to the memory overhead
so much, compared to holding 50K/200K objects anyway?
> Kafka Broker keeps file handles open for all log files (even if its not
> written to/read from)
> ---------------------------------------------------------------------------------------------
>
> Key: KAFKA-2580
> URL: https://issues.apache.org/jira/browse/KAFKA-2580
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.2.1
> Reporter: Vinoth Chandar
> Assignee: Grant Henke
>
> We noticed this in one of our clusters where we stage logs for a longer
> amount of time. It appears that the Kafka broker keeps file handles open even
> for non active (not written to or read from) files. (in fact, there are some
> threads going back to 2013
> http://grokbase.com/t/kafka/users/132p65qwcn/keeping-logs-forever)
> Needless to say, this is a problem and forces us to either artificially bump
> up ulimit (its already at 100K) or expand the cluster (even if we have
> sufficient IO and everything).
> Filing this ticket, since I could find anything similar. Very interested to
> know if there are plans to address this (given how Samza's changelog topic is
> meant to be a persistent large state use case).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
