[
https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943698#comment-14943698
]
Joel Koshy commented on KAFKA-2580:
-----------------------------------
I was wondering if we can do with something much simpler - basically close out
file handles if they haven't been accessed after "x" minutes. The file-handle
cache approach has some benefit over this in that it may allow you to close out
unused file handles quicker than the other approach, but in both cases you have
to account for the worst case scenario - which is the worst case expected
number of bootstrapping consumers * number of segments in the logs that they
consume from. Log recovery is another scenario where we may need to open
several logs over a short span of time but I think that can be addressed by
closing out segments immediately after scanning them during recovery. That
said, I'm not very clear on how useful all of this is - maybe that's because I
don't do Kafka operations on a day to day basis :) [~toddpalino] what do you
think?
> Kafka Broker keeps file handles open for all log files (even if its not
> written to/read from)
> ---------------------------------------------------------------------------------------------
>
> Key: KAFKA-2580
> URL: https://issues.apache.org/jira/browse/KAFKA-2580
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.2.1
> Reporter: Vinoth Chandar
>
> We noticed this in one of our clusters where we stage logs for a longer
> amount of time. It appears that the Kafka broker keeps file handles open even
> for non active (not written to or read from) files. (in fact, there are some
> threads going back to 2013
> http://grokbase.com/t/kafka/users/132p65qwcn/keeping-logs-forever)
> Needless to say, this is a problem and forces us to either artificially bump
> up ulimit (its already at 100K) or expand the cluster (even if we have
> sufficient IO and everything).
> Filing this ticket, since I could find anything similar. Very interested to
> know if there are plans to address this (given how Samza's changelog topic is
> meant to be a persistent large state use case).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
