[
https://issues.apache.org/jira/browse/KAFKA-2580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14908905#comment-14908905
]
Guozhang Wang edited comment on KAFKA-2580 at 9/25/15 11:56 PM:
----------------------------------------------------------------
Hey [~vinothchandar], could you let me know the log file segment size and log
retention settings? I would like to know better of your use cases and how it
results in this many open file handlers.
I think at LI the ulimit has been set at 200K with segment size set as 1GB by
default, which works fine AFAIK ([~jjkoshy] can help confirming that). Adding a
feature that closes inactive segments' open file handlers and re-open them upon
being read / written again is possible, but would be tricky.
was (Author: guozhang):
Hey [~vinothchandar], could you let me know the log file segment size and log
retention settings? I would like to know better of your use cases and how it
results in this many open file handlers.
I think at LI the ulimit has been set at 200K with segment size set as 1GB by
default, which works fine AFAIK. Adding a feature that closes inactive
segments' open file handlers and re-open them upon being read / written again
is possible, but would be tricky.
> Kafka Broker keeps file handles open for all log files (even if its not
> written to/read from)
> ---------------------------------------------------------------------------------------------
>
> Key: KAFKA-2580
> URL: https://issues.apache.org/jira/browse/KAFKA-2580
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.2.1
> Reporter: Vinoth Chandar
>
> We noticed this in one of our clusters where we stage logs for a longer
> amount of time. It appears that the Kafka broker keeps file handles open even
> for non active (not written to or read from) files. (in fact, there are some
> threads going back to 2013
> http://grokbase.com/t/kafka/users/132p65qwcn/keeping-logs-forever)
> Needless to say, this is a problem and forces us to either artificially bump
> up ulimit (its already at 100K) or expand the cluster (even if we have
> sufficient IO and everything).
> Filing this ticket, since I could find anything similar. Very interested to
> know if there are plans to address this (given how Samza's changelog topic is
> meant to be a persistent large state use case).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
