In summary, it appears the primary issue is that Kafka keeps file handles of each log segment open. Is there a way to configure this, or is a way to do so planned? It appears that an option to deduplicate instead of delete was added recently, so doesn't the file handle issue exist with that as well (since files aren't being deleted)?
Our use case is that we'd like to log data we don't need away and
potentially replay it at some point. We don't want to delete old logs. I
googled around a bit and I only discovered this particular post:
http://mail-archives.apache.org/mod_mbox/incubator-kafka-users/201210.mbox/%3CCAFbh0Q2=eJcDT6NvTAPtxhXSk64x0Yms-G-AOqOoy=ftvvm...@mail.gmail.com%3E
- Keeping logs forever Anthony Grimes
- Re: Keeping logs forever Eric Tschetter
- Re: Keeping logs forever Milind Parikh
- Re: Keeping logs forever Jay Kreps
- Re: Keeping logs forever graham sanderson
- Re: Keeping logs forever Eric Tschetter
- Re: Keeping logs forever Jay Kreps
- Re: Re: Keeping logs forever Anthony Grimes