I have my purge set to 3 days and there are logs in there that are 3 days old but haven't been purged yet.
Older log segments are purged only when the log segment rolls over. This is controlled by the log.segment.bytes config. You might want to lower this in your cluster to let Kafka purge the segments relatively sooner. However, note that the side effect of doing so is increase in the # of open file handles. You might have to bump up the allowable # of open file handles per broker. Regarding your original question, yes you can delete older log and index files (provided you leave around the latest log and index file). Once you delete these, you will have to bounce the broker so it can clean the file handles from it's internal data structures. Hope that helps! -Neha On Tue, Mar 25, 2014 at 11:52 AM, Tom Amon <ta46...@gmail.com> wrote: > My apologies for mail bombing the list. I'm banging my head against a > production issue. > > In short, can I delete old log and index files manually? I have two brokers > out of five that are hosting ~1200 partitions each, though based on my > understanding from a previous email they should really only be hosting > about 700 and are sized as such. I'm running out of disk space rapidly. I > have my purge set to 3 days and there are logs in there that are 3 days old > but haven't been purged yet. I'm worried I'm going to run out of disk space > before they are deleted. > > Is this safe? > > I think I need to specify the size as well to help mitigate this but > without knowing how many partitions will be hosted it's tough to size that > as well. > > Thanks for your patience. >
