Re: kafka deleted old logs but not released

2015-01-31 Thread Jaikiran Pai
Hi Yonghui, I have tried a few ways with different retention strategies to try and reproduce this issue, but haven't been able to do it. Since it looks like you can consistently reproduce this, would you able to share a sample reproducible application (maybe as a github repo) for this issue?

Re: kafka deleted old logs but not released

2015-01-26 Thread nitin sharma
hi All, I am facing the same issue.. i have got Kafka_2.9.2-0.8.1.1.jar deployed in my application. My Operation team had reported significant increase in the directory where Kafak-log folder exists . when i checked the size of parent directory is it 99GB (100%) utilized.. This was very confusin

Re: kafka deleted old logs but not released

2015-01-25 Thread Yonghui Zhao
I have fixed this issue like this patch https://reviews.apache.org/r/29755/diff/5/. I find rename failure still happens: server.log.2015-01-26-06:[2015-01-26 06:10:54,513] ERROR File rename failed, forcefully deleting file (kafka.log.Log) server.log.2015-01-26-06:[2015-01-26 06:10:54,600] ERROR F

Re: kafka deleted old logs but not released

2015-01-25 Thread Jay Kreps
Also, what is the configuration for the servers? In particular it would be good to know the retention and/or log compaction settings as those delete files. -Jay On Sun, Jan 25, 2015 at 4:34 AM, Jaikiran Pai wrote: > Hi Yonghui, > > Do you still have this happening? If yes, can you tell us a bit

Re: kafka deleted old logs but not released

2015-01-25 Thread Jaikiran Pai
Hi Yonghui, Do you still have this happening? If yes, can you tell us a bit more about your setup? Is there something else that accesses or maybe deleting these log files? For more context to this question, please read the discussion related to this here http://mail-archives.apache.org/mod_mb

Re: kafka deleted old logs but not released

2015-01-08 Thread Jaikiran Pai
Create a JIRA for this https://issues.apache.org/jira/browse/KAFKA-1853 -Jaikiran On Thursday 08 January 2015 01:18 PM, Jaikiran Pai wrote: Apart from the fact that the file rename is failing (the API notes that there are chances of the rename failing), it looks like the implementation in FileM

Re: kafka deleted old logs but not released

2015-01-07 Thread Jaikiran Pai
Apart from the fact that the file rename is failing (the API notes that there are chances of the rename failing), it looks like the implementation in FileMessageSet's rename can cause a couple of issues, one of them being a leak. The implementation looks like this https://github.com/apache/ka

Re: kafka deleted old logs but not released

2015-01-07 Thread Yonghui Zhao
CentOS release 6.3 (Final) 2015-01-07 22:18 GMT+08:00 Harsha : > Yonghui, >Which OS you are running. > -Harsha > > On Wed, Jan 7, 2015, at 01:38 AM, Yonghui Zhao wrote: > > Yes and I found the reason rename in deletion is failed. > > In rename progress the files is deleted? and then

Re: kafka deleted old logs but not released

2015-01-07 Thread Harsha
Yonghui, Which OS you are running. -Harsha On Wed, Jan 7, 2015, at 01:38 AM, Yonghui Zhao wrote: > Yes and I found the reason rename in deletion is failed. > In rename progress the files is deleted? and then exception blocks file > closed in kafka. > But I don't know how can rename fai

Re: kafka deleted old logs but not released

2015-01-07 Thread Yonghui Zhao
Yes and I found the reason rename in deletion is failed. In rename progress the files is deleted? and then exception blocks file closed in kafka. But I don't know how can rename failure happen, [2015-01-07 00:10:48,685] ERROR Uncaught exception in scheduled task 'kafka-log-retention' (kafka.utils

Re: kafka deleted old logs but not released

2015-01-06 Thread Jun Rao
Do you mean that the Kafka broker still holds a file handler on a deleted file? Do you see those files being deleted in the Kafka log4j log? Thanks, Jun On Tue, Jan 6, 2015 at 4:46 AM, Yonghui Zhao wrote: > Hi, > > We use kafka_2.10-0.8.1.1 in our server. Today we found disk space alert. > > W