I have fixed this issue like this patch https://reviews.apache.org/r/29755/diff/5/.
I find rename failure still happens: server.log.2015-01-26-06:[2015-01-26 06:10:54,513] ERROR File rename failed, forcefully deleting file (kafka.log.Log) server.log.2015-01-26-06:[2015-01-26 06:10:54,600] ERROR File rename failed, forcefully deleting file (kafka.log.Log) server.log.2015-01-26-06:[2015-01-26 06:10:54,685] ERROR File rename failed, forcefully deleting file (kafka.log.Log) server.log.2015-01-26-06:[2015-01-26 06:10:54,797] ERROR File rename failed, forcefully deleting file (kafka.log.Log) .... And use lsof I can still find some files opened by kafka but deleted, but those files sizes are 0. java 3228 root 34uw REG 253,2 0 26084228 /home/work/data/soft/kafka-0.8/data/.lock (deleted) java 3228 root 35u REG 253,2 0 26084232 /home/work/data/soft/kafka-0.8/data/cube-0/00000000000000000000.log (deleted) java 3228 root 36u REG 253,2 0 26869778 /home/work/data/soft/kafka-0.8/data/_oakbay_v2_search_topic_misearch_appstore-search-0/00000000000000003116.log (deleted) java 3228 root 37u REG 253,2 0 26084234 /home/work/data/soft/kafka-0.8/data/_oakbay_v2_search_topic_mishop-search_mishop_v1-0/00000000000000000000.log (deleted) Here is my configuration: Binary: kafka_2.10-0.8.1.1 Retention config: # The minimum age of a log file to be eligible for deletion log.retention.hours=168 # A size-based retention policy for logs. Segments are pruned from the log as long as the remaining # segments don't drop below log.retention.bytes. #log.retention.bytes=1073741824 # The maximum size of a log segment file. When this size is reached a new log segment will be created. log.segment.bytes=536870912 # The interval at which log segments are checked to see if they can be deleted according # to the retention policies log.retention.check.interval.ms=60000 # By default the log cleaner is disabled and the log retention policy will default to just delete segments after their retention expires. # If log.cleaner.enable=true is set the cleaner will be enabled and individual logs can then be marked for log compaction. log.cleaner.enable=false OS: CentOS release 6.4 (Final) JDK: *java version "1.6.0_37"* Java(TM) SE Runtime Environment (build 1.6.0_37-b06) Java HotSpot(TM) 64-Bit Server VM (build 20.12-b01, mixed mode) JDK is too old, but not sure if this result in rename failure. 2015-01-26 0:42 GMT+08:00 Jay Kreps <jay.kr...@gmail.com>: > Also, what is the configuration for the servers? In particular it would be > good to know the retention and/or log compaction settings as those delete > files. > > -Jay > > On Sun, Jan 25, 2015 at 4:34 AM, Jaikiran Pai <jai.forums2...@gmail.com> > wrote: > >> Hi Yonghui, >> >> Do you still have this happening? If yes, can you tell us a bit more >> about your setup? Is there something else that accesses or maybe deleting >> these log files? For more context to this question, please read the >> discussion related to this here http://mail-archives.apache. >> org/mod_mbox/kafka-dev/201501.mbox/%3C54C47E9B.5060401%40gmail.com%3E >> >> >> -Jaikiran >> >> >>> On Thursday 08 January 2015 11:19 AM, Yonghui Zhao wrote: >>> >>>> CentOS release 6.3 (Final) >>>> >>>> >>>> 2015-01-07 22:18 GMT+08:00 Harsha <ka...@harsha.io>: >>>> >>>> Yonghui, >>>>> Which OS you are running. >>>>> -Harsha >>>>> >>>>> On Wed, Jan 7, 2015, at 01:38 AM, Yonghui Zhao wrote: >>>>> >>>>>> Yes and I found the reason rename in deletion is failed. >>>>>> In rename progress the files is deleted? and then exception blocks >>>>>> file >>>>>> closed in kafka. >>>>>> But I don't know how can rename failure happen, >>>>>> >>>>>> [2015-01-07 00:10:48,685] ERROR Uncaught exception in scheduled task >>>>>> 'kafka-log-retention' (kafka.utils.KafkaScheduler) >>>>>> kafka.common.KafkaStorageException: Failed to change the log file >>>>>> suffix >>>>>> from to .deleted for log segment 70781650 >>>>>> at kafka.log.LogSegment.changeFileSuffixes(LogSegment. >>>>>> scala:249) >>>>>> at kafka.log.Log.kafka$log$Log$$ >>>>>> asyncDeleteSegment(Log.scala:636) >>>>>> at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:627) >>>>>> at >>>>>> kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:415) >>>>>> at >>>>>> kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:415) >>>>>> at scala.collection.immutable.List.foreach(List.scala:318) >>>>>> at kafka.log.Log.deleteOldSegments(Log.scala:415) >>>>>> at >>>>>> >>>>>> >>>>>> kafka.log.LogManager.kafka$log$LogManager$$cleanupExpiredSegments(LogManager.scala:325) >>>>> >>>>> >>>>>> at >>>>>> kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:356) >>>>>> >>>>>> at >>>>>> kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:354) >>>>>> >>>>>> at >>>>>> >>>>>> scala.collection.TraversableLike$WithFilter$$ >>>>> anonfun$foreach$1.apply(TraversableLike.scala:772) >>>>> >>>>>> at scala.collection.Iterator$class.foreach(Iterator.scala: >>>>>> 727) >>>>>> at scala.collection.AbstractIterator.foreach( >>>>>> Iterator.scala:1157) >>>>>> at >>>>>> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) >>>>>> at scala.collection.AbstractIterable.foreach( >>>>>> Iterable.scala:54) >>>>>> at >>>>>> >>>>>> scala.collection.TraversableLike$WithFilter. >>>>> foreach(TraversableLike.scala:771) >>>>> >>>>>> at kafka.log.LogManager.cleanupLogs(LogManager.scala:354) >>>>>> at >>>>>> >>>>>> >>>>>> kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:141) >>>>> >>>>> >>>>>> at >>>>>> kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100) >>>>>> at >>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) >>>>>> >>>>>> at >>>>>> >>>>>> >>>>>> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) >>>>> >>>>> >>>>>> at >>>>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) >>>>>> at >>>>>> >>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ >>>>> ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) >>>>> >>>>>> at >>>>>> >>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ >>>>> ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180) >>>>> >>>>>> at >>>>>> >>>>>> java.util.concurrent.ScheduledThreadPoolExecutor$ >>>>> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204) >>>>> >>>>>> at >>>>>> >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker. >>>>> runTask(ThreadPoolExecutor.java:886) >>>>> >>>>>> at >>>>>> >>>>>> >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >>>>> >>>>> >>>>>> at java.lang.Thread.run(Thread.java:662) >>>>>> >>>>>> >>>>>> 2015-01-07 13:56 GMT+08:00 Jun Rao <j...@confluent.io>: >>>>>> >>>>>> Do you mean that the Kafka broker still holds a file handler on a >>>>>>> >>>>>> deleted >>>>> >>>>>> file? Do you see those files being deleted in the Kafka log4j log? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Jun >>>>>>> >>>>>>> On Tue, Jan 6, 2015 at 4:46 AM, Yonghui Zhao <zhaoyong...@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>> Hi, >>>>>>>> >>>>>>>> We use kafka_2.10-0.8.1.1 in our server. Today we found disk space >>>>>>>> >>>>>>> alert. >>>>> >>>>>> We find many kafka data files are deleted, but still opened by kafka. >>>>>>>> >>>>>>>> such as: >>>>>>>> >>>>>>>> _yellowpageV2-0/00000000000068170670.log (deleted) >>>>>>>> java 8446 root 724u REG 253,2 >>>>>>>> >>>>>>> 536937911 >>>>> >>>>>> 26087362 >>>>>>>> >>>>>>>> >>>>>>>> /home/work/data/soft/kafka-0.8/data/_oakbay_v2_search_ >>>>> topic_ypgsearch_yellowpageV2-0/00000000000068818668.log >>>>> >>>>>> (deleted) >>>>>>>> java 8446 root 725u REG 253,2 >>>>>>>> >>>>>>> 536910838 >>>>> >>>>>> 26087364 >>>>>>>> >>>>>>>> >>>>>>>> /home/work/data/soft/kafka-0.8/data/_oakbay_v2_search_ >>>>> topic_ypgsearch_yellowpageV2-0/00000000000069457098.log >>>>> >>>>>> (deleted) >>>>>>>> java 8446 root 726u REG 253,2 >>>>>>>> >>>>>>> 536917902 >>>>> >>>>>> 26087368 >>>>>>>> >>>>>>>> >>>>>>>> /home/work/data/soft/kafka-0.8/data/_oakbay_v2_search_ >>>>> topic_ypgsearch_yellowpageV2-0/00000000000070104914.log >>>>> >>>>>> (deleted) >>>>>>>> >>>>>>>> >>>>>>>> Is there anything wrong or wrong configed? >>>>>>>> >>>>>>>> >>> >> >
