[
https://issues.apache.org/jira/browse/KAFKA-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14550716#comment-14550716
]
Manikumar Reddy commented on KAFKA-2201:
----------------------------------------
On Linux machines, tuning system ulimits (virtual memory, max lock memory,
max_map_count ) may solve this issue. Also for try to upgrade (or) downgrade
Java version.
> Open file handle leak
> ---------------------
>
> Key: KAFKA-2201
> URL: https://issues.apache.org/jira/browse/KAFKA-2201
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.2.1
> Environment: Debian Linux 7, 64 bit
> Oracle JDK 1.7.0u40, 64-bit
> Reporter: Albert Visagie
>
> The kafka broker crashes with the following stack trace from the server.log
> roughly every 18 hours:
> [2015-05-19 07:39:22,924] FATAL [KafkaApi-0] Halting due to unrecoverable I/O
> error while handling produce request: (kafka.server.KafkaApis)
> kafka.common.KafkaStorageException: I/O exception in append to log 'nnnnnnn-1'
> at kafka.log.Log.append(Log.scala:266)
> at
> kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition.scala:379)
> at
> kafka.cluster.Partition$$anonfun$appendMessagesToLeader$1.apply(Partition.scala:365)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.utils.Utils$.inReadLock(Utils.scala:541)
> at kafka.cluster.Partition.appendMessagesToLeader(Partition.scala:365)
> at
> kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:291)
> at
> kafka.server.KafkaApis$$anonfun$appendToLocalLog$2.apply(KafkaApis.scala:282)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at kafka.server.KafkaApis.appendToLocalLog(KafkaApis.scala:282)
> at
> kafka.server.KafkaApis.handleProducerOrOffsetCommitRequest(KafkaApis.scala:204)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:59)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:59)
> at java.lang.Thread.run(Thread.java:724)
> Caused by: java.io.IOException: Map failed
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:888)
> at
> kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:286)
> at
> kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:276)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.log.OffsetIndex.resize(OffsetIndex.scala:276)
> at
> kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply$mcV$sp(OffsetIndex.scala:265)
> at
> kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply(OffsetIndex.scala:265)
> at
> kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply(OffsetIndex.scala:265)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at kafka.log.OffsetIndex.trimToValidSize(OffsetIndex.scala:264)
> at kafka.log.Log.roll(Log.scala:563)
> at kafka.log.Log.maybeRoll(Log.scala:539)
> at kafka.log.Log.append(Log.scala:306)
> ... 21 more
> Caused by: java.lang.OutOfMemoryError: Map failed
> at sun.nio.ch.FileChannelImpl.map0(Native Method)
> at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:885)
> ... 33 more
> The Kafka broker's open filehandles as seen by
> lsof | grep pid | wc -l
> grows steadily as it runs. Under our load it lasts about 18 hours before
> crashing with the stack trace above.
> We were experimenting with settings under Log Retention Policy in
> server.properties:
> log.retention.hours=168
> log.retention.bytes=107374182
> log.segment.bytes=1073741
> log.retention.check.interval.ms=3000
> The result is that the broker rolls over segments quite rapidly. We don't
> have to run it that way of course.
> We are running only one broker at the moment.
> lsof shows many open files without size and absent from ls in the log
> directory with the suffix ".deleted"
> This is kafka 0.8.2.1 with scala 2.10.4 as downloaded from the website last
> week.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
