Hello Experts , We have deployed 10 node kafka cluster in production. Recently two of the nodes went down due to network problem and we brought it up after 24 hours. At the time of bootstrapping the kafka service on the failed nodes , we have seen the below error & broker failed to come up.
Kafka Version : kafka_2.11-2.2.0 JVM Options : /a/java64/jdk1.8.0/bin/java -Xmx15G -Xms10G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true -Xloggc:/a/opt/kafka/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=100M -Davoid_insecure_jmxremote [2019-09-03 10:54:10,630] ERROR Error while deleting the clean shutdown file in dir /tmp/data (kafka.server.LogDirFailureChannel) java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940) at kafka.log.AbstractIndex.<init>(AbstractIndex.scala:126) at kafka.log.OffsetIndex.<init>(OffsetIndex.scala:53) at kafka.log.LogSegment$.open(LogSegment.scala:632) at kafka.log.Log$$anonfun$kafka$log$Log$$loadSegmentFiles$3.apply(Log.scala:467) at kafka.log.Log$$anonfun$kafka$log$Log$$loadSegmentFiles$3.apply(Log.scala:454) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) at kafka.log.Log.kafka$log$Log$$loadSegmentFiles(Log.scala:454) at kafka.log.Log$$anonfun$loadSegments$1.apply$mcV$sp(Log.scala:565) at kafka.log.Log$$anonfun$loadSegments$1.apply(Log.scala:559) at kafka.log.Log$$anonfun$loadSegments$1.apply(Log.scala:559) at kafka.log.Log.retryOnOffsetOverflow(Log.scala:2024) at kafka.log.Log.loadSegments(Log.scala:559) at kafka.log.Log.<init>(Log.scala:292) at kafka.log.Log$.apply(Log.scala:2157) at kafka.log.LogManager.kafka$log$LogManager$$loadLog(LogManager.scala:265) at kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$11$$anonfun$apply$15$$anonfun$apply$2.apply$mcV$sp(LogManager.scala:345) at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:63) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937) ... 25 more Any hint to solve this problem ? Thanks in advance! --Senthil