Hello, We are encountering such issues on AWS EBS based system, nothing in AWS logs.
Is it memory or IO issue, as it states in one line in dir /srv/kafka/disk1 due to IOException (kafka.server.LogDirFailureChannel) java.io.IOException: Map failed and in second Caused by: java.lang.OutOfMemoryError: Map failed > [2022-02-07 21:35:43,046] WARN [ReplicaManager broker=77] Unable to > describe replica dirs for /srv/kafka/disk1 (kafka.server.ReplicaManager) > org.apache.kafka.common.errors.KafkaStorageException: Log directory > /srv/kafka/disk1 is offline [2022-02-07 21:34:54,355] ERROR Failed to clean up log for > invent_points_of_interest-39 in dir /srv/kafka/disk1 due to IOException > (kafka.server.LogDirFailureChannel) java.io.IOException: Map failed at > sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:938) at kafka.log. > AbstractIndex.<init>(AbstractIndex.scala:124) at kafka.log.OffsetIndex.< > init>(OffsetIndex.scala:54) at kafka.log.LazyIndex$.$anonfun$forOffset$1( > LazyIndex.scala:106) at kafka.log.LazyIndex.$anonfun$get$1(LazyIndex > .scala:63) at kafka.log.LazyIndex.get(LazyIndex.scala:60) at kafka.log. > LogSegment.offsetIndex(LogSegment.scala:64) at kafka.log.Cleaner > .buildOffsetMapForSegment(LogCleaner.scala:932) at kafka.log.Cleaner. > $anonfun$buildOffsetMap$5(LogCleaner.scala:908) at kafka.log.Cleaner. > $anonfun$buildOffsetMap$5$adapted(LogCleaner.scala:904) at > scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563) at > scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561) at > scala.collection.AbstractIterable.foreach(Iterable.scala:919) at > scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:889) at > kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:904) at kafka.log. > Cleaner.doClean(LogCleaner.scala:523) at kafka.log.Cleaner.clean( > LogCleaner.scala:511) at kafka.log.LogCleaner$CleanerThread.cleanLog( > LogCleaner.scala:380) at kafka.log.LogCleaner$CleanerThread > .cleanFilthiestLog(LogCleaner.scala:352) at kafka.log.LogCleaner > $CleanerThread.tryCleanFilthiestLog(LogCleaner.scala:332) at kafka.log. > LogCleaner$CleanerThread.doWork(LogCleaner.scala:321) at kafka.utils. > ShutdownableThread.run(ShutdownableThread.scala:96) Caused by: java.lang. > OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native > Method) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:935) ... 21 > more [2022-02-07 21:34:54,356] WARN [ReplicaManager broker=77] Stopping serving replicas in dir /srv/kafka/disk1 (kafka.server.ReplicaManager) -02-07 21:34: 54,588] WARN [ReplicaManager broker=77] While recording the replica LEO, the partition multibooks-250 hasn't been created. (kafka.server. ReplicaManager) [2022-02-07 21:34:54,696] WARN Stopping serving logs in dir /srv/kafka/disk1 (kafka.log.LogManager) >
