Cassandra crashed in Two out of 10 nodes in my cluster within 1 day, the error
is: ERROR [CompactionExecutor:3389] 2018-07-10 11:27:58,857
CassandraDaemon.java:228 - Exception in thread
Thread[CompactionExecutor:3389,1,main] org.apache.cassandra.io.FSReadError:
java.io.IOException: Map failed at
org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
~[apache-cassandra-3.11.2.jar:3.11.2] at
org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
~[apache-cassandra-3.11.2.jar:3.11.2] at
org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
~[apache-cassandra-3.11.2.jar:3.11.2] at
org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:170)
~[apache-cassandra-3.11.2.jar:3.11.2] at
org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:73)
~[apache-cassandra-3.11.2.jar:3.11.2] at
org.apache.cassandra.io.util.MmappedRegions.<init>(MmappedRegions.java:61)
~[apache-cassandra-3.11.2.jar:3.11.2] at
org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
~[apache-cassandra-3.11.2.jar:3.11.2] at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
~[apache-cassandra-3.11.2.jar:3.11.2] at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
~[apache-cassandra-3.11.2.jar:3.11.2] at
org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179)
~[apache-cassandra-3.11.2.jar:3.11.2] at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134)
~[apache-cassandra-3.11.2.jar:3.11.2] at
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
~[apache-cassandra-3.11.2.jar:3.11.2] at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
~[apache-cassandra-3.11.2.jar:3.11.2] at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
~[apache-cassandra-3.11.2.jar:3.11.2] at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.2.jar:3.11.2] at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
~[apache-cassandra-3.11.2.jar:3.11.2] at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
~[apache-cassandra-3.11.2.jar:3.11.2] at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:275)
~[apache-cassandra-3.11.2.jar:3.11.2] at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_65] at
java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_65]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_65] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_65] at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.2.jar:3.11.2] at
java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_65] Caused by:
java.io.IOException: Map failed at
sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:939) ~[na:1.8.0_65]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
~[apache-cassandra-3.11.2.jar:3.11.2] ... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed at
sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_65] at
sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:936) ~[na:1.8.0_65]
... 24 common frames omitted Each node has 128 GB ram which 32 GB allocated as
Cassandra Heap. Sent using Zoho Mail
