[
https://issues.apache.org/jira/browse/CASSANDRA-12005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16864029#comment-16864029
]
Chakravarthi Manepalli commented on CASSANDRA-12005:
----------------------------------------------------
I have faced a similar issue, heap overflows after 15-30 min time from the node
restart.
Setup environment - single node running in localhost with 64GB RAM and 8GB HEAP
memory
nodetool info:
ID : 35c11cc6-f881-4685-b211-654c4a21bf59
Gossip active : true
Thrift active : false
Native Transport active: true
Load : 4.57 MiB
Generation No : 1560507197
Uptime (seconds) : 250
Heap Memory (MB) : 301.67 / 8112.00
Off Heap Memory (MB) : 0.01
Data Center : datacenter1
Rack : rack1
Exceptions : 0
Key Cache : entries 128, size 16.5 KiB, capacity 100 MiB, 375 hits, 510
requests, 0.735 recent hit rate, 14400 save period in seconds
Row Cache : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN
recent hit rate, 0 save period in seconds
Counter Cache : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 requests,
NaN recent hit rate, 7200 save period in seconds
Chunk Cache : entries 19, size 1.19 MiB, capacity 480 MiB, 495 misses, 1126
requests, 0.560 recent hit rate, 48.843 microseconds miss latency
Percent Repaired : 100.0%
Token : (invoke with -T/--tokens to see all 256 tokens)
The node shows running in service status but unable to take cqlsh or access
nodetool.
ERROR [Native-Transport-Requests-1] 2019-06-14 14:41:35,747
JVMStabilityInspector.java:74 - OutOfMemory error letting the JVM handle the
error:
java.lang.OutOfMemoryError: Java heap space
at org.apache.cassandra.db.Clustering.make(Clustering.java:81)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.CBuilder$ArrayBackedBuilder.buildWith(CBuilder.java:210)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.MultiCBuilder$MultiClusteringBuilder.build(MultiCBuilder.java:416)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.cql3.restrictions.ClusteringColumnRestrictions.valuesAsClustering(ClusteringColumnRestrictions.java:113)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.cql3.restrictions.StatementRestrictions.getClusteringColumns(StatementRestrictions.java:770)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.cql3.statements.SelectStatement.getRequestedRows(SelectStatement.java:765)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.cql3.statements.SelectStatement.makeClusteringIndexFilter(SelectStatement.java:628)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.cql3.statements.SelectStatement.getSliceCommands(SelectStatement.java:519)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.cql3.statements.SelectStatement.getQuery(SelectStatement.java:306)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:282)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:224)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:255)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:240)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
[apache-cassandra-3.11.3.jar:3.11.3]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
[netty-all-4.0.44.Final.jar:4.0.44.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
[netty-all-4.0.44.Final.jar:4.0.44.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
[netty-all-4.0.44.Final.jar:4.0.44.Final]
at
io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348)
[netty-all-4.0.44.Final.jar:4.0.44.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_212]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
[apache-cassandra-3.11.3.jar:3.11.3]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
WARN [epollEventLoopGroup-2-5] 2019-06-14 14:41:35,748 Slf4JLogger.java:151 -
An exceptionCaught() event was fired, and it reached at the tail of the
pipeline. It usually means the last handler in the pipeline did not handle the
exception.
java.lang.OutOfMemoryError: Java heap space
at org.apache.cassandra.db.Clustering.make(Clustering.java:81)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.CBuilder$ArrayBackedBuilder.buildWith(CBuilder.java:210)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.MultiCBuilder$MultiClusteringBuilder.build(MultiCBuilder.java:416)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.cql3.restrictions.ClusteringColumnRestrictions.valuesAsClustering(ClusteringColumnRestrictions.java:113)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.cql3.restrictions.StatementRestrictions.getClusteringColumns(StatementRestrictions.java:770)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.cql3.statements.SelectStatement.getRequestedRows(SelectStatement.java:765)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.cql3.statements.SelectStatement.makeClusteringIndexFilter(SelectStatement.java:628)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.cql3.statements.SelectStatement.getSliceCommands(SelectStatement.java:519)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.cql3.statements.SelectStatement.getQuery(SelectStatement.java:306)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:282)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:117)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:224)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:255)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:240)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.transport.messages.QueryMessage.execute(QueryMessage.java:116)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:517)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.transport.Message$Dispatcher.channelRead0(Message.java:410)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357)
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at
io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:35)
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at
io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:348)
~[netty-all-4.0.44.Final.jar:4.0.44.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_212]
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
~[apache-cassandra-3.11.3.jar:3.11.3]
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
~[apache-cassandra-3.11.3.jar:3.11.3]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_212]
> Out of memory error in MessagingService
> ---------------------------------------
>
> Key: CASSANDRA-12005
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12005
> Project: Cassandra
> Issue Type: Bug
> Components: Legacy/Streaming and Messaging
> Environment: Ubuntu 14.04.4 LTS 3.13.0-79-generic #123-Ubuntu SMP
> x86_64
> Cassandra ReleaseVersion: 2.2.5
> java version "1.8.0_31"
> Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)
> Reporter: Chris Powell
> Priority: Normal
>
> I am periodically loosing nodes due to the below OOM error. The nodes restart
> perfectly fine. It appears intermittent and randomly affects nodes. There are
> no other warnings or errors in the log files.
> I am using the {{GCG1}} with the following options:
> {quote}
> JVM_OPTS="$JVM_OPTS -XX:+UseG1GC"
> JVM_OPTS="$JVM_OPTS -XX:SurvivorRatio=8"
> JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=1"
> JVM_OPTS="$JVM_OPTS -XX:+UseTLAB"
> JVM_OPTS="$JVM_OPTS -XX:+AlwaysPreTouch"
> JVM_OPTS="$JVM_OPTS -XX:-UseBiasedLocking"
> JVM_OPTS="$JVM_OPTS -XX:+ResizeTLAB"
> JVM_OPTS="$JVM_OPTS -XX:MaxGCPauseMillis=500"
> JVM_OPTS="$JVM_OPTS -XX:G1RSetUpdatingPauseTimePercent=10"
> JVM_OPTS="$JVM_OPTS -XX:InitiatingHeapOccupancyPercent=25"
> {quote}
> ERROR [MessagingService-Incoming-/10.184.11.109] 2016-06-14 13:00:20,237
> CassandraDaemon.java:185 - Exception in thread
> Thread[MessagingService-Incoming-/10.184.11.109,5,main]
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:361)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:322)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:126)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:109)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:101)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:109)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:200)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:177)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:91)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> ERROR [MessagingService-Incoming-/10.184.11.109] 2016-06-14 13:00:20,239
> JVMStabilityInspector.java:117 - JVM state determined to be unstable.
> Exiting forcefully due to:
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:361)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.utils.ByteBufferUtil.readWithLength(ByteBufferUtil.java:322)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.ColumnSerializer.deserializeColumnBody(ColumnSerializer.java:126)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:109)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.ColumnSerializer.deserialize(ColumnSerializer.java:101)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:109)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserializeOneCf(Mutation.java:322)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:302)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:330)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:272)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.net.MessageIn.read(MessageIn.java:99)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessage(IncomingTcpConnection.java:200)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.net.IncomingTcpConnection.receiveMessages(IncomingTcpConnection.java:177)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:91)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
