Hi,
I've written about this before, but I can't find the original thread.
I have one node on my 16 node cassandra cluster (running 8.1) that will
not stay running. All nodes were configured the same way, but this one
kept going down, so I changed these lines in my cassandra-env.sh:
JVM_OPTS="$JVM_OPTS -XX:MaxTenuringThreshold=5"
JVM_OPTS="$JVM_OPTS -XX:ParallelGCThreads=8"
JVM_OPTS="$JVM_OPTS -XX:CMSInitiatingOccupancyFraction=60"
This changed it so that it didn't crash as quickly, but it also never
used more than 5G of the 8G heap I assign to cassandra. At the very
end of its run, the heap climbs suddenly and then the node crashes with
this error log:
INFO [ScheduledTasks:1] 2011-08-29 04:56:23,313 StatusLogger.java
(line 65) AntiEntropyStage 0 0
INFO [ScheduledTasks:1] 2011-08-29 04:56:31,647 StatusLogger.java
(line 65) MigrationStage 0 0
INFO [ScheduledTasks:1] 2011-08-29 04:56:31,648 StatusLogger.java
(line 65) StreamStage 0 0
INFO [ScheduledTasks:1] 2011-08-29 04:56:31,648 StatusLogger.java
(line 65) MemtablePostFlusher 1 5
INFO [ScheduledTasks:1] 2011-08-29 04:56:31,648 StatusLogger.java
(line 65) FILEUTILS-DELETE-POOL 0 0
INFO [ScheduledTasks:1] 2011-08-29 04:56:31,648 StatusLogger.java
(line 65) FlushWriter 1 5
INFO [ScheduledTasks:1] 2011-08-29 04:56:31,649 StatusLogger.java
(line 65) MiscStage 0 0
INFO [ScheduledTasks:1] 2011-08-29 04:56:31,649 StatusLogger.java
(line 65) FlushSorter 0 0
INFO [ScheduledTasks:1] 2011-08-29 04:56:31,649 StatusLogger.java
(line 65) InternalResponseStage 0 0
INFO [ScheduledTasks:1] 2011-08-29 04:56:31,649 StatusLogger.java
(line 65) HintedHandoff 1 5
INFO [ScheduledTasks:1] 2011-08-29 04:56:31,650 StatusLogger.java
(line 69) CompactionManager n/a 14
INFO [ScheduledTasks:1] 2011-08-29 04:56:31,650 StatusLogger.java
(line 81) MessagingService n/a 0,0
INFO [ScheduledTasks:1] 2011-08-29 04:56:31,650 StatusLogger.java
(line 85) ColumnFamily Memtable ops,data Row cache
size/cap Key cache size/cap
INFO [GossipStage:1] 2011-08-29 04:56:31,647 Gossiper.java (line 606)
InetAddress /10.28.0.170 is now UP
INFO [GossipTasks:1] 2011-08-29 04:56:31,647 Gossiper.java (line 620)
InetAddress /10.28.0.162 is now dead.
INFO [GossipStage:1] 2011-08-29 04:57:16,441 Gossiper.java (line 606)
InetAddress /10.28.0.168 is now UP
INFO [GossipTasks:1] 2011-08-29 04:57:16,442 Gossiper.java (line 620)
InetAddress /10.28.0.188 is now dead.
INFO [ScheduledTasks:1] 2011-08-29 04:57:16,439 StatusLogger.java
(line 88) system.NodeIdInfo 0,0
0/0 0/0
ERROR [Thread-43] 2011-08-29 04:57:16,438 AbstractCassandraDaemon.java
(line 113) Fatal exception in thread Thread[Thread-43,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
shut down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(ThreadPoolExecutor.java:758)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
INFO [HintedHandoff:1] 2011-08-29 04:57:16,437
HintedHandOffManager.java (line 320) Started hinted handoff for endpoint
/10.28.0.178
ERROR [Thread-34] 2011-08-29 04:56:54,282 AbstractCassandraDaemon.java
(line 113) Fatal exception in thread Thread[Thread-34,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
shut down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-21] 2011-08-29 04:56:54,279 AbstractCassandraDaemon.java
(line 113) Fatal exception in thread Thread[Thread-21,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
shut down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(ThreadPoolExecutor.java:758)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-44] 2011-08-29 04:56:53,654 AbstractCassandraDaemon.java
(line 113) Fatal exception in thread Thread[Thread-44,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
shut down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-32] 2011-08-29 04:56:53,654 AbstractCassandraDaemon.java
(line 113) Fatal exception in thread Thread[Thread-32,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
shut down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(ThreadPoolExecutor.java:758)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
ERROR [Thread-28] 2011-08-29 04:56:45,107 AbstractCassandraDaemon.java
(line 113) Fatal exception in thread Thread[Thread-28,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
shut down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(ThreadPoolExecutor.java:758)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
INFO [HintedHandoff:1] 2011-08-29 04:57:31,844
HintedHandOffManager.java (line 376) Finished hinted handoff of 0 rows
to endpoint /10.28.0.178
INFO [ScheduledTasks:1] 2011-08-29 04:57:31,844 StatusLogger.java
(line 88) system.IndexInfo 0,0
0/0 1/1
INFO [ScheduledTasks:1] 2011-08-29 04:57:31,847 StatusLogger.java
(line 88) system.LocationInfo 0,0
0/0 1/1
INFO [ScheduledTasks:1] 2011-08-29 04:57:31,847 StatusLogger.java
(line 88) system.Migrations 0,0
0/0 0/0
INFO [ScheduledTasks:1] 2011-08-29 04:57:31,847 StatusLogger.java
(line 88) system.HintsColumnFamily 0,0
0/0 0/0
INFO [ScheduledTasks:1] 2011-08-29 04:57:31,847 StatusLogger.java
(line 88) system.Schema 0,0
0/0 1/1
INFO [GossipTasks:1] 2011-08-29 04:57:31,836 Gossiper.java (line 620)
InetAddress /10.28.0.180 is now dead.
ERROR [Thread-27] 2011-08-29 04:57:31,823 AbstractCassandraDaemon.java
(line 113) Fatal exception in thread Thread[Thread-27,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
shut down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(ThreadPoolExecutor.java:758)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
INFO [GossipStage:1] 2011-08-29 04:57:16,442 Gossiper.java (line 606)
InetAddress /10.28.0.166 is now UP
INFO [GossipTasks:1] 2011-08-29 04:57:31,848 Gossiper.java (line 620)
InetAddress /10.28.0.172 is now dead.
INFO [ScheduledTasks:1] 2011-08-29 04:57:31,848 StatusLogger.java
(line 88) test.analytics 0,0
0/0 0/0
INFO [ScheduledTasks:1] 2011-08-29 04:57:31,849 StatusLogger.java
(line 88) analytics.analytics 0,0
0/0 9092/9092
INFO [ScheduledTasks:1] 2011-08-29 04:57:31,849 StatusLogger.java
(line 88) creds.cred_all_hash 3415,5449198
0/0 0/0
ERROR [Thread-19] 2011-08-29 04:57:43,020 AbstractCassandraDaemon.java
(line 113) Fatal exception in thread Thread[Thread-19,5,main]
java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has
shut down
at
org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:73)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
at
java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(ThreadPoolExecutor.java:758)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
at
org.apache.cassandra.net.MessagingService.receive(MessagingService.java:444)
at
org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:117)
INFO [ScheduledTasks:1] 2011-08-29 04:57:43,556 StatusLogger.java
(line 88) creds.boxed_transactions 0,0
0/0 0/0
INFO [GossipTasks:1] 2011-08-29 04:57:43,558 Gossiper.java (line 620)
InetAddress /10.28.0.178 is now dead.
INFO [ScheduledTasks:1] 2011-08-29 04:57:43,558 StatusLogger.java
(line 88) creds.cred_users 66046,59484126
0/0 6/6
INFO [GossipTasks:1] 2011-08-29 04:57:43,558 Gossiper.java (line 620)
InetAddress /10.28.0.190 is now dead.
INFO [ScheduledTasks:1] 2011-08-29 04:57:43,559 StatusLogger.java
(line 88) creds.cred_all_link 2616,4824235
0/0 0/0
INFO [ScheduledTasks:1] 2011-08-29 04:57:43,559 StatusLogger.java
(line 88) creds.cred_all_mention 4713,5517271
0/0 0/0
INFO [ScheduledTasks:1] 2011-08-29 04:57:43,559 StatusLogger.java
(line 88) creds.user_ids
