Yes, they are virtual machines, but we are using KVM. Is there any solutions for this issue or we should use physical machines?
On Mon, Apr 28, 2014 at 10:38 AM, Prem Yadav <ipremya...@gmail.com> wrote: > Are the virtual machines? The last time I had this issues was because of > VMWare "ballooning". > If not, what versions of Cassandra and Java are you using? > > > On Mon, Apr 28, 2014 at 6:30 PM, Gary Zhao <garyz...@gmail.com> wrote: > >> BTW, the CPU usage on this node is pretty high, but data size is pretty >> small. >> >> PID USERNAME THR PRI NICE SIZE RES SHR STATE TIME CPU COMMAND >> 28674 cassandr 89 25 0 9451M 8970M 525M sleep 32.1H 329% java >> >> UN 8.92 GB 256 35.4% c2d9d02e-bdb3-47cb-af1b-eabc2eeb503b rack1 >> UN 5.32 GB 256 33.1% 874a269f-96bc-4db6-9d15-7cac6771a2ea rack1 >> DN 339.94 MB 256 31.5% fba5b071-aee1-451f-ae42-49c3bbe70bbe rack1 >> >> >> On Mon, Apr 28, 2014 at 10:25 AM, Gary Zhao <garyz...@gmail.com> wrote: >> >>> Hello >>> >>> I have a three nodes cluster. I noticed one node was always down. >>> Restarting Cassandra fixes it but it will go down again after a couple of >>> days. I'm pretty new to Cassandra so I'm wondering how I should >>> troubleshoot it. Logs is as below. >>> >>> INFO [StorageServiceShutdownHook] 2014-04-28 13:21:05,091 >>> ThriftServer.java (line 141) Stop listening to thrift clients >>> ERROR [GossipStage:4] 2014-04-28 13:11:59,877 CassandraDaemon.java (line >>> 196) Exception in thread Thread[GossipStage:4,5,main] >>> java.lang.OutOfMemoryError: Java heap space >>> ERROR [ACCEPT-/10.20.132.44] 2014-04-28 13:06:10,261 >>> CassandraDaemon.java (line 196) Exception in thread Thread[ACCEPT-/ >>> 10.20.132.44,5,main] >>> java.lang.OutOfMemoryError: Java heap space >>> ERROR [GossipStage:3] 2014-04-28 12:54:02,116 CassandraDaemon.java (line >>> 196) Exception in thread Thread[GossipStage:3,5,main] >>> java.lang.OutOfMemoryError: Java heap space >>> ERROR [GossipStage:2] 2014-04-28 12:52:27,644 CassandraDaemon.java (line >>> 196) Exception in thread Thread[GossipStage:2,5,main] >>> java.lang.OutOfMemoryError: Java heap space >>> ERROR [Thread-222] 2014-04-28 12:50:18,689 CassandraDaemon.java (line >>> 196) Exception in thread Thread[Thread-222,5,main] >>> java.lang.OutOfMemoryError: Java heap space >>> ERROR [GossipTasks:1] 2014-04-28 12:47:12,879 CassandraDaemon.java (line >>> 196) Exception in thread Thread[GossipTasks:1,5,main] >>> java.lang.OutOfMemoryError: Java heap space >>> ERROR [GossipTasks:1] 2014-04-28 13:24:59,113 CassandraDaemon.java (line >>> 196) Exception in thread Thread[GossipTasks:1,5,main] >>> java.lang.IllegalThreadStateException >>> at java.lang.Thread.start(Thread.java:704) >>> at >>> org.apache.cassandra.service.CassandraDaemon$2.uncaughtException(CassandraDaemon.java:202) >>> at >>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.handleOrLog(DebuggableThreadPoolExecutor.java:220) >>> at >>> org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:79) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) >>> at >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:744) >>> INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Server.java >>> (line 181) Stop listening for CQL clients >>> INFO [StorageServiceShutdownHook] 2014-04-28 13:25:35,105 Gossiper.java >>> (line 1251) Announcing shutdown >>> INFO [StorageServiceShutdownHook] 2014-04-28 13:26:12,524 >>> MessagingService.java (line 667) Waiting for messaging service to quiesce >>> >>> Thanks >>> Gary >>> >> >> >
