Re: Node Dead/Up

2012-10-24 Thread Jason Wee
On Wed, Oct 24, 2012 at 2:32 PM, aaron morton wrote: > I don't see errors in the logs, but I do see > a lot of dropped mutations and reads. Any correlation? > > Yes. > The dropped messages mean the server is overloaded. > > +1 . Been there..overloaded system normally produce frequent dropped muta

Re: Node Dead/Up

2012-10-23 Thread aaron morton
> I don't see errors in the logs, but I do see > a lot of dropped mutations and reads. Any correlation? Yes. The dropped messages mean the server is overloaded. Look for log messages from the GCInspector in /var/log/cassandra/system.log and/or an overloaded IO system see http://spyced.blogspo

Re: Node Dead/Up

2012-10-23 Thread Jason Hill
thanks for the replies. I'll check the load on the node that is reported as DOWN/UP. At first glace it does not appear to be overloaded. But, I will dig in deeper, is there a specific indicator on an ubuntu server that would be useful to me? Also, I didn't make it clear, but in my original post,

Re: Node Dead/Up

2012-10-23 Thread aaron morton
> check 10.50.10.21 for what is the system load. +1 And take a look in the logs on 10.21. 10.21 is being seen as down by the other nodes. it could be: * 10.21 failing to gossip fast enough, say by being overloaded to in long ParNew GC pauses. * This node failing to process gossip fast , say b

Re: Node Dead/Up

2012-10-23 Thread Jason Wee
check 10.50.10.21 for what is the system load. On Tue, Oct 23, 2012 at 10:41 AM, Jason Hill wrote: > Hello, > > I'm on version 1.0.11. > > I'm seeing this in my system log with occasional frequency: > > INFO [GossipTasks:1] 2012-10-23 02:26:34,449 Gossiper.java (line 818) > InetAddress /10.50.10

Node Dead/Up

2012-10-22 Thread Jason Hill
Hello, I'm on version 1.0.11. I'm seeing this in my system log with occasional frequency: INFO [GossipTasks:1] 2012-10-23 02:26:34,449 Gossiper.java (line 818) InetAddress /10.50.10.21 is now dead. INFO [GossipStage:1] 2012-10-23 02:26:34,620 Gossiper.java (line 804) InetAddress /10.50.10.21 is