Re: faillout from AWS outage - HELP

2012-07-02 Thread Deno Vichas
yes, you where correct. resetting the flag and bouncing cassandra fixed it. thx On 7/2/2012 8:59 AM, Joost van de Wijgerd wrote: Could be related to the leap second bug.. try this: Bug: https://lkml.org/lkml/2012/6/30/122 Simple fix to reset the leap second flag: date; date `date +"%m%d%H%M

Re: Node won't start (faillout from AWS outage) - HELP

2012-07-02 Thread Deno Vichas
you're right! somehow while rebooting it tried to upgrade. this broke node has 1.1.1 where my others are running 1.0.7. wtf?? - deno On 7/2/2012 10:25 AM, Brian Jeltema wrote: Just the opposite, I think. The property value exists in the yaml file but does not have a corresponding defin

Re: Node won't start (faillout from AWS outage) - HELP

2012-07-02 Thread Deno Vichas
should be the same version. i did notice some extra files in my etc dir. there's a few dpkg-new files. ubuntu@ip-10-40-207-151:~/datastax_ami$ ll /etc/cassandra/ total 100 drwxr-xr-x 2 root root 4096 2012-07-02 18:13 ./ drwxr-xr-x 99 root root 4096 2012-07-02 16:31 ../ -rw-r--r-- 1 root ro

Re: Node won't start (faillout from AWS outage) - HELP

2012-07-02 Thread Deno Vichas
no, it's there... after editing the first couple lines of the config file there spot where it complains changes. On 7/2/2012 10:20 AM, Robin Verlangen wrote: Your missing the "sliced_buffer_size_in_kb" property in your cassandra.yaml file. 2012/7/2 Deno Vichas mailto:d...@syncopated.net>>

Re: Node won't start (faillout from AWS outage) - HELP

2012-07-02 Thread Brian Jeltema
Just the opposite, I think. The property value exists in the yaml file but does not have a corresponding definition in the Config class. Typically caused by a version mismatch in my experience. On Jul 2, 2012, at 1:20 PM, Robin Verlangen wrote: > Your missing the "sliced_buffer_size_in_kb" prop

Re: Node won't start (faillout from AWS outage) - HELP

2012-07-02 Thread Robin Verlangen
Your missing the "sliced_buffer_size_in_kb" property in your cassandra.yaml file. 2012/7/2 Deno Vichas > i'm seeing the following exception in my log now. this is even after > re-creating my config file. > > > INFO [main] 2012-07-02 16:58:17,850 AbstractCassandraDaemon.java (line > 121) JVM v

Re: Node won't start (faillout from AWS outage) - HELP

2012-07-02 Thread Deno Vichas
i'm seeing the following exception in my log now. this is even after re-creating my config file. INFO [main] 2012-07-02 16:58:17,850 AbstractCassandraDaemon.java (line 121) JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_26 INFO [main] 2012-07-02 16:58:17,853 AbstractCassandraDa

Re: faillout from AWS outage - HELP

2012-07-02 Thread Deno Vichas
the node that doesn't want to start just spit out - /EC2 is experiencing some issues and has not allocated all of the resources in under 10 minutes. Aborting the clustering of this reservation. Please try again. Please visit http://datastax.com/ami for this AMI's feature set. /

Re: faillout from AWS outage - HELP

2012-07-02 Thread Joost van de Wijgerd
Could be related to the leap second bug.. try this: Bug: https://lkml.org/lkml/2012/6/30/122 Simple fix to reset the leap second flag: date; date `date +"%m%d%H%M%C%y.%S"`; date; mvg Joost On Mon, Jul 2, 2012 at 5:43 PM, Deno Vichas wrote: > all, > > my 4 node cluster seems pretty screwed up

faillout from AWS outage - HELP

2012-07-02 Thread Deno Vichas
all, my 4 node cluster seems pretty screwed up after the AWS outage. we found all our machines with their cpu stuck at 100%. so i went to restart each cassandra node one by one. i did node with token id 0 first. i came back but doesn't look like it doing anything. once i thought it was u