Re: Help with Cassandra Row Caches

2011-11-12 Thread Peter Schuller
> We set row cache too high, 1 or so and now all our 6 nodes fail > with OOM. I believe that high row cache causes OOMs. If you get into a fallen-and-cant-get-up scenario like this, I suggest a rolling restart of nodes, and on each node: (1) shut it down (2) removed saved caches (3) start it

Re: Help with Cassandra Row Caches

2011-11-12 Thread Oleg Tsvinev
The problem was fixed by _decreasing_ heap size and applying changes on the least loaded node. Tyler, where saved caches are stored? /var/lib/cassandra/saved_caches? Would it help relieving heap load? On Fri, Nov 11, 2011 at 6:29 PM, Tyler Hobbs wrote: > You could also delete any saved caches, wh

Re: Help with Cassandra Row Caches

2011-11-11 Thread Tyler Hobbs
You could also delete any saved caches, which might give you some more breathing room. On Fri, Nov 11, 2011 at 7:53 PM, Mohit Anchlia wrote: > Can you temporarily increase the size of Heap and try? > > On Fri, Nov 11, 2011 at 5:21 PM, Oleg Tsvinev > wrote: > > Hi everybody, > > > > We set row ca

Re: Help with Cassandra Row Caches

2011-11-11 Thread Mohit Anchlia
Can you temporarily increase the size of Heap and try? On Fri, Nov 11, 2011 at 5:21 PM, Oleg Tsvinev wrote: > Hi everybody, > > We set row cache too high, 1 or so and now all our 6 nodes fail > with OOM. I believe that high row cache causes OOMs. > > Now, we trying to change row cache sizes u

Help with Cassandra Row Caches

2011-11-11 Thread Oleg Tsvinev
Hi everybody, We set row cache too high, 1 or so and now all our 6 nodes fail with OOM. I believe that high row cache causes OOMs. Now, we trying to change row cache sizes using cassandra-cli update column family command but Cassandra nodes keep dying. Any help is appreciated, it's a product