So, I did a lot of dial turning and heap tuning (came across this nice writeup about JVM tuning http://blog.mikiobraun.de/2010/08/cassandra-gc-tuning.html) still no luck with 1.2.9. I gave up and upgraded to 1.2.12 and since then things are much much better. I don't run into the heap issue that I used to with 1.2.9.
1.2.12 also looks more stable and seems to work for us. -sandeep On Mon, Nov 25, 2013 at 11:10 AM, Christopher J. Bottaro < cjbott...@academicworks.com> wrote: > Yes, we saw this same behavior. > > A couple of months ago, we moved a large portion of our data out of > Postgres and into Cassandra. The initial migration was done in a > "distributed" manner: we had 600 (or 800, can't remember) processes > reading from Postgres and writing to Cassandra in tight loops. This caused > the exact behavior you described. We also did a read before a write. > > After we got through the initial data migration, our normal workload is > *much* less writes (and reads for that matter) such that our cluster can > easily handle it, so we didn't investigate further. > > -- C > > > On Sat, Nov 23, 2013 at 10:55 PM, srmore <comom...@gmail.com> wrote: > >> Hello, >> We moved to cassandra 1.2.9 from 1.0.11 to take advantage of the off-heap >> bloom filters and other improvements. >> >> We see a lot of messages dropped under high load conditions. We noticed >> that when we do heavy read AND write simultaneously (we read first and >> check whether the key exists if not we write it) Cassandra heap increases >> dramatically and then gossip marks the node down (as a result of high load >> on the node). >> >> >> Under heavy 'reads only' we don't see this behavior. Has anyone seen >> this behavior ? any suggestions. >> >> Thanks ! >> >> >> >
