Yes, we saw this same behavior. A couple of months ago, we moved a large portion of our data out of Postgres and into Cassandra. The initial migration was done in a "distributed" manner: we had 600 (or 800, can't remember) processes reading from Postgres and writing to Cassandra in tight loops. This caused the exact behavior you described. We also did a read before a write.
After we got through the initial data migration, our normal workload is *much* less writes (and reads for that matter) such that our cluster can easily handle it, so we didn't investigate further. -- C On Sat, Nov 23, 2013 at 10:55 PM, srmore <comom...@gmail.com> wrote: > Hello, > We moved to cassandra 1.2.9 from 1.0.11 to take advantage of the off-heap > bloom filters and other improvements. > > We see a lot of messages dropped under high load conditions. We noticed > that when we do heavy read AND write simultaneously (we read first and > check whether the key exists if not we write it) Cassandra heap increases > dramatically and then gossip marks the node down (as a result of high load > on the node). > > > Under heavy 'reads only' we don't see this behavior. Has anyone seen this > behavior ? any suggestions. > > Thanks ! > > >
