> It hits an OOM. To add a little more colour because I stepped through this with someone the other day.
When memtables are not removed from the memtable flush queue (because they have not been written) the queue will fill up. When this happens the flush process will block trying to fill the queue, and will hold the internal switch lock used to syncronise around the commit log. This will prevent write threads from progressing. All the while writes will continue to be delivered to the node and the Mutation thread pool queue will fill. All of this results in extreme memory pressure, the JVM will spend a lot of time running GC to try and free some space. While all the GC is going on chances are the other nodes will see the failing node ad flapping as it fails to keep up with gossip. None of this will work and eventually the JVM will raise an OOM error that is normally trapped and results in the node trying to shut down. During the shutdown process it will try to disable the rpc / native transports and gossip. It's a simple thing to test and a useful example to walk through (by looking at the logs) with an Ops team if they are just starting out. Cheers ----------------- Aaron Morton Freelance Cassandra Consultant New Zealand @aaronmorton http://www.thelastpickle.com On 10/07/2013, at 5:36 AM, Robert Coli <rc...@eventbrite.com> wrote: > On Mon, Jul 8, 2013 at 5:58 PM, Faraaz Sareshwala <fsareshw...@quantcast.com> > wrote: > What does cassandra do when it is at its data capacity (disk drives and > memtable > is full) and writes continue to pour in? My intuition says that cassandra > won't > be able to handle the new writes (they will either get silently dropped or > cassandra will hit an OOM -- does anyone know which one?). The sstables on > disk > won't magically disappear so cassandra won't be able to service the write > requests. > > It hits an OOM. > > =Rob >
