Re: Handling disk-full scenarios

2010-06-08 Thread Jonathan Ellis
Sounds like you ran into https://issues.apache.org/jira/browse/CASSANDRA-1169. The only workaround until that is fixed is to re-run repair. On Tue, Jun 8, 2010 at 7:17 AM, Ian Soboroff wrote: > And three days later, AE stages are still running full-bore.  So I conclude > this is not a very good

Re: Handling disk-full scenarios

2010-06-08 Thread Ian Soboroff
And three days later, AE stages are still running full-bore. So I conclude this is not a very good approach. I wonder what will happen when I lose a disk (which is essentially the same as what I did -- rm the data directory). What happens if I lose a disk while the AE stages are running? Since

Re: Handling disk-full scenarios

2010-06-04 Thread Ian Soboroff
Story continued, in hopes this experience is useful to someone... I shut down the node, removed the huge file, restarted the node, and told everybody to repair. Two days later, AE stages are still running. Ian On Thu, Jun 3, 2010 at 2:21 AM, Jonathan Ellis wrote: > this is why JBOD configurat

Re: Handling disk-full scenarios

2010-06-02 Thread Jonathan Ellis
this is why JBOD configuration is contraindicated for cassandra. http://wiki.apache.org/cassandra/CassandraHardware On Tue, Jun 1, 2010 at 1:08 PM, Ian Soboroff wrote: > My nodes have 5 disks and are using them separately as data disks.  The > usage on the disks is not uniform, and one is nearly

Capacity planning and Re: Handling disk-full scenarios

2010-06-02 Thread Ian Soboroff
Reading some more (someone break in when I lose my clue ;-) Reading the streams page in the wiki about anticompaction, I think the best approach to take when a node gets its disks overfull, is to set the compaction thresholds to 0 on all nodes, decommission the overfull node, wait for stuff to get

Re: Handling disk-full scenarios

2010-06-02 Thread Ian Soboroff
Ok, answered part of this myself. You can stop a node, move files around on the data disks, as long as they stay in the right keyspace directories, and all is fine. Now, I have a single Data.db file which is 900GB and is compacted. The drive its on is only 1.5TB, so it can't anticompact at all.