> And the data could be more evenly balanced, obviously. However the nodes > fails to startup because due of lacking disk space (instead of starting up > and denies further writes it appears to try to process the [6.6G!] commit > logs). So, I cannot perform any actions on it no more like re-balancing the > ring or reading old data from it and rotating it somewhere else. So, what to > do now?
So even given deletion of obsolete sstables on start-up, it goes out of disk just from the commit log replay of only 6 gig? Sounds like you're very, very full. Some potential options may be: * Replace the node completely with a new one with sufficient disk / another token location (but carefully, by adding the new node first). * Restarting the nodes with sstables gone and perform a repair to re-populate it with a minimum amount of overhead (by not writing new data to it). * Removing some of the sstables with the aim of allowing commit log replay to complete, followed by repair. But those two assume RF > 1 and that you are willing to lose writes (potentially) according to the consistency level you have used for writes. It strikes me that dealing with out-of-disk conditions is probably a good topic of an in-depth wiki page for what to do in various cases, depending on usage. The above three suggested options may or may not be a good idea depending on circumstances. In particular I don't remember if a repair or bootstrap still, in 0.7, consumes disk-space on the nodes *from* which data is being sent (anyone?). In any case: Monitoring disk-space is very very important. > BTW what precisely does the Owns column mean? The percentage of the token space owned by the node. -- / Peter Schuller
