Do you think it's the tombstones that take up the disk space? Shouldn't the tombstones be moved along with the data?
On Mon, May 31, 2010 at 3:29 PM, Maxim Kramarenko <maxi...@trackstudio.com>wrote: > Hello! > > You likely need wait for GCGraceSeconds seconds or modify this param. > > http://spyced.blogspot.com/2010/02/distributed-deletes-in-cassandra.html > === > Thus, a delete operation can't just wipe out all traces of the data being > removed immediately: if we did, and a replica did not receive the delete > operation, when it becomes available again it will treat the replicas that > did receive the delete as having missed a write update, and repair them! So, > instead of wiping out data on delete, Cassandra replaces it with a special > value called a tombstone. The tombstone can then be propagated to replicas > that missed the initial remove request. > ... > Here, we defined a constant, GCGraceSeconds, and had each node track > tombstone age locally. Once it has aged past the constant, it can be GC'd. > === > > > > On 31.05.2010 16:23, Ran Tavory wrote: > >> I hope I understand nodetool cleanup correctly - it should clean up all >> data that does not (currently) belong to this node. If so, I think it >> might not be working correctly. >> >> Look at nodes 192.168.252.124 and 192.168.252.99 below >> >> 192.168.252.99Up 279.35 MB >> 3544607988759775661076818827414252202 |<--| >> 192.168.252.124Up 167.23 MB >> 56713727820156410577229101238628035242 | ^ >> 192.168.252.125Up 82.91 MB >> 85070591730234615865843651857942052863 v | >> 192.168.254.57Up 366.6 MB >> 113427455640312821154458202477256070485 | ^ >> 192.168.254.58Up 88.44 MB >> 141784319550391026443072753096570088106 v | >> 192.168.254.59Up 88.45 MB >> 170141183460469231731687303715884105727 |-->| >> >> I wanted 124 to take all the load from 99. So I issued a move command. >> >> $ nodetool -h cass99 -p 9004 move 56713727820156410577229101238628035243 >> >> This command tells 99 to take the space b/w >> (56713727820156410577229101238628035242, >> 56713727820156410577229101238628035243] >> which is basically just one item in the token space, almost nothing... I >> wanted it to be very slim (just playing around). >> >> So, next I get this: >> >> 192.168.252.124Up 803.33 MB >> 56713727820156410577229101238628035242 |<--| >> 192.168.252.99Up 352.85 MB >> 56713727820156410577229101238628035243 | ^ >> 192.168.252.125Up 134.24 MB >> 85070591730234615865843651857942052863 v | >> 192.168.254.57Up 676.41 MB >> 113427455640312821154458202477256070485 | ^ >> 192.168.254.58Up 99.74 MB >> 141784319550391026443072753096570088106 v | >> 192.168.254.59Up 99.94 MB >> 170141183460469231731687303715884105727 |-->| >> >> The tokens are correct, but it seems that 99 still has a lot of data. >> Why? OK, that might be b/c it didn't delete its moved data. >> So next I issued a nodetool cleanup, which should have taken care of >> that. Only that it didn't, the node 99 still has 352 MB of data. Why? >> So, you know what, I waited for 1h. Still no good, data wasn't cleaned up. >> I restarted the server. Still, data wasn't cleaned up... I issued a >> cleanup again... still no good... what's up with this node? >> >> >> > -- > Best regards, > Maxim mailto:maxi...@trackstudio.com > > LinkedIn Profile: http://www.linkedin.com/in/maximkr > Google Talk/Jabber: maxi...@gmail.com > ICQ number: 307863079 > Skype Chat: maxim.kramarenko > Yahoo! Messenger: maxim_kramarenko >
