Hello!
You likely need wait for GCGraceSeconds seconds or modify this param.
http://spyced.blogspot.com/2010/02/distributed-deletes-in-cassandra.html
===
Thus, a delete operation can't just wipe out all traces of the data
being removed immediately: if we did, and a replica did not receive the
delete operation, when it becomes available again it will treat the
replicas that did receive the delete as having missed a write update,
and repair them! So, instead of wiping out data on delete, Cassandra
replaces it with a special value called a tombstone. The tombstone can
then be propagated to replicas that missed the initial remove request.
...
Here, we defined a constant, GCGraceSeconds, and had each node track
tombstone age locally. Once it has aged past the constant, it can be GC'd.
===
On 31.05.2010 16:23, Ran Tavory wrote:
I hope I understand nodetool cleanup correctly - it should clean up all
data that does not (currently) belong to this node. If so, I think it
might not be working correctly.
Look at nodes 192.168.252.124 and 192.168.252.99 below
192.168.252.99Up 279.35 MB
3544607988759775661076818827414252202 |<--|
192.168.252.124Up 167.23 MB
56713727820156410577229101238628035242 | ^
192.168.252.125Up 82.91 MB
85070591730234615865843651857942052863 v |
192.168.254.57Up 366.6 MB
113427455640312821154458202477256070485 | ^
192.168.254.58Up 88.44 MB
141784319550391026443072753096570088106 v |
192.168.254.59Up 88.45 MB
170141183460469231731687303715884105727 |-->|
I wanted 124 to take all the load from 99. So I issued a move command.
$ nodetool -h cass99 -p 9004 move 56713727820156410577229101238628035243
This command tells 99 to take the space b/w
(56713727820156410577229101238628035242, 56713727820156410577229101238628035243]
which is basically just one item in the token space, almost nothing... I
wanted it to be very slim (just playing around).
So, next I get this:
192.168.252.124Up 803.33 MB
56713727820156410577229101238628035242 |<--|
192.168.252.99Up 352.85 MB
56713727820156410577229101238628035243 | ^
192.168.252.125Up 134.24 MB
85070591730234615865843651857942052863 v |
192.168.254.57Up 676.41 MB
113427455640312821154458202477256070485 | ^
192.168.254.58Up 99.74 MB
141784319550391026443072753096570088106 v |
192.168.254.59Up 99.94 MB
170141183460469231731687303715884105727 |-->|
The tokens are correct, but it seems that 99 still has a lot of data.
Why? OK, that might be b/c it didn't delete its moved data.
So next I issued a nodetool cleanup, which should have taken care of
that. Only that it didn't, the node 99 still has 352 MB of data. Why?
So, you know what, I waited for 1h. Still no good, data wasn't cleaned up.
I restarted the server. Still, data wasn't cleaned up... I issued a
cleanup again... still no good... what's up with this node?
--
Best regards,
Maxim mailto:maxi...@trackstudio.com
LinkedIn Profile: http://www.linkedin.com/in/maximkr
Google Talk/Jabber: maxi...@gmail.com
ICQ number: 307863079
Skype Chat: maxim.kramarenko
Yahoo! Messenger: maxim_kramarenko
