Thanks for the detailed explanation of what you're seeing. As it turns out, your results are typical.
Usually the deletion happens automatically across all nodes, but as you've seen, there's no guarantee of that. Reading a key will trigger read repair to clean up lingering objects, and is effectively the only way to do so. Active anti-entropy in Riak 1.3.0 has most of what's necessary to automatically clean up all deleted objects, but that is not a current feature. I'll make sure a ticket is filed with that feature request. -John Daily Technical Evangelist jda...@basho.com On Mar 7, 2013, at 3:10 PM, Daniel Iwan <iwan.dan...@gmail.com> wrote: > In our tests we are adding 3000 keys into 3-node Riak db right after nodes > have joined. > For each key one node reads it and modifies it and another node does the same > but also deletes the key when it sees other change (key is no longer needed). > After all keys are processed our test framework checks if all keys have been > processed and deleted (bucket listing) > > Now, since deletion is not instant in Riak I believe here is 3 second delay, > we are doing that check in a loop listing and counting number of keys in a > bucket to check the progress. So far we've seen number of keys going down, > eventually reaching zero. Sometimes the number of keys remained greater than > zero, but after attempt to get a value we got null (Java client) which > indicates key has been deleted. > It's not perfect but it worked so far. > > When we changed to ring size 512 things got bit hairy. > Our test fails and we have 24, 37 or 44 keys hanging around never going down > to zero, but rather going up and down between those numbers. > > Test fails eventually. After that when I checked riak-admin transfer I could > see 96 partitions to be transfer on every node. BTW Active Transfer section > was showing nothing (I'm pretty sure there should be MB/s now) > In the end partitions pending for transfer hit zero few minuter after test > failed but that did not change anything. > Listing keys in the bucket via http client showed between 24-44 keys. > Manual reading via http client shows keys does not exist, and it looks like > read-repair removes it, but somehow number of keys cannot reach 0 > > When those keys will be removed? Do I need to get every key to trigger > read-repair? > Any suggestions what to change to improve that behaviour? > > Bucket has allow_multi set to true, which I believe is necessary for that > kind of usage. > Riak db is in version 1.2.1 and backend LevelDB > > Regards > Daniel > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
