In our tests we are adding 3000 keys into 3-node Riak db right after nodes have joined. For each key one node reads it and modifies it and another node does the same but also deletes the key when it sees other change (key is no longer needed). After all keys are processed our test framework checks if all keys have been processed and deleted (bucket listing)
Now, since deletion is not instant in Riak I believe here is 3 second delay, we are doing that check in a loop listing and counting number of keys in a bucket to check the progress. So far we've seen number of keys going down, eventually reaching zero. Sometimes the number of keys remained greater than zero, but after attempt to get a value we got null (Java client) which indicates key has been deleted. It's not perfect but it worked so far. When we changed to ring size 512 things got bit hairy. Our test fails and we have 24, 37 or 44 keys hanging around never going down to zero, but rather going up and down between those numbers. Test fails eventually. After that when I checked riak-admin transfer I could see 96 partitions to be transfer on every node. BTW Active Transfer section was showing nothing (I'm pretty sure there should be MB/s now) In the end partitions pending for transfer hit zero few minuter after test failed but that did not change anything. Listing keys in the bucket via http client showed between 24-44 keys. Manual reading via http client shows keys does not exist, and it looks like read-repair removes it, but somehow number of keys cannot reach 0 When those keys will be removed? Do I need to get every key to trigger read-repair? Any suggestions what to change to improve that behaviour? Bucket has allow_multi set to true, which I believe is necessary for that kind of usage. Riak db is in version 1.2.1 and backend LevelDB Regards Daniel
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
