On July 19, 2013 02:11:10 PM Kelly McLaughlin wrote: > Matthew, > > I did some testing with your code and I was able to reproduce what you were > seeing. I would occasionally see an error similar to the following: > > Failed to fetch item 460 err Object not found > > This behavior is a result of the trade-offs of using an eventually > consistent database like Riak. It is not the case that your inserts are > failing to write or the data is being lost, but what is actually happening > is that the quick read after writing with the default request options does > not provide any guarantee that you will read your writes. So basically when > you make the read, the replicas that are responding to your request have > not seen the latest value yet and so you end up with "Not Found" as the > response. If you did another read attempt for one of those objects reported > missing, it would succeed because Riak's read-repair would have kicked in > to make sure each replica has the value. To increase the likelihood of > reading your writes you should set the optional request parameters pr and > pw to ensure that all of the primary replicas are available prior to > performing a read or write request. I altered your code to use those > options and put the updates in a gist [1] (it's my first stab at Go so my > changes may not be very idiomatic). Additionally I changed to riak driver > so that NotFoundOk was false instead of true. With these changes I was able > to run the test 50 times in a row with no errors where previously I would > see at least one error every 10 iterations or so. Hope that helps. Kelly,
Thanks for looking into this. I looked over the documentation and your suggestions, and I think I have found a race causing my data loss. I read what you wrote above, but I don't think this applies in this situation. If I stop the program after it finds a missing element, but before it deletes all the keys, I've found that any key it finds missing is also missing if I look at it using Curl. Also, I've looked at the returned headers, and both "X-Riak-Deleted: true" and a vclock are returned, which suggests to me Riak has actually deleted the data on purpose. I've included sample output at [2]. (I used a newer version of my program in a gist[1]). The updated program has an extra flag, --quit_early=false , that exits before the deletion if a problem was found during the fetch stage. Since your version did fix my issue as well, I played around with the various settings. I discovered to fix the issue, I only have to use an r value of 3 when first fetching the item to get a vclock for the insert operation. As long as that value is a 3, everything seems to work perfectly. Adjusting the other values (I didn't test pr, but I left all the cluster nodes running, so I don't think it affects anything) produced zero change (including when w/rw/dw were all set to 3, or r was set to 3 elsewhere). You can try this using the updated program[1] by passing "-r=3" to it. Using the above knowledge, I looked at whether vclocks were being returned for the insert operation when keys went missing during the fetch stage. In all cases, vclocks were missing. I also checked to see if there are any siblings being generated (which I didn't think would be created), and some were generated as well. Looking at them, they contain one sibling containing the key's deletion, and then my data (example in [2]). >From those sets of facts, I'm guessing there is some race in Riak where either the deletion or the creation gets committed first. If the creation is committed first, for some reason the deletion also deletes it. I'm guessing that it happens since the vclocks show the creation happening before the deletion. For some reason, when r=3 during the insert stage's fetch causes Riak to commit the deletion cluster wide and the commit always appears to come after. I think this is a bug, since the deletion shouldn't see the insert operation as history. Since Riak makes this mistakes, it is actually losing data. I don't think the siblings should be created created either (in the r=3 case, they are not created), but that is easily handled. If this is a bug, I'll file it on Github (is the riak_kv project the correct place in this situation?), or if this is expected behaviour I do have a workaround. A few other data points: I retried this using delete_mode = keep, and it fixed this issue as well. Using delete_mode=immediate made the issue more rare. I also find it happens almost everytime if you run it quickly, but if you give Riak sometime to synchronize state, it does reduce the chance of seeing it. [1] https://gist.github.com/MJDSys/6026486 [2] https://gist.github.com/MJDSys/420090505ebbe2e0241d -- Matthew
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
