Hi Daniel - I don't have any ideas at this point. Has this scenario happened again?
-- Luke Bakken Engineer [email protected] On Wed, Jan 25, 2017 at 2:11 PM, Daniel Miller <[email protected]> wrote: > Thanks for the quick response, Luke. > > There is nothing unusual about the keys. The format is a name + UUID + some > other random URL-encoded charaters, like most other keys in our cluster. > > There are no errors near the time of the incident in any of the logs (the > last [error] is from over a month before). I see lots of messages like this > in console.log: > > /var/log/riak/console.log > 2017-01-20 15:38:10.184 [info] > <0.22902.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 2 keys during > active anti-entropy exchange of > {776422744832042175295707567380525354192214163456,3} between > {776422744832042175295707567380525354192214163456,'[email protected]'} > and > {822094670998632891489572718402909198556462055424,'[email protected]'} > 2017-01-20 15:40:39.640 [info] > <0.21789.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 1 keys during > active anti-entropy exchange of > {936274486415109681974235595958868809467081785344,3} between > {959110449498405040071168171470060731649205731328,'[email protected]'} > and > {981946412581700398168100746981252653831329677312,'[email protected]'} > 2017-01-20 15:46:40.918 [info] > <0.13986.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 2 keys during > active anti-entropy exchange of > {662242929415565384811044689824565743281594433536,3} between > {685078892498860742907977265335757665463718379520,'[email protected]'} > and > {707914855582156101004909840846949587645842325504,'[email protected]'} > 2017-01-20 15:48:25.597 [info] > <0.29943.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 2 keys during > active anti-entropy exchange of > {776422744832042175295707567380525354192214163456,3} between > {776422744832042175295707567380525354192214163456,'[email protected]'} > and > {799258707915337533392640142891717276374338109440,'[email protected]'} > > Thanks! > Daniel > > > > On Wed, Jan 25, 2017 at 9:45 AM, Luke Bakken <[email protected]> wrote: >> >> Hi Daniel - >> >> This is a strange scenario. I recommend looking at all of the log >> files for "[error]" or other entries at about the same time as these >> PUTs or 404 responses. >> >> Is there anything unusual about the key being used? >> -- >> Luke Bakken >> Engineer >> [email protected] >> >> >> On Wed, Jan 25, 2017 at 6:40 AM, Daniel Miller <[email protected]> wrote: >> > I have a 9-node Riak CS cluster that has been working flawlessly for >> > about 3 >> > months. The cluster configuration, including backend and bucket >> > parameters >> > such as N-value are using default settings. I'm using the S3 API to >> > communicate with the cluster. >> > >> > Within the past week I had an issue where two objects were PUT resulting >> > in >> > a 200 (success) response, but all subsequent GET requests for those two >> > keys >> > return status of 404 (not found). Other than the fact that they are now >> > missing, there was nothing out of the ordinary with these particular to >> > PUTs. Maybe I'm missing something, but this seems like a scenario that >> > should never happen. All information included here about PUTs and GETs >> > comes >> > from reviewing the CS access logs. Both objects were PUT on the same >> > node, >> > however GET requests returning 404 have been observed on all nodes. >> > There is >> > plenty of other traffic on the cluster involving GETs and PUTs that are >> > not >> > failing. I'm unsure of how to troubleshoot further to find out what may >> > have >> > happened to those objects and why they are now missing. What is the best >> > approach to figure out why an object that was successfully PUT seems to >> > be >> > missing? >> > >> > Thanks! >> > Daniel Miller >> > >> > _______________________________________________ >> > riak-users mailing list >> > [email protected] >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > > > _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
