Thanks for the quick response, Luke.
There is nothing unusual about the keys. The format is a name + UUID + some
other random URL-encoded charaters, like most other keys in our cluster.
There are no errors near the time of the incident in any of the logs (the
last [error] is from over a month before). I see lots of messages like this
in console.log:
/var/log/riak/console.log
2017-01-20 15:38:10.184 [info]
<0.22902.1193>@riak_kv_exchange_fsm:key_exchange:263
Repaired 2 keys during active anti-entropy exchange of {
776422744832042175295707567380525354192214163456,3} between {
776422744832042175295707567380525354192214163456,'riak-fa...@fake3.fake.com'}
and {822094670998632891489572718402909198556462055424,'riak-
fa...@fake9.fake.com'}
2017-01-20 15:40:39.640 [info]
<0.21789.1193>@riak_kv_exchange_fsm:key_exchange:263
Repaired 1 keys during active anti-entropy exchange of {
936274486415109681974235595958868809467081785344,3} between {
959110449498405040071168171470060731649205731328,'riak-fa...@fake3.fake.com'}
and {981946412581700398168100746981252653831329677312,'riak-
fa...@fake5.fake.com'}
2017-01-20 15:46:40.918 [info]
<0.13986.1193>@riak_kv_exchange_fsm:key_exchange:263
Repaired 2 keys during active anti-entropy exchange of {
662242929415565384811044689824565743281594433536,3} between {
685078892498860742907977265335757665463718379520,'riak-fa...@fake3.fake.com'}
and {707914855582156101004909840846949587645842325504,'riak-
fa...@fake6.fake.com'}
2017-01-20 15:48:25.597 [info]
<0.29943.1193>@riak_kv_exchange_fsm:key_exchange:263
Repaired 2 keys during active anti-entropy exchange of {
776422744832042175295707567380525354192214163456,3} between {
776422744832042175295707567380525354192214163456,'riak-fa...@fake3.fake.com'}
and {799258707915337533392640142891717276374338109440,'riak-
fa...@fake0.fake.com'}
Thanks!
Daniel
On Wed, Jan 25, 2017 at 9:45 AM, Luke Bakken <lbak...@basho.com> wrote:
> Hi Daniel -
>
> This is a strange scenario. I recommend looking at all of the log
> files for "[error]" or other entries at about the same time as these
> PUTs or 404 responses.
>
> Is there anything unusual about the key being used?
> --
> Luke Bakken
> Engineer
> lbak...@basho.com
>
>
> On Wed, Jan 25, 2017 at 6:40 AM, Daniel Miller <dmil...@dimagi.com> wrote:
> > I have a 9-node Riak CS cluster that has been working flawlessly for
> about 3
> > months. The cluster configuration, including backend and bucket
> parameters
> > such as N-value are using default settings. I'm using the S3 API to
> > communicate with the cluster.
> >
> > Within the past week I had an issue where two objects were PUT resulting
> in
> > a 200 (success) response, but all subsequent GET requests for those two
> keys
> > return status of 404 (not found). Other than the fact that they are now
> > missing, there was nothing out of the ordinary with these particular to
> > PUTs. Maybe I'm missing something, but this seems like a scenario that
> > should never happen. All information included here about PUTs and GETs
> comes
> > from reviewing the CS access logs. Both objects were PUT on the same
> node,
> > however GET requests returning 404 have been observed on all nodes.
> There is
> > plenty of other traffic on the cluster involving GETs and PUTs that are
> not
> > failing. I'm unsure of how to troubleshoot further to find out what may
> have
> > happened to those objects and why they are now missing. What is the best
> > approach to figure out why an object that was successfully PUT seems to
> be
> > missing?
> >
> > Thanks!
> > Daniel Miller
> >
> > _______________________________________________
> > riak-users mailing list
> > riak-users@lists.basho.com
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
