Hi Luke,
Sorry for the late response and thanks for following up. I haven't seen it
happen since. At this point I'm going to wait and see if it happens again
and hopefully get more details about what might be causing it.
Daniel
On Thu, Feb 9, 2017 at 1:02 PM, Luke Bakken <lbak...@basho.com> wrote:
> Hi Daniel -
>
> I don't have any ideas at this point. Has this scenario happened again?
>
> --
> Luke Bakken
> Engineer
> lbak...@basho.com
>
>
> On Wed, Jan 25, 2017 at 2:11 PM, Daniel Miller <dmil...@dimagi.com> wrote:
> > Thanks for the quick response, Luke.
> >
> > There is nothing unusual about the keys. The format is a name + UUID +
> some
> > other random URL-encoded charaters, like most other keys in our cluster.
> >
> > There are no errors near the time of the incident in any of the logs (the
> > last [error] is from over a month before). I see lots of messages like
> this
> > in console.log:
> >
> > /var/log/riak/console.log
> > 2017-01-20 15:38:10.184 [info]
> > <0.22902.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 2 keys
> during
> > active anti-entropy exchange of
> > {776422744832042175295707567380525354192214163456,3} between
> > {776422744832042175295707567380525354192214163456,'riak-
> fa...@fake3.fake.com'}
> > and
> > {822094670998632891489572718402909198556462055424,'riak-
> fa...@fake9.fake.com'}
> > 2017-01-20 15:40:39.640 [info]
> > <0.21789.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 1 keys
> during
> > active anti-entropy exchange of
> > {936274486415109681974235595958868809467081785344,3} between
> > {959110449498405040071168171470060731649205731328,'riak-
> fa...@fake3.fake.com'}
> > and
> > {981946412581700398168100746981252653831329677312,'riak-
> fa...@fake5.fake.com'}
> > 2017-01-20 15:46:40.918 [info]
> > <0.13986.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 2 keys
> during
> > active anti-entropy exchange of
> > {662242929415565384811044689824565743281594433536,3} between
> > {685078892498860742907977265335757665463718379520,'riak-
> fa...@fake3.fake.com'}
> > and
> > {707914855582156101004909840846949587645842325504,'riak-
> fa...@fake6.fake.com'}
> > 2017-01-20 15:48:25.597 [info]
> > <0.29943.1193>@riak_kv_exchange_fsm:key_exchange:263 Repaired 2 keys
> during
> > active anti-entropy exchange of
> > {776422744832042175295707567380525354192214163456,3} between
> > {776422744832042175295707567380525354192214163456,'riak-
> fa...@fake3.fake.com'}
> > and
> > {799258707915337533392640142891717276374338109440,'riak-
> fa...@fake0.fake.com'}
> >
> > Thanks!
> > Daniel
> >
> >
> >
> > On Wed, Jan 25, 2017 at 9:45 AM, Luke Bakken <lbak...@basho.com> wrote:
> >>
> >> Hi Daniel -
> >>
> >> This is a strange scenario. I recommend looking at all of the log
> >> files for "[error]" or other entries at about the same time as these
> >> PUTs or 404 responses.
> >>
> >> Is there anything unusual about the key being used?
> >> --
> >> Luke Bakken
> >> Engineer
> >> lbak...@basho.com
> >>
> >>
> >> On Wed, Jan 25, 2017 at 6:40 AM, Daniel Miller <dmil...@dimagi.com>
> wrote:
> >> > I have a 9-node Riak CS cluster that has been working flawlessly for
> >> > about 3
> >> > months. The cluster configuration, including backend and bucket
> >> > parameters
> >> > such as N-value are using default settings. I'm using the S3 API to
> >> > communicate with the cluster.
> >> >
> >> > Within the past week I had an issue where two objects were PUT
> resulting
> >> > in
> >> > a 200 (success) response, but all subsequent GET requests for those
> two
> >> > keys
> >> > return status of 404 (not found). Other than the fact that they are
> now
> >> > missing, there was nothing out of the ordinary with these particular
> to
> >> > PUTs. Maybe I'm missing something, but this seems like a scenario that
> >> > should never happen. All information included here about PUTs and GETs
> >> > comes
> >> > from reviewing the CS access logs. Both objects were PUT on the same
> >> > node,
> >> > however GET requests returning 404 have been observed on all nodes.
> >> > There is
> >> > plenty of other traffic on the cluster involving GETs and PUTs that
> are
> >> > not
> >> > failing. I'm unsure of how to troubleshoot further to find out what
> may
> >> > have
> >> > happened to those objects and why they are now missing. What is the
> best
> >> > approach to figure out why an object that was successfully PUT seems
> to
> >> > be
> >> > missing?
> >> >
> >> > Thanks!
> >> > Daniel Miller
> >> >
> >> > _______________________________________________
> >> > riak-users mailing list
> >> > riak-users@lists.basho.com
> >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >> >
> >
> >
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
