Yes, it really looks like the above scenario. It started happening when I
only had no more than 20 custom buckets - is that too much? Whats the
suggested limit? Will there be other issues when using a lot of buckets or
is it just for the first gossip period?
On 1 May 2014 01:12, Evan Vigil-McClanahan <emcclana...@basho.com> wrote:
> I suspect that, given the large heap messages, you're seeing the known
> issues where when a custom bucket is created and the moving that
> metadata around the ring takes increasingly long. It isn't
> recommended to create a large number of custom buckets at the moment.
>
> What the process likely looks like is this:
>
> 1. you do the set.
> 2. bucket metadata starts being gossip around the ring.
> 3. you do the put. it succeeds, but on some nodes, the metadata
> hasn't been committed, so it's put into the default backend.
> 4. gossip finishes.
> 5. you do the get. it fails because now the bucket for this backend
> has made 2 replicas unreachable.
> 6. read repair happens, repopulating the 2 missing replicas.
> 7. you re-get and it works.
>
> On Wed, Apr 30, 2014 at 4:04 PM, Oleksiy Krivoshey <oleks...@gmail.com>
> wrote:
> > The objects are really small (50 - 500 bytes).
> >
> > It doesn't happen if the bucket was already created.
> > It also does't happen if I don't call SetBucket at all (so using default
> > backend and options).
> > And it seems it doesn't happen if I call SetBucket but don't set
> `backend`
> > property.
> >
> >
> >
> > On 1 May 2014 00:51, Evan Vigil-McClanahan <emcclana...@basho.com>
> wrote:
> >>
> >> Does this continue if the bucket hasn't been created recently? Does
> >> it matter how large the object is? Is it particularly large in this
> >> case?
> >>
> >> On Wed, Apr 30, 2014 at 3:47 PM, Oleksiy Krivoshey <oleks...@gmail.com>
> >> wrote:
> >> > Hi guys,
> >> >
> >> > can someone please suggest what can be the reason for 'get'
> immediately
> >> > following successful 'put' to fail?
> >> >
> >> > I'm running a fully connected, healthy, 5 node Riak 2.0-beta1 cluster.
> >> > Using
> >> > a multiple backend feature, so the order of operations is:
> >> >
> >> > 1. 'SetBucket' for a new bucket with a backend name. Wait for
> successful
> >> > reply
> >> > 2. 'Put' object to bucket. Wait for successful reply (return_head:
> true,
> >> > so
> >> > I get clock, vtag, etc back)
> >> > 3. 'Get' object from step 2. Returns empty response (missing object)
> >> >
> >> > If I repeat 'Get' after just few moments - I will successfully
> retrieve
> >> > the
> >> > object.
> >> >
> >> > CAP options are all defaults (n: 3, etc). Happens with both custom
> >> > bitcask
> >> > and custom leveldb backends.
> >> >
> >> > There are no errors in riak logs, but a lot of the following:
> >> >
> >> > 2014-04-30 21:41:05.656 [info]
> >> > <0.95.0>@riak_core_sysmon_handler:handle_event:92 monitor large_heap
> >> > <0.10689.74>
> >> >
> >> >
> [{initial_call,{erlang,apply,2}},{almost_current_function,{eval_bits,expr_grp,4}},{message_queue_len,0}]
> >> >
> >> >
> [{old_heap_block_size,0},{heap_block_size,79467343},{mbuf_size,0},{stack_size,394},{old_heap_size,0},{heap_size,28858704}]
> >> >
> >> > --
> >> > Oleksiy
> >> >
> >> > _______________________________________________
> >> > riak-users mailing list
> >> > riak-users@lists.basho.com
> >> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >> >
> >
> >
> >
> >
> > --
> > Oleksiy Krivoshey
>
--
Oleksiy Krivoshey
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
