Hi, Luke. Thank you for your answer. I did not understand you completely about transfer-limit. How does it relate to my problem. Transfer limit - is a limit of concurrent data transfer from different nodes. Am I wright? You mean that riak can handoff one partition from several nodes concurrently?
Now I have transfer-limit 1 on all riak nodes. But I am not sure that my cluster will be converged ever. All nodes experiences low memory and are killed by OOM Killer periodically. I try to add new nodes to the cluster but due problem with OOM killer this process is very-very slow. In the official docs I've read: "Sibling explosion occurs when an object rapidly collects siblings that are not reconciled. This can lead to a variety of problems, including degraded performance, especially if many objects in a cluster suffer from siblings explosion. At the extreme, having an enormous object in a node can cause reads of that object to crash the entire node. Other issues include undue latency <http://docs.basho.com/riak/kv/2.1.4/using/performance/latency-reduction> and out-of-memory errors." I mentioned that new nodes in the cluster do not experience such problems (I mean out of RAM). Regarding to siblings maybe your are right, this is manifest object. I can recognize key name but not bucket name. But more than 100 siblings on many keys is really confused me. Each time I try to PUT some object to Riak via Riak CS S3 interface I got errors with siblings. On Fri, Jun 3, 2016 at 6:43 PM Luke Bakken <lbak...@basho.com> wrote: > Hi Vladyslav, > > If you recognize the full name of the object raising the sibling > warning, it is most likely a manifest object. Sometimes, during hinted > handoff, you can see these messages. They should resolve after handoff > completes. > > Please see the documentation for the transfer-limit command as well: > > http://docs.basho.com/riak/kv/2.1.4/using/admin/riak-admin/#transfer-limit > > -- > Luke Bakken > Engineer > lbak...@basho.com > > > On Fri, Jun 3, 2016 at 2:55 AM, Vladyslav Zakhozhai > <v.zakhoz...@smartweb.com.ua> wrote: > > Hi. > > > > I have a trouble with PUT to Riak CS cluster. During this process I > > periodically see the following message in Riak error.log: > > > > 2016-06-03 11:15:55.201 [error] > > <0.15536.142>@riak_kv_vnode:encode_and_put:2253 Put failure: too many > > siblings for object OBJECT_NAME (101) > > > > and also > > > > 2016-06-03 12:41:50.678 [error] > > <0.20448.515>@riak_api_pb_server:handle_info:331 Unrecognized message > > {7345880,{error,{too_many_siblings,101}}} > > > > Here OBJECT_NAME - is the name of object in Riak which has too many > > siblings. > > > > I definitely sure that this objects are static. Nobody deletes is, nobody > > rewrites it. I have no idea why more than 100 siblings of this object > > occurs. > > > > The following effect of this issue occurs: > > > > Great amount of keys are loaded to RAM. I almost out of RAM (Do each > sibling > > has it own key or key duplicate?). > > Nodes are slow - adding new nodes are too slow > > Presence of "too many siblings" affects ownership handoffs > > > > So I have several questions: > > > > Do hinted or ownership handoffs can affect siblings count (I mean can > > siblings be created during ownership of hinted handoffs) > > Is there any workaround of this issue. Do I need remove siblings > manually or > > it removes during merges, read repairs and so on > > > > > > My configuration: > > > > riak from basho's packages - 2.1.3-1 > > riak cs from basho's packages - 2.1.0-1 > > 24 riak/riak-cs nodes > > 32 GB RAM per node > > AAE is disabled > > > > > > I appreciate you help. > > > > _______________________________________________ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
