Hi Alexander, Excellent! Thanks for the feedback - I will see what I can find there.
Regards, Ryan On Tue, Nov 1, 2016 at 11:06 AM, Alexander Sicular <sicul...@gmail.com> wrote: > Hi Ryan, yes, you can change a number of settings. Have you had a look > at http://docs.basho.com/riak/kv/2.1.4/using/admin/riak-admin/# > transfer-limit > and http://lists.basho.com/pipermail/riak-users_lists. > basho.com/2014-July/015529.html > ? > > -Alexander > > On Tue, Nov 1, 2016 at 2:43 AM, Ryan Maclear <ry...@miranetworks.net> > wrote: > > Good Day, > > > > We have a 4 node riak cluster running inside AWS. The riak is riak-kv > 2.1.2 > > with AAE enabled on Ubuntu 14.04.4 LTS > > > > We are in the process of replacing one node with another using the > process > > described here: > > > > http://docs.basho.com/riak/kv/2.1.4/using/cluster- > operations/replacing-node/ > > > > We have successfully replaced two of the nodes so far but we are having a > > problem with the third. If we look at /var/log/riak/console.log we see > the > > start of the hinted handoff, and some time later (sometimes minutes and > > sometimes hours) we see: > > > > 2016-10-31 06:30:40.090 [error] > > <0.19834.2101>@riak_core_handoff_sender:start_fold:272 hinted transfer > of > > riak_kv_vnode from 'r...@aew54.miranetworks.net' > > 274031556999544297163190906134303066185487351808 to > > 'r...@aew75.miranetworks.net' > > 274031556999544297163190906134303066185487351808 failed because of TCP > recv > > timeout > > 2016-10-31 06:30:40.090 [error] > > <0.187.0>@riak_core_handoff_manager:handle_info:303 An outbound handoff > of > > partition riak_kv_vnode 274031556999544297163190906134303066185487351808 > was > > terminated for reason: {shutdown,timeout} > > > > So the handoff was terminated due to a tcp timeout. The handoff then > starts > > again. > > > > This has been going on for some times (about two weeks now). > > > > The current member status is as follows: > > > > riak-admin member-status > > ================================= Membership > > ================================== > > Status Ring Pending Node > > ------------------------------------------------------------ > ------------------- > > leaving 0.0% -- 'r...@aew54.miranetworks.net' > > valid 25.0% -- 'r...@aew59.miranetworks.net' > > valid 25.0% -- 'r...@aew73.miranetworks.net' > > valid 25.0% -- 'r...@aew74.miranetworks.net' > > valid 25.0% -- 'r...@aew75.miranetworks.net' > > ------------------------------------------------------------ > ------------------- > > Valid:4 / Leaving:1 / Exiting:0 / Joining:0 / Down:0 > > > > > > Here are some questions: > > > > 1. What is the default tcp timeout? > > 2. Is there any way to increase this timeout? > > 3. Is there any way to increase the rate of handoff? > > 4. Are there any other parameters we can tune to try and avoid this? > > > > The output from riak-admin transfers is as follows: > > > > 'r...@aew54.miranetworks.net' waiting to handoff 1 partitions > > > > Active Transfers: > > > > transfer type: hinted > > vnode type: riak_kv_vnode > > partition: 274031556999544297163190906134303066185487351808 > > started: 2016-11-01 05:30:47 [2.10 hr ago] > > last update: 2016-11-01 07:36:51 [3.03 s ago] > > total size: 78393086512 bytes > > objects transferred: 11440967 > > > > 1513 Objs/s > > riak@aew54.miranetworks.n =======> riak@aew75.miranetworks.n > > et et > > |====== | 15% > > 1.53 MB/s > > > > > > Thanks, > > Ryan Maclear > > > > _______________________________________________ > > riak-users mailing list > > riak-users@lists.basho.com > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
