Hi Justin,
Could you run "riak-debug" on each node of this cluster and email me
the generated archives directly? I'm wondering what has caused so much
AAE to become necessary.
Thanks!
--
Luke Bakken
CSE
lbak...@basho.com
On Thu, Jan 2, 2014 at 4:39 PM, Justin Long <justinl...@outlook.com> wrote:
> Ok here’s an update on the issue, I’m really hoping someone can chime in. I
> disabled Anti-Entropy, restarted Riak, and then tried to run a repair on
> indexes. After running the command [riak_kv_vnode:repair(P) || P <-
> Partitions]. Riak then returned the error
>
> ** exception exit: {timeout,{gen_server,call,
>
> [{riak_core_vnode_manager,'riak@192.168.3.2'},
> {repair,riak_kv,
>
> {riak_kv_vnode,22835963083295358096932575511191922182123945984},
>
> {riak_kv_vnode,repair_filter}}]}}
> in function gen_server:call/2 (gen_server.erl, line 180)
>
> We then tried to use the Basho Riak Data Migrator tool to back up the most
> recent data on the node, but it takes 2 minutes to export a simple KV object
> (and these are less than 32kb in size). Thanks for your help.
>
> FYI Using Riak 1.4.6
>
>
>
> On Jan 2, 2014, at 11:31 AM, Justin Long <justinl...@outlook.com> wrote:
>
> Hi everyone,
>
> Experiencing some weird behaviour in a cluster of 3 nodes. As of this
> morning, we noticed a massive performance lag and when checking the logs
> Active Anti-Entropy Exchange suddenly began initiating a huge series of
> transfers. Here’s what we see in the logs:
>
> 2014-01-02 19:26:34.396 [info]
> <0.1571.1800>@riak_kv_exchange_fsm:key_exchange:206 Repaired 882 keys during
> active anti-entropy exchange of
> {228359630832953580969325755111919221821239459840,3} between
> {228359630832953580969325755111919221821239459840,'riak@192.168.3.2'} and
> {251195593916248939066258330623111144003363405824,'riak@192.168.3.3’}
>
> When we check the logs on all 3 nodes we only had a few process crashes 10
> hours ago, nothing in between that time and now. It all came on suddenly.
> Any idea if this is normal and will fix itself or if we need to intervene?
>
> Thanks!
>
>
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
