Handoff stalled on 1.0.2 riak cluster
Hi. We had an issue where one of the riak servers died (had to be force removed from cluster). After we did that things got really bad and most data was unreachable for hours. I added a new node to replace the old one at one point as well - that never got any data and even now about a day later it hasn't gotten any data. What seems to be the issue now is that there are a few nodes are waiting on handoff of 1 partition. When I look at ring_status I see this: Attempting to restart script through sudo -u riak == Claimant === Claimant: 'riak@r-001.x.x.x Status: up Ring Ready: true == Ownership Handoff == Owner: riak@r-004.x.x.x Next Owner: riak@r-003.x.x.x Index: 930565495644285842450002452081070828921550798848 Waiting on: [] Complete: [riak_kv_vnode,riak_pipe_vnode,riak_search_vnode] --- == Unreachable Nodes == All nodes are up and reachable Ok, so it looks like the problem described in the Release Notes for 1.0.2 here https://github.com/basho/riak/blob/1.0.2-release/RELEASE-NOTES.org. Unfortunately I've run that code (through riak attach) with no result. It's been in this state for 12 hours now I think. What can we do to fix our cluster? I upgraded to 1.0.3 hoping it would fix our problems but that didn't help. I cannot upgrade to 1.1.x because we mainly use Luwak for large object support and that's discontinued in 1.1.x as far as I know. Thanks for your help, John ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Handoff stalled on 1.0.2 riak cluster
Hi John,
Assuming things aren't back to normal... A few things:
Attach to any running node and run this:
rpc:multicall([node() | nodes()], riak_core_vnode_manager, force_handoffs, []).
This will attempt to force handoff. If this restarts handoff, you've
got new issue that we'll need to track down. Please report back if
this gets handoffs running again .
Another possible fix:
Take a look at https://github.com/basho/riak_core/pull/153
This was fixed on 1.1, but it might be what's hitting you (though,
admittedly, your issue does seem like a perfect match for the issue
from the 1.0.2 release notes).
If this is what's ailing you, there's a work-around here:
https://github.com/basho/riak_core/pull/153#issuecomment-4527706
If neither of these work, let us know and we'll take a deeper look.
Specifically:
a) any log files you could send along would be helpful
b) the output of the following diagnostic:
f(Members).
Members = riak_core_ring:all_members(element(2,
riak_core_ring_manager:get_raw_ring())).
[{N, rpc:call(N, riak_core_handoff_manager, status, [])} || N <- Members].
Thanks, John.
Mark
On Sun, Jun 3, 2012 at 5:06 AM, John Axel Eriksson wrote:
> Hi.
>
> We had an issue where one of the riak servers died (had to be force
> removed from cluster). After we did that things got really bad and most
> data was unreachable for hours. I added a new node to replace the old one
> at one point as well - that never got any data and even now about a day
> later it hasn't gotten any data.
> What seems to be the issue now is that there are a few nodes are waiting
> on handoff of 1 partition. When I look at ring_status I see this:
>
> Attempting to restart script through sudo -u riak
> == Claimant
> ===
> Claimant: 'riak@r-001.x.x.x
> Status: up
> Ring Ready: true
>
> == Ownership Handoff
> ==
> Owner: riak@r-004.x.x.x
> Next Owner: riak@r-003.x.x.x
>
> Index: 930565495644285842450002452081070828921550798848
> Waiting on: []
> Complete: [riak_kv_vnode,riak_pipe_vnode,riak_search_vnode]
>
>
> ---
>
> == Unreachable Nodes
> ==
> All nodes are up and reachable
>
>
> Ok, so it looks like the problem described in the Release Notes for 1.0.2
> here https://github.com/basho/riak/blob/1.0.2-release/RELEASE-NOTES.org.
> Unfortunately I've run that code (through riak attach) with no result.
>
> It's been in this state for 12 hours now I think. What can we do to fix
> our cluster?
>
> I upgraded to 1.0.3 hoping it would fix our problems but that didn't help.
> I cannot upgrade to 1.1.x because we mainly use Luwak for large object
> support
> and that's discontinued in 1.1.x as far as I know.
>
> Thanks for your help,
> John
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: links and object deletion
Hi José, On Fri, Jun 1, 2012 at 2:34 PM, José Ghislain Quenum < jose.que...@googlemail.com> wrote: > Hi all, > Suppose I have two objects O1 and O2, with O1 having a link (say tag is > watches) to O2. What is the proper way to delete O2? Should I just delete > O2 and then leave the link inconsistent until I find out and fix it? Or > should I always check whether that object has links to it and clean them up > before deleting the object. Please advise > There's no real issue if you leave a link dangling to an object that you've deleted. The two things to be aware of: * there will be some extra metadata on the object * if you walk the link to the non-existant object it'll return a 404. Mark > José > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
HTTP-based client for Riak database, written in Dart
Hi, I'm working on a Dart-based project that uses Riak database as a backend, and I'm releasing the related riak-dart client codes: http://code.google.com/p/riak-dart/ The project covers the basic use-cases of Riak: * fetch object * store object * with easy conditional updates * secondary index support + helpers * delete object (with easy conditional delete) * query secondary index Contributions are welcome, especially in the following areas: * testing, logging, documentation, examples * query secondary index with large result set (streaming processing) * conditional fetch object (etag, not-modified-since support) * Riak Protobuf API support * Riak Map-Reduce API support Regards, Istvan ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
