Thanks for the response. To handle not founds and tombstones I need this in
every map function?
map_something({error, notfound}, _, _) ->
[];
map_something(RiakObj, _, _) ->
Metadata = riak_object:get_metadata(RiakObj),
case dict:is_key(<<"X-Riak-Deleted">>, Metadata) of
true ->
[]; % I have a tombstone
false ->
[ok] % I have a valid item
end.
Is there a better or built-in way to do this filtering?
--
Jeremy
On Wed, Mar 6, 2013 at 4:08 PM, John Daily <jda...@basho.com> wrote:
> I'm sorry this went unanswered, Jeremy. Thanks for the follow up.
>
> Your code needs to be able to handle notfound errors and tombstones[1]
> regardless of ownership handoff. The coverage for the 2i or listkeys input
> will be calculated up front, with the work distributed to a node where the
> key is expected to be found, but it's always possible that the node
> selected may not have the data you want due to network or system errors
> that haven't yet healed.
>
> Both the notfound and the fitting error in the logs (which is benign, and
> related to the notfound problem) should be less common under Riak 1.3,
> although ownership handoff will still exacerbate the problem.
>
> [1] https://github.com/basho/riak_kv/issues/358
>
> -John Daily
> Technical Evangelist
> jda...@basho.com
>
>
> On Mar 6, 2013, at 3:14 PM, Jeremy Raymond <jeraym...@gmail.com> wrote:
>
> So I should expect {error, notfound} inputs to map jobs while ownership
> handoff is in progress? Are the not found items actually unavailable during
> handoff or is this just not found on the old node, but will be picked up by
> the new node during the same mapreduce job?
>
> --
> Jeremy
>
>
> On Thu, Feb 28, 2013 at 11:28 AM, Jeremy Raymond <jeraym...@gmail.com>wrote:
>
>> Yesterday I added a new node to my cluster. During the time when
>> ownership handoff was happening (several hours of work) mapreduce map
>> functions were receiving {error, notfound} as inputs. My Erlang mapred
>> functions weren't designed to handle this. They hadn't encountered this
>> before during normal operation. After the ownership handoff process
>> completed the {error, notfound} inputs have stopped.
>>
>> Any explanations for the {error, notfound} inputs during ownership
>> handoff? Is this because a node is attempting to process an object now
>> moved to another node? If this is the case would the notfound object be
>> found on the other node in the same mapreduce job (i.e. still visible to
>> the overall mapred process)? Should I assume {error, notfound} inputs for
>> all mapred jobs as a valid possible input and always handle it?
>>
>> I've also accumulated about 50MB of "Pipe worker startup failed:fitting
>> was gone before startup" on each node during the ownership_transfer
>> process. These messages are benign?
>>
>> Thanks a lot.
>>
>> --
>> Jeremy
>>
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
