Hi,
Without having access to the mapreduce functions you are running, I would
assume that a mapreduce job both writing data to disk as well as deleting the
written record from Riak might be quite slow. This is not really a use case
mapreduce was designed for, and when a mapreduce job crashes or times out it is
difficult to know how far along the processing of different records it got.
I would therefore recommend considering running this type of archiving and
delete job as an external batch process instead as it will give you better
control over the execution and avoid timeout problems.
Best regards,
Christian
On 8 Apr 2013, at 00:49, Matt Black <matt.bl...@jbadigital.com> wrote:
> Dear list,
>
> I'm currently getting a timeout during a single phase of a multi-phase map
> reduce query. Is there anything I can do to assist this in running?
>
> It's purpose is to backup and remove objects from Riak, so it will run
> periodically during quiet times moving old data out of Riak into file storage.
>
> Traceback (most recent call last):
> File "./tools/rolling_backup.py", line 185, in <module>
> main()
> File "./tools/rolling_backup.py", line 181, in main
> args.func(**kwargs)
> File "/srv/backup/tools/mapreduce.py", line 295, in do_map_reduce
> raise e
> Exception:
> {"phase":2,"error":"timeout","input":"[<<\"cart-products\">>,<<\"cd67d7f6e2688bc2089e6fa79506ac05-2\">>,{struct,[{<<\"uid\">>,<<\"cd67d7f6e2688bc2089e6fa79506ac05\">>},{<<\"cart\">>,{struct,[{<<\"expired_ts\">>,<<\"2013-03-05T19:12:23.906228\">>},{<<\"last_updated\">>,<<\"2013-03-05T19:12:23.906242\">>},{<<\"tags\">>,{struct,[{<<\"type\">>,<<\"AB\">>}]}},{<<\"completed\">>,false},{<<\"created\">>,<<\"2013-03-04T02:10:18.638413\">>},{<<\"products\">>,[{struct,[{<<\"cost\">>,0},{<<\"bundleName\">>,<<\"Product\">>},...]},...]},...]}},...]}]","type":"exit","stack":"[{riak_kv_w_reduce,'-js_runner/1-fun-0-',3,[{file,\"src/riak_kv_w_reduce.erl\"},{line,283}]},{riak_kv_w_reduce,reduce,3,[{file,\"src/riak_kv_w_reduce.erl\"},{line,206}]},{riak_kv_w_reduce,maybe_reduce,2,[{file,\"src/riak_kv_w_reduce.erl\"},{line,157}]},{riak_pipe_vnode_worker,process_input,3,[{file,\"src/riak_pipe_vnode_worker.erl\"},{line,444}]},{riak_pipe_vnode_worker,wait_for_input,2,[{file,\"src/riak_pipe_vnode_worker.erl\"},{line,376}]},{gen_fsm,handle_msg,7,[{file,\"gen_fsm.erl\"},{line,494}]},{proc_lib,...}]"}
>
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
