On Thu, 27 Aug 2015 at 18:32:31 +0000 Girish Shankarraman <
gshankarra...@vmware.com> wrote:
>
> Hello,
>
> Currently using Riak 2.1.1. I have 7 nodes in my cluster.
> I am looking for some information on how these worker_limits are
> configured .
>
>
> I have 50 linux(client) hosts trying to run the map reduce jobs on Riak. I
> am getting the error below where some of the hosts complain about the
> worker_limit being reached.
>
> Looking for some insights on whether I can tune the system to avoid this
> error? Couldn't find too much documentation around the worker_limit.
>
> {"phase":0,"error":"[worker_limit_reached]","input":"{<<\"provisionentry\">>,<<\"R89Okhz49SDje0y0qvcnkK7xLH0\">>}","type":"result","stack":"[]"}
> with query MapReduce(path='/mapred', reply_headers={'content-length':
> '144', 'access-control-allow-headers': 'Content-Type', 'server':
> 'MochiWeb/1.1 WebMachine/1.10.8 (that head fake, tho)', 'connection':
> 'close', 'date': 'Thu, 27 Aug 2015 00:32:22 GMT',
> 'access-control-allow-origin': '*', 'access-control-allow-methods': 'POST,
> GET, OPTIONS', 'content-type': 'application/json'}, verb='POST',
> headers={'Content-Type': 'application/json'}
>
> Thanks,
>
> - Girish Shankarraman
>
Hi, Girish. The `worker_limit` is a riak_pipe tunable. It's there to keep
you from trying to do "too much" at once. Your MR jobs spin up riak_pipe
workers, and the number of them per vnode is what this tunable limits.
It has been a while since I last recorded this, but to calculate the number
of workers per vnode that an MR job spins up, use roughly this formula:
(using key-listing/2i/search ? 1 : 0) +
(number of map stages) +
(number of map stages with pre-reduce enabled) +
(number of reduce stages)
Multiply that sum by the number of MR jobs you want to have in-flight
concurrently. That will give you an overestimate of the worker_limit to use
(because not all stages use a worker on every vnode). The default is a
somewhat arbitrary 50.
Note that you'll need to re-examine the other limits in the system as well
(like the number of JS VMs if your MR jobs use Javascript).
Back in the day, you'd set this by adding a line like so to riak's
app.config:
{riak_pipe, [{worker_limit, 50}]}
That might be something like this in riak.conf, but I'm not sure:
riak_pipe.worker_limit = 50
-Bryan
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
