Francisco --
0) I know the MapReduce machinery isn't as user-friendly as it could be. We are
making plans now on ways to dramatically improve things on this front in the
nearish future. I hope to have more details to share soon. We've committed
improvements to the error reporting logic on tip so if you have the ability to
upgrade to tip you should get improved error messages.
1) Reduce functions are currently called in two different ways. The first way
is to invoke the reduce function with a list of items produced from the map
phase. Internally the MapReduce machinery batches up data as it streams in from
the map phase and sends the batches to the reduce phase.
The second way to invoke a reduce function at the end of the reduce with the
list of data produced via earlier reduce invocations. This is also referred to
as "re-reduce". Riak reduce functions always re-reduce. Re-reduction is a
common source of problems as it tends to catch people off guard. However, your
function looks like it should work. You could try wrapping the function body in
a try/catch and using ejsLog to log the exception to a file. That should shed
some light on what exactly if failing.
2) How are you reloading new Javascript functions? 'riak-admin js_reload'
should cause all the Javascript VMs to restart and start from a clean slate. If
that's not working correctly I'd be very interested in knowing that so I can
fix it before our next release.
--Kevin
On Sep 19, 2010, at 1:09 PM, francisco treacy wrote:
> So, I've had a weekend plagued of reduce function errors and general
> weirdness, probably due to my ignorance on the subject.
>
> Functions seem to be cached, which makes total sense. But it happened
> to me several times sending a *different* function yet Riak would
> completely ignore it and keep on running an older version (i started
> noticing when using ejsLog). Is there a way of disabling caching? Hope
> i'm missing something big here cause that's super annoying.
>
> (I've also noticed strange behaviour when executing map/reduce jobs in
> parallel and getting results mixed up... *sometimes* (possibly race
> conditions) -- but first I want to make sure it's not the client. And
> no, there's no global leakage.).
>
> Then... what's wrong with this reduce function? It used to run fine,
> but now it doesn't anymore. Oh, wait - yes it does again now that I
> restarted Riak:
>
> function(values) {
> return [
> values.reduce(function(acc, value) {
> return acc + (value.fleet || value || 0);
> }, 0)
> ];
> }
>
> What can explain this behaviour? ( * error logs attached)
>
>
> I'm well aware of
> http://github.com/basho/riak_kv/blob/master/priv/mapred_builtins.js
> and the few blog posts on the matter. Are there any docs in-between
> the basic examples and the source code? (i will delve into it once I
> get enough Erlang-fu, promise :)
>
> I mean, when and how reduces (and re-reduces) are invoked and what you
> should expect will be fed to your function, how are they cached and
> what should programmers know about, what can/can't be done, more
> advanced examples (how do "commutative, associative, and idempotent"
> functions reflect in practical terms), etc.
>
> I'd love to help with whatever I can (might be good material for a blog post).
>
> Thanks,
> Francisco
>
> (*) These are the logs:
>
> =SUPERVISOR REPORT==== 19-Sep-2010::18:10:32 ===
> Supervisor: {local,luke_phase_sup}
> Context: child_terminated
> Reason: {error,failed_reduce}
> Offender: [{pid,<0.5424.22>},
> {name,undefined},
> {mfa,
> {luke_phase,start_link,
> [riak_kv_reduce_phase,1,
> [accumulate,converge],
> undefined,<0.5421.22>,66000,
> [{javascript,
> {reduce,
> {jsanon,
> <<"function (values) {\n return [\n
> values.reduce(function(acc, value) {\n return
> acc + (value.fleet || value || 0);\n }, 0)\n ];\n
> }">>},
> none,true}}]]}},
> {restart_type,temporary},
> {shutdown,brutal_kill},
> {child_type,worker}]
>
>
> =SUPERVISOR REPORT==== 19-Sep-2010::18:10:32 ===
> Supervisor: {local,luke_phase_sup}
> Context: child_terminated
> Reason: {error,failed_reduce}
> Offender: [{pid,<0.5425.22>},
> {name,undefined},
> {mfa,
> {luke_phase,start_link,
> [riak_kv_reduce_phase,1,
> [accumulate,converge],
> undefined,<0.5421.22>,66000,
> [{javascript,
> {reduce,
> {jsanon,
> <<"function (values) {\n return [\n
> values.reduce(function(acc, value) {\n return
> acc + (value.fleet || value || 0);\n }, 0)\n ];\n
> }">>},
> none,true}}]]}},
> {restart_type,temporary},
> {shutdown,brutal_kill},
> {child_type,worker}]
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
