2010/9/19 Kevin Smith <ksm...@basho.com>:
> 0) I know the MapReduce machinery isn't as user-friendly as it could be. We
> are making plans now on ways to dramatically improve things on this front in
> the nearish future. I hope to have more details to share soon. We've
> committed improvements to the error reporting logic on tip so if you have the
> ability to upgrade to tip you should get improved error messages.
Thanks, Kevin, that's great news. Looking forward to hearing more.
> 1) Reduce functions are currently called in two different ways. The first way
> is to invoke the reduce function with a list of items produced from the map
> phase. Internally the MapReduce machinery batches up data as it streams in
> from the map phase and sends the batches to the reduce phase.
>
> The second way to invoke a reduce function at the end of the reduce with the
> list of data produced via earlier reduce invocations. This is also referred
> to as "re-reduce". Riak reduce functions always re-reduce. Re-reduction is a
> common source of problems as it tends to catch people off guard. However,
> your function looks like it should work. You could try wrapping the function
> body in a try/catch and using ejsLog to log the exception to a file. That
> should shed some light on what exactly if failing.
Yea, that was my understanding but indeed, it may have caught me off guard :)
Unfortunately I couldn't reproduce the problem with our sum function.
It works just fine now. What I am totally certain about is that I saw
it failing (logs are the proof) and by simply './riak restart'-ing the
problem went away. Hope not to experience something similar in
production!
> 2) How are you reloading new Javascript functions? 'riak-admin js_reload'
> should cause all the Javascript VMs to restart and start from a clean slate.
> If that's not working correctly I'd be very interested in knowing that so I
> can fix it before our next release.
Good point, I am *not* using named functions so I imagine `riak-admin
js_reload` does not apply to my case (or does it - I never triggered
it)? Every job I submit is comprised of anonymous functions (except
Riak built-ins, that is) -- I am developing plenty of ad-hoc queries
to be run once in a while on our production data.
The rest of the weirdness could have come from my side (possibly badly
coded reduce functions). Better error reporting, as well as guidelines
and gotchas when building reduces and advanced examples, would
definitely help. What I can say right now is that I'll be closely
following this matter, and come up asap with any issues I may
encounter.
Francisco
>
> On Sep 19, 2010, at 1:09 PM, francisco treacy wrote:
>
>> So, I've had a weekend plagued of reduce function errors and general
>> weirdness, probably due to my ignorance on the subject.
>>
>> Functions seem to be cached, which makes total sense. But it happened
>> to me several times sending a *different* function yet Riak would
>> completely ignore it and keep on running an older version (i started
>> noticing when using ejsLog). Is there a way of disabling caching? Hope
>> i'm missing something big here cause that's super annoying.
>>
>> (I've also noticed strange behaviour when executing map/reduce jobs in
>> parallel and getting results mixed up... *sometimes* (possibly race
>> conditions) -- but first I want to make sure it's not the client. And
>> no, there's no global leakage.).
>>
>> Then... what's wrong with this reduce function? It used to run fine,
>> but now it doesn't anymore. Oh, wait - yes it does again now that I
>> restarted Riak:
>>
>> function(values) {
>> return [
>> values.reduce(function(acc, value) {
>> return acc + (value.fleet || value || 0);
>> }, 0)
>> ];
>> }
>>
>> What can explain this behaviour? ( * error logs attached)
>>
>>
>> I'm well aware of
>> http://github.com/basho/riak_kv/blob/master/priv/mapred_builtins.js
>> and the few blog posts on the matter. Are there any docs in-between
>> the basic examples and the source code? (i will delve into it once I
>> get enough Erlang-fu, promise :)
>>
>> I mean, when and how reduces (and re-reduces) are invoked and what you
>> should expect will be fed to your function, how are they cached and
>> what should programmers know about, what can/can't be done, more
>> advanced examples (how do "commutative, associative, and idempotent"
>> functions reflect in practical terms), etc.
>>
>> I'd love to help with whatever I can (might be good material for a blog
>> post).
>>
>> Thanks,
>> Francisco
>>
>> (*) These are the logs:
>>
>> =SUPERVISOR REPORT==== 19-Sep-2010::18:10:32 ===
>> Supervisor: {local,luke_phase_sup}
>> Context: child_terminated
>> Reason: {error,failed_reduce}
>> Offender: [{pid,<0.5424.22>},
>> {name,undefined},
>> {mfa,
>> {luke_phase,start_link,
>> [riak_kv_reduce_phase,1,
>> [accumulate,converge],
>> undefined,<0.5421.22>,66000,
>> [{javascript,
>> {reduce,
>> {jsanon,
>> <<"function (values) {\n return [\n
>> values.reduce(function(acc, value) {\n return
>> acc + (value.fleet || value || 0);\n }, 0)\n ];\n
>> }">>},
>> none,true}}]]}},
>> {restart_type,temporary},
>> {shutdown,brutal_kill},
>> {child_type,worker}]
>>
>>
>> =SUPERVISOR REPORT==== 19-Sep-2010::18:10:32 ===
>> Supervisor: {local,luke_phase_sup}
>> Context: child_terminated
>> Reason: {error,failed_reduce}
>> Offender: [{pid,<0.5425.22>},
>> {name,undefined},
>> {mfa,
>> {luke_phase,start_link,
>> [riak_kv_reduce_phase,1,
>> [accumulate,converge],
>> undefined,<0.5421.22>,66000,
>> [{javascript,
>> {reduce,
>> {jsanon,
>> <<"function (values) {\n return [\n
>> values.reduce(function(acc, value) {\n return
>> acc + (value.fleet || value || 0);\n }, 0)\n ];\n
>> }">>},
>> none,true}}]]}},
>> {restart_type,temporary},
>> {shutdown,brutal_kill},
>> {child_type,worker}]
>>
>> _______________________________________________
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
