Also, your query may have hit some "not_found" objects - you'll probably want to filter those out in the reduce phase using values = Riak.filterNotFound(values); That could be the source of some errors.
Sean Cribbs <s...@basho.com> Developer Advocate Basho Technologies, Inc. http://basho.com/ On Sep 20, 2010, at 12:07 PM, francisco treacy wrote: > 2010/9/19 Kevin Smith <ksm...@basho.com>: >> 0) I know the MapReduce machinery isn't as user-friendly as it could be. We >> are making plans now on ways to dramatically improve things on this front in >> the nearish future. I hope to have more details to share soon. We've >> committed improvements to the error reporting logic on tip so if you have >> the ability to upgrade to tip you should get improved error messages. > > Thanks, Kevin, that's great news. Looking forward to hearing more. > >> 1) Reduce functions are currently called in two different ways. The first >> way is to invoke the reduce function with a list of items produced from the >> map phase. Internally the MapReduce machinery batches up data as it streams >> in from the map phase and sends the batches to the reduce phase. >> >> The second way to invoke a reduce function at the end of the reduce with the >> list of data produced via earlier reduce invocations. This is also referred >> to as "re-reduce". Riak reduce functions always re-reduce. Re-reduction is a >> common source of problems as it tends to catch people off guard. However, >> your function looks like it should work. You could try wrapping the function >> body in a try/catch and using ejsLog to log the exception to a file. That >> should shed some light on what exactly if failing. > > Yea, that was my understanding but indeed, it may have caught me off guard :) > > Unfortunately I couldn't reproduce the problem with our sum function. > It works just fine now. What I am totally certain about is that I saw > it failing (logs are the proof) and by simply './riak restart'-ing the > problem went away. Hope not to experience something similar in > production! > >> 2) How are you reloading new Javascript functions? 'riak-admin js_reload' >> should cause all the Javascript VMs to restart and start from a clean slate. >> If that's not working correctly I'd be very interested in knowing that so I >> can fix it before our next release. > > Good point, I am *not* using named functions so I imagine `riak-admin > js_reload` does not apply to my case (or does it - I never triggered > it)? Every job I submit is comprised of anonymous functions (except > Riak built-ins, that is) -- I am developing plenty of ad-hoc queries > to be run once in a while on our production data. > > The rest of the weirdness could have come from my side (possibly badly > coded reduce functions). Better error reporting, as well as guidelines > and gotchas when building reduces and advanced examples, would > definitely help. What I can say right now is that I'll be closely > following this matter, and come up asap with any issues I may > encounter. > > Francisco > > >> >> On Sep 19, 2010, at 1:09 PM, francisco treacy wrote: >> >>> So, I've had a weekend plagued of reduce function errors and general >>> weirdness, probably due to my ignorance on the subject. >>> >>> Functions seem to be cached, which makes total sense. But it happened >>> to me several times sending a *different* function yet Riak would >>> completely ignore it and keep on running an older version (i started >>> noticing when using ejsLog). Is there a way of disabling caching? Hope >>> i'm missing something big here cause that's super annoying. >>> >>> (I've also noticed strange behaviour when executing map/reduce jobs in >>> parallel and getting results mixed up... *sometimes* (possibly race >>> conditions) -- but first I want to make sure it's not the client. And >>> no, there's no global leakage.). >>> >>> Then... what's wrong with this reduce function? It used to run fine, >>> but now it doesn't anymore. Oh, wait - yes it does again now that I >>> restarted Riak: >>> >>> function(values) { >>> return [ >>> values.reduce(function(acc, value) { >>> return acc + (value.fleet || value || 0); >>> }, 0) >>> ]; >>> } >>> >>> What can explain this behaviour? ( * error logs attached) >>> >>> >>> I'm well aware of >>> http://github.com/basho/riak_kv/blob/master/priv/mapred_builtins.js >>> and the few blog posts on the matter. Are there any docs in-between >>> the basic examples and the source code? (i will delve into it once I >>> get enough Erlang-fu, promise :) >>> >>> I mean, when and how reduces (and re-reduces) are invoked and what you >>> should expect will be fed to your function, how are they cached and >>> what should programmers know about, what can/can't be done, more >>> advanced examples (how do "commutative, associative, and idempotent" >>> functions reflect in practical terms), etc. >>> >>> I'd love to help with whatever I can (might be good material for a blog >>> post). >>> >>> Thanks, >>> Francisco >>> >>> (*) These are the logs: >>> >>> =SUPERVISOR REPORT==== 19-Sep-2010::18:10:32 === >>> Supervisor: {local,luke_phase_sup} >>> Context: child_terminated >>> Reason: {error,failed_reduce} >>> Offender: [{pid,<0.5424.22>}, >>> {name,undefined}, >>> {mfa, >>> {luke_phase,start_link, >>> [riak_kv_reduce_phase,1, >>> [accumulate,converge], >>> undefined,<0.5421.22>,66000, >>> [{javascript, >>> {reduce, >>> {jsanon, >>> <<"function (values) {\n return [\n >>> values.reduce(function(acc, value) {\n return >>> acc + (value.fleet || value || 0);\n }, 0)\n ];\n >>> }">>}, >>> none,true}}]]}}, >>> {restart_type,temporary}, >>> {shutdown,brutal_kill}, >>> {child_type,worker}] >>> >>> >>> =SUPERVISOR REPORT==== 19-Sep-2010::18:10:32 === >>> Supervisor: {local,luke_phase_sup} >>> Context: child_terminated >>> Reason: {error,failed_reduce} >>> Offender: [{pid,<0.5425.22>}, >>> {name,undefined}, >>> {mfa, >>> {luke_phase,start_link, >>> [riak_kv_reduce_phase,1, >>> [accumulate,converge], >>> undefined,<0.5421.22>,66000, >>> [{javascript, >>> {reduce, >>> {jsanon, >>> <<"function (values) {\n return [\n >>> values.reduce(function(acc, value) {\n return >>> acc + (value.fleet || value || 0);\n }, 0)\n ];\n >>> }">>}, >>> none,true}}]]}}, >>> {restart_type,temporary}, >>> {shutdown,brutal_kill}, >>> {child_type,worker}] >>> >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
