HTTP link walking will get you back the data in the way that you'd expect. It's a two-step process using PBC. MR link phases will give you a list of [bucket, key, tag] that you can then use to pull back the records from Riak.
--- Jeremiah Peschka Founder, Brent Ozar PLF, LLC On Thursday, June 23, 2011 at 8:52 AM, Andrew Berman wrote: > Ah, ok, that makes sense. One more question, when I use the HTTP link > walking, I do get the data back as expected, so is there a way to > replicate this in a Map-Reduce job or using the Erlang PBC (which I > forgot to mention is what I'm using and the reason I'm not using the > HTTP link walking method)? > > --Andrew > > On Thu, Jun 23, 2011 at 8:39 AM, Mathias Meyer <math...@basho.com > (mailto:math...@basho.com)> wrote: > > Andrew, > > > > the data looks like JSON, but it's not valid JSON. Have a look at the list > > that's in the data section (which is your BERT encoded data), the first > > character in that list is 131, which is not a valid UTF-8 character, and > > JSON only allows valid UTF-8 characters. With a binary-encoded format, > > there's always a chance for a control character like that to blow up the > > JSON generated before and after the MapReduce code is executed. With JSON, > > content agnosticism only goes as far as the set of legal characters allows. > > > > On a side note, if the data were a valid representation of a string, you > > would see it as a string in the log file as well, not just as a list of > > numbers. > > > > Mathias Meyer > > Developer Advocate, Basho Technologies > > > > > > On Donnerstag, 23. Juni 2011 at 17:31, Andrew Berman wrote: > > > > > But isn't the value itself JSON? Meaning this part: > > > > > > {struct, > > > [{<<"bucket">>,<<"user">>}, > > > {<<"key">>,<<"LikiWUPJSFuxtrhCYpsPfg">>}, > > > {<<"vclock">>, > > > > > > <<"a85hYGBgzGDKBVIsLKaZdzOYEhnzWBmes6Yd58sCAA==">>}, > > > {<<"values">>, > > > [{struct, > > > [{<<"metadata">>, > > > {struct, > > > [{<<"X-Riak-VTag">>, > > > <<"1KnL9Dlma9Yg4eMhRuhwtx">>}, > > > {<<"X-Riak-Last-Modified">>, > > > <<"Fri, 10 Jun 2011 03:05:11 GMT">>}]}}, > > > {<<"data">>, > > > > > > <<131,108,0,0,0,18,104,2,100,0,6,114,...>>}]}]} > > > > > > So the only thing that is not JSON is the data itself, but when I get > > > the value, shouldn't I be getting the all the info above which is JSON > > > encoded? > > > > > > Thank you all for your help, > > > > > > Andrew > > > > > > On Thu, Jun 23, 2011 at 8:17 AM, Sean Cribbs <s...@basho.com > > > (mailto:s...@basho.com)> wrote: > > > > The object has to be JSON-encoded to be marshalled into the Javascript > > > > VM, > > > > and also on the way out if the Accept header indicates > > > > application/json. So > > > > you have two places where it needs to be encodable into JSON. > > > > On Thu, Jun 23, 2011 at 11:14 AM, Andrew Berman <rexx...@gmail.com > > > > (mailto:rexx...@gmail.com)> wrote: > > > > > > > > > > Mathias, > > > > > > > > > > I thought Riak was content agnostic when it came to the data being > > > > > stored? The map phase is not running Riak.mapValuesJson, so why is > > > > > the data itself going through the JSON parser? The JSON value > > > > > returned by v with all the info is valid and I see the struct atom in > > > > > there so mochijson2 can parse it properly, but I'm not clear why > > > > > mochijson2 would be coughing at the data part. > > > > > > > > > > --Andrew > > > > > > > > > > On Thu, Jun 23, 2011 at 5:32 AM, Mathias Meyer <math...@basho.com > > > > > (mailto:math...@basho.com)> wrote: > > > > > > Andrew, > > > > > > > > > > > > you're indeed hitting a JSON encoding problem here. BERT is binary > > > > > > data, > > > > > > and won't make the JSON parser happy when trying to deserialize it, > > > > > > before > > > > > > handing it into the map phase. You have two options here, and none > > > > > > of them > > > > > > will involve JavaScript as the MapReduce language. > > > > > > > > > > > > 1.) Use the Protobuff API, use Erlang functions to return the value > > > > > > or > > > > > > object (e.g. riak_mapreduce:map_object_value or > > > > > > riak_kv_mapreduce:map_identity), and then run MapReduce queries > > > > > > with the > > > > > > content type 'application/x-erlang-binary'. However, you're > > > > > > constrained by > > > > > > client libraries here, e.g. Ruby and Python don't support this > > > > > > content type > > > > > > for MapReduce on the Protobuffs interface yet, so you'd either > > > > > > implement > > > > > > something custom, or resort to a client that does, > > > > > > riak-erlang-client comes > > > > > > to mind, though it was proven to be possible using the Java client > > > > > > too, > > > > > > thanks to Russell. See [1] and [2] > > > > > > > > > > > > 2.) Convert the result from BERT into a JSON-parseable structure > > > > > > inside > > > > > > an Erlang map function, before it's returned to the client. > > > > > > > > > > > > The second approach certainly is less restrictive in terms of API > > > > > > usage, > > > > > > but certainly involves some overhead inside of the MapReduce > > > > > > request itself, > > > > > > but is less prone to encoding/decoding issues with JSON. > > > > > > > > > > > > Mathias Meyer > > > > > > Developer Advocate, Basho Technologies > > > > > > > > > > > > [1] > > > > > > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-June/004447.html > > > > > > [2] > > > > > > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-June/004485.html > > > > > > > > > > > > On Donnerstag, 23. Juni 2011 at 07:59, Andrew Berman wrote: > > > > > > > > > > > > > Hey Ryan, > > > > > > > > > > > > > > Here is the error from the sasl log. It looks like some sort of > > > > > > > encoding error. Any thoughts on how to fix this? I am storing the > > > > > > > data as BERT encoded binary and I set the content-type as > > > > > > > application/octet-stream. > > > > > > > > > > > > > > Thanks for your help! > > > > > > > > > > > > > > Andrew > > > > > > > > > > > > > > ERROR REPORT==== 9-Jun-2011::21:37:05 === > > > > > > > ** Generic server <0.5996.21> terminating > > > > > > > ** Last message in was {batch_dispatch, > > > > > > > {map, > > > > > > > {jsanon,<<"function(value) {return [value];}">>}, > > > > > > > [{struct, > > > > > > > [{<<"bucket">>,<<"user">>}, > > > > > > > {<<"key">>,<<"LikiWUPJSFuxtrhCYpsPfg">>}, > > > > > > > {<<"vclock">>, > > > > > > > > > > > > > > <<"a85hYGBgzGDKBVIsLKaZdzOYEhnzWBmes6Yd58sCAA==">>}, > > > > > > > {<<"values">>, > > > > > > > [{struct, > > > > > > > [{<<"metadata">>, > > > > > > > {struct, > > > > > > > [{<<"X-Riak-VTag">>, > > > > > > > <<"1KnL9Dlma9Yg4eMhRuhwtx">>}, > > > > > > > {<<"X-Riak-Last-Modified">>, > > > > > > > <<"Fri, 10 Jun 2011 03:05:11 GMT">>}]}}, > > > > > > > {<<"data">>, > > > > > > > > > > > > > > <<131,108,0,0,0,18,104,2,100,0,6,114,...>>}]}]}]}, > > > > > > > <<"user">>,none]}} > > > > > > > ** When Server state == > > > > > > > {state,<0.143.0>,riak_kv_js_map,#Port<0.92614>,true} > > > > > > > ** Reason for termination == > > > > > > > ** {function_clause,[{js_driver,eval_js, > > > > > > > [#Port<0.92614>,{error,bad_encoding},5000]}, > > > > > > > {riak_kv_js_vm,invoke_js,2}, > > > > > > > {riak_kv_js_vm,define_invoke_anon_js,3}, > > > > > > > {riak_kv_js_vm,handle_call,3}, > > > > > > > {gen_server,handle_msg,5}, > > > > > > > {proc_lib,init_p_do_apply,3}]} > > > > > > > > > > > > > > =CRASH REPORT==== 9-Jun-2011::21:37:05 === > > > > > > > crasher: > > > > > > > initial call: riak_kv_js_vm:init/1 > > > > > > > pid: <0.5996.21> > > > > > > > registered_name: [] > > > > > > > exception exit: > > > > > > > > > > > > > > {function_clause,[{js_driver,eval_js,[#Port<0.92614>,{error,bad_encoding},5000]},{riak_kv_js_vm,invoke_js,2},{riak_kv_js_vm,define_invoke_anon_js,3},{riak_kv_js_vm,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]} > > > > > > > in function gen_server:terminate/6 > > > > > > > in call from proc_lib:init_p_do_apply/3 > > > > > > > ancestors: [riak_kv_js_sup,riak_kv_sup,<0.128.0>] > > > > > > > messages: [] > > > > > > > links: [<0.142.0>,<0.6009.21>] > > > > > > > dictionary: [] > > > > > > > trap_exit: false > > > > > > > status: running > > > > > > > heap_size: 4181 > > > > > > > stack_size: 24 > > > > > > > reductions: 2586 > > > > > > > neighbours: > > > > > > > neighbour: > > > > > > > [{pid,<0.6009.21>},{registered_name,[]},{initial_call,{riak_kv_mapper,init,[Argument__1]}},{current_function,{gen,do_call,4}},{ancestors,[riak_kv_mapper_sup,riak_kv_sup,<0.128.0>]},{messages,[]},{links,[<0.5996.21>,<12337.6227.21>,<0.162.0>]},{dictionary,[]},{trap_exit,false},{status,waiting},{heap_size,987},{stack_size,53},{reductions,1043}] > > > > > > > =SUPERVISOR REPORT==== 9-Jun-2011::21:37:05 === > > > > > > > Supervisor: {local,riak_kv_js_sup} > > > > > > > Context: child_terminated > > > > > > > Reason: > > > > > > > > > > > > > > {function_clause,[{js_driver,eval_js,[#Port<0.92614>,{error,bad_encoding},5000]},{riak_kv_js_vm,invoke_js,2},{riak_kv_js_vm,define_invoke_anon_js,3},{riak_kv_js_vm,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]} > > > > > > > Offender: > > > > > > > > > > > > > > [{pid,<0.5996.21>},{name,undefined},{mfargs,{riak_kv_js_vm,start_link,undefined}},{restart_type,temporary},{shutdown,2000},{child_type,worker}] > > > > > > > > > > > > > > On Wed, Jun 22, 2011 at 6:10 PM, Ryan Zezeski <rzeze...@basho.com > > > > > > > (mailto:rzeze...@basho.com) > > > > > > > (mailto:rzeze...@basho.com)> wrote: > > > > > > > > > > > > > > > > Andrew, > > > > > > > > Maybe you could elaborate on the error? I tested this against > > > > > > > > master > > > > > > > > (commit below) just now with success. > > > > > > > > 2b1a474f836d962fa035f48c05452e22fc6c2193 Change dependency to > > > > > > > > allow > > > > > > > > for R14B03 as well as R14B02 > > > > > > > > -Ryan > > > > > > > > On Wed, Jun 22, 2011 at 7:03 PM, Andrew Berman > > > > > > > > <rexx...@gmail.com (mailto:rexx...@gmail.com) > > > > > > > > (mailto:rexx...@gmail.com)> wrote: > > > > > > > > > > > > > > > > > > Hello, > > > > > > > > > I'm having issues link walking using the Map Reduce link > > > > > > > > > function. > > > > > > > > > I am using HEAD from Git, so it's possible that's the issue, > > > > > > > > > but here is > > > > > > > > > what is happening. > > > > > > > > > I've got two buckets, user and user_email where user_email > > > > > > > > > contains > > > > > > > > > a link to the user. > > > > > > > > > When I run this: > > > > > > > > > { > > > > > > > > > "inputs": [ > > > > > > > > > [ > > > > > > > > > "user_email", > > > > > > > > > "myem...@email.com (mailto:myem...@email.com)" > > > > > > > > > ] > > > > > > > > > ], > > > > > > > > > "query": [ > > > > > > > > > { > > > > > > > > > "link": { > > > > > > > > > "bucket": "user", > > > > > > > > > "tag": "user" > > > > > > > > > } > > > > > > > > > } > > > > > > > > > ] > > > > > > > > > } > > > > > > > > > I only get [["user","LikiWUPJSFuxtrhCYpsPfg","user"]] > > > > > > > > > returned. The > > > > > > > > > second I add a map function, even the simplest one > > > > > > > > > (function(v) { [v] } I > > > > > > > > > get a "map_reduce error": > > > > > > > > > { > > > > > > > > > "inputs": [ > > > > > > > > > [ > > > > > > > > > "user_email", > > > > > > > > > "myem...@email.com (mailto:myem...@email.com)" > > > > > > > > > ] > > > > > > > > > ], > > > > > > > > > "query": [ > > > > > > > > > { > > > > > > > > > "link": {"bucket":"user", "tag":"user"} > > > > > > > > > } > > > > > > > > > ,{ > > > > > > > > > "map": { > > > > > > > > > "language": "javascript", > > > > > > > > > "source": "function(v) { return[v]; }" > > > > > > > > > } > > > > > > > > > } > > > > > > > > > ] > > > > > > > > > } > > > > > > > > > Is this functionality broken? I am following what it says on > > > > > > > > > the > > > > > > > > > Wiki for the MapRed version of link walking. When I use HTTP > > > > > > > > > link walking, > > > > > > > > > it works correctly. > > > > > > > > > Thanks, > > > > > > > > > Andrew > > > > > > > > > _______________________________________________ > > > > > > > > > riak-users mailing list > > > > > > > > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > > > > > > > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > > > > > > > > > > _______________________________________________ > > > > > > > riak-users mailing list > > > > > > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > > > > > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > > > > > > _______________________________________________ > > > > > riak-users mailing list > > > > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > > > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > > > > > > > > > > > > -- > > > > Sean Cribbs <s...@basho.com (mailto:s...@basho.com)> > > > > Developer Advocate > > > > Basho Technologies, Inc. > > > > http://www.basho.com/ > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
