Mathias,
I took the BERT encoding and then encoded that as Base64 which should
pass the test of valid UTF-8 characters. However, now I'm starting to
think that maybe doing two encodings and storing that for the purpose
of saving space is not worth the trade-off in performance vs just
storing the data in JSON format. Do you guys have any thoughts on
this?
Thanks,
Andrew
On Thu, Jun 23, 2011 at 8:39 AM, Mathias Meyer <math...@basho.com> wrote:
> Andrew,
>
> the data looks like JSON, but it's not valid JSON. Have a look at the list
> that's in the data section (which is your BERT encoded data), the first
> character in that list is 131, which is not a valid UTF-8 character, and JSON
> only allows valid UTF-8 characters. With a binary-encoded format, there's
> always a chance for a control character like that to blow up the JSON
> generated before and after the MapReduce code is executed. With JSON, content
> agnosticism only goes as far as the set of legal characters allows.
>
> On a side note, if the data were a valid representation of a string, you
> would see it as a string in the log file as well, not just as a list of
> numbers.
>
> Mathias Meyer
> Developer Advocate, Basho Technologies
>
>
> On Donnerstag, 23. Juni 2011 at 17:31, Andrew Berman wrote:
>
>> But isn't the value itself JSON? Meaning this part:
>>
>> {struct,
>> [{<<"bucket">>,<<"user">>},
>> {<<"key">>,<<"LikiWUPJSFuxtrhCYpsPfg">>},
>> {<<"vclock">>,
>>
>> <<"a85hYGBgzGDKBVIsLKaZdzOYEhnzWBmes6Yd58sCAA==">>},
>> {<<"values">>,
>> [{struct,
>> [{<<"metadata">>,
>> {struct,
>> [{<<"X-Riak-VTag">>,
>> <<"1KnL9Dlma9Yg4eMhRuhwtx">>},
>> {<<"X-Riak-Last-Modified">>,
>> <<"Fri, 10 Jun 2011 03:05:11 GMT">>}]}},
>> {<<"data">>,
>>
>> <<131,108,0,0,0,18,104,2,100,0,6,114,...>>}]}]}
>>
>> So the only thing that is not JSON is the data itself, but when I get
>> the value, shouldn't I be getting the all the info above which is JSON
>> encoded?
>>
>> Thank you all for your help,
>>
>> Andrew
>>
>> On Thu, Jun 23, 2011 at 8:17 AM, Sean Cribbs <s...@basho.com
>> (mailto:s...@basho.com)> wrote:
>> > The object has to be JSON-encoded to be marshalled into the Javascript VM,
>> > and also on the way out if the Accept header indicates application/json. So
>> > you have two places where it needs to be encodable into JSON.
>> > On Thu, Jun 23, 2011 at 11:14 AM, Andrew Berman <rexx...@gmail.com
>> > (mailto:rexx...@gmail.com)> wrote:
>> > >
>> > > Mathias,
>> > >
>> > > I thought Riak was content agnostic when it came to the data being
>> > > stored? The map phase is not running Riak.mapValuesJson, so why is
>> > > the data itself going through the JSON parser? The JSON value
>> > > returned by v with all the info is valid and I see the struct atom in
>> > > there so mochijson2 can parse it properly, but I'm not clear why
>> > > mochijson2 would be coughing at the data part.
>> > >
>> > > --Andrew
>> > >
>> > > On Thu, Jun 23, 2011 at 5:32 AM, Mathias Meyer <math...@basho.com
>> > > (mailto:math...@basho.com)> wrote:
>> > > > Andrew,
>> > > >
>> > > > you're indeed hitting a JSON encoding problem here. BERT is binary
>> > > > data,
>> > > > and won't make the JSON parser happy when trying to deserialize it,
>> > > > before
>> > > > handing it into the map phase. You have two options here, and none of
>> > > > them
>> > > > will involve JavaScript as the MapReduce language.
>> > > >
>> > > > 1.) Use the Protobuff API, use Erlang functions to return the value or
>> > > > object (e.g. riak_mapreduce:map_object_value or
>> > > > riak_kv_mapreduce:map_identity), and then run MapReduce queries with
>> > > > the
>> > > > content type 'application/x-erlang-binary'. However, you're
>> > > > constrained by
>> > > > client libraries here, e.g. Ruby and Python don't support this content
>> > > > type
>> > > > for MapReduce on the Protobuffs interface yet, so you'd either
>> > > > implement
>> > > > something custom, or resort to a client that does, riak-erlang-client
>> > > > comes
>> > > > to mind, though it was proven to be possible using the Java client too,
>> > > > thanks to Russell. See [1] and [2]
>> > > >
>> > > > 2.) Convert the result from BERT into a JSON-parseable structure inside
>> > > > an Erlang map function, before it's returned to the client.
>> > > >
>> > > > The second approach certainly is less restrictive in terms of API
>> > > > usage,
>> > > > but certainly involves some overhead inside of the MapReduce request
>> > > > itself,
>> > > > but is less prone to encoding/decoding issues with JSON.
>> > > >
>> > > > Mathias Meyer
>> > > > Developer Advocate, Basho Technologies
>> > > >
>> > > > [1]
>> > > > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-June/004447.html
>> > > > [2]
>> > > > http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-June/004485.html
>> > > >
>> > > > On Donnerstag, 23. Juni 2011 at 07:59, Andrew Berman wrote:
>> > > >
>> > > > > Hey Ryan,
>> > > > >
>> > > > > Here is the error from the sasl log. It looks like some sort of
>> > > > > encoding error. Any thoughts on how to fix this? I am storing the
>> > > > > data as BERT encoded binary and I set the content-type as
>> > > > > application/octet-stream.
>> > > > >
>> > > > > Thanks for your help!
>> > > > >
>> > > > > Andrew
>> > > > >
>> > > > > ERROR REPORT==== 9-Jun-2011::21:37:05 ===
>> > > > > ** Generic server <0.5996.21> terminating
>> > > > > ** Last message in was {batch_dispatch,
>> > > > > {map,
>> > > > > {jsanon,<<"function(value) {return [value];}">>},
>> > > > > [{struct,
>> > > > > [{<<"bucket">>,<<"user">>},
>> > > > > {<<"key">>,<<"LikiWUPJSFuxtrhCYpsPfg">>},
>> > > > > {<<"vclock">>,
>> > > > >
>> > > > > <<"a85hYGBgzGDKBVIsLKaZdzOYEhnzWBmes6Yd58sCAA==">>},
>> > > > > {<<"values">>,
>> > > > > [{struct,
>> > > > > [{<<"metadata">>,
>> > > > > {struct,
>> > > > > [{<<"X-Riak-VTag">>,
>> > > > > <<"1KnL9Dlma9Yg4eMhRuhwtx">>},
>> > > > > {<<"X-Riak-Last-Modified">>,
>> > > > > <<"Fri, 10 Jun 2011 03:05:11 GMT">>}]}},
>> > > > > {<<"data">>,
>> > > > >
>> > > > > <<131,108,0,0,0,18,104,2,100,0,6,114,...>>}]}]}]},
>> > > > > <<"user">>,none]}}
>> > > > > ** When Server state ==
>> > > > > {state,<0.143.0>,riak_kv_js_map,#Port<0.92614>,true}
>> > > > > ** Reason for termination ==
>> > > > > ** {function_clause,[{js_driver,eval_js,
>> > > > > [#Port<0.92614>,{error,bad_encoding},5000]},
>> > > > > {riak_kv_js_vm,invoke_js,2},
>> > > > > {riak_kv_js_vm,define_invoke_anon_js,3},
>> > > > > {riak_kv_js_vm,handle_call,3},
>> > > > > {gen_server,handle_msg,5},
>> > > > > {proc_lib,init_p_do_apply,3}]}
>> > > > >
>> > > > > =CRASH REPORT==== 9-Jun-2011::21:37:05 ===
>> > > > > crasher:
>> > > > > initial call: riak_kv_js_vm:init/1
>> > > > > pid: <0.5996.21>
>> > > > > registered_name: []
>> > > > > exception exit:
>> > > > >
>> > > > > {function_clause,[{js_driver,eval_js,[#Port<0.92614>,{error,bad_encoding},5000]},{riak_kv_js_vm,invoke_js,2},{riak_kv_js_vm,define_invoke_anon_js,3},{riak_kv_js_vm,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>> > > > > in function gen_server:terminate/6
>> > > > > in call from proc_lib:init_p_do_apply/3
>> > > > > ancestors: [riak_kv_js_sup,riak_kv_sup,<0.128.0>]
>> > > > > messages: []
>> > > > > links: [<0.142.0>,<0.6009.21>]
>> > > > > dictionary: []
>> > > > > trap_exit: false
>> > > > > status: running
>> > > > > heap_size: 4181
>> > > > > stack_size: 24
>> > > > > reductions: 2586
>> > > > > neighbours:
>> > > > > neighbour:
>> > > > > [{pid,<0.6009.21>},{registered_name,[]},{initial_call,{riak_kv_mapper,init,[Argument__1]}},{current_function,{gen,do_call,4}},{ancestors,[riak_kv_mapper_sup,riak_kv_sup,<0.128.0>]},{messages,[]},{links,[<0.5996.21>,<12337.6227.21>,<0.162.0>]},{dictionary,[]},{trap_exit,false},{status,waiting},{heap_size,987},{stack_size,53},{reductions,1043}]
>> > > > > =SUPERVISOR REPORT==== 9-Jun-2011::21:37:05 ===
>> > > > > Supervisor: {local,riak_kv_js_sup}
>> > > > > Context: child_terminated
>> > > > > Reason:
>> > > > >
>> > > > > {function_clause,[{js_driver,eval_js,[#Port<0.92614>,{error,bad_encoding},5000]},{riak_kv_js_vm,invoke_js,2},{riak_kv_js_vm,define_invoke_anon_js,3},{riak_kv_js_vm,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
>> > > > > Offender:
>> > > > >
>> > > > > [{pid,<0.5996.21>},{name,undefined},{mfargs,{riak_kv_js_vm,start_link,undefined}},{restart_type,temporary},{shutdown,2000},{child_type,worker}]
>> > > > >
>> > > > > On Wed, Jun 22, 2011 at 6:10 PM, Ryan Zezeski <rzeze...@basho.com
>> > > > > (mailto:rzeze...@basho.com)
>> > > > > (mailto:rzeze...@basho.com)> wrote:
>> > > > > >
>> > > > > > Andrew,
>> > > > > > Maybe you could elaborate on the error? I tested this against
>> > > > > > master
>> > > > > > (commit below) just now with success.
>> > > > > > 2b1a474f836d962fa035f48c05452e22fc6c2193 Change dependency to allow
>> > > > > > for R14B03 as well as R14B02
>> > > > > > -Ryan
>> > > > > > On Wed, Jun 22, 2011 at 7:03 PM, Andrew Berman <rexx...@gmail.com
>> > > > > > (mailto:rexx...@gmail.com)
>> > > > > > (mailto:rexx...@gmail.com)> wrote:
>> > > > > > >
>> > > > > > > Hello,
>> > > > > > > I'm having issues link walking using the Map Reduce link
>> > > > > > > function.
>> > > > > > > I am using HEAD from Git, so it's possible that's the issue, but
>> > > > > > > here is
>> > > > > > > what is happening.
>> > > > > > > I've got two buckets, user and user_email where user_email
>> > > > > > > contains
>> > > > > > > a link to the user.
>> > > > > > > When I run this:
>> > > > > > > {
>> > > > > > > "inputs": [
>> > > > > > > [
>> > > > > > > "user_email",
>> > > > > > > "myem...@email.com (mailto:myem...@email.com)"
>> > > > > > > ]
>> > > > > > > ],
>> > > > > > > "query": [
>> > > > > > > {
>> > > > > > > "link": {
>> > > > > > > "bucket": "user",
>> > > > > > > "tag": "user"
>> > > > > > > }
>> > > > > > > }
>> > > > > > > ]
>> > > > > > > }
>> > > > > > > I only get [["user","LikiWUPJSFuxtrhCYpsPfg","user"]] returned.
>> > > > > > > The
>> > > > > > > second I add a map function, even the simplest one (function(v)
>> > > > > > > { [v] } I
>> > > > > > > get a "map_reduce error":
>> > > > > > > {
>> > > > > > > "inputs": [
>> > > > > > > [
>> > > > > > > "user_email",
>> > > > > > > "myem...@email.com (mailto:myem...@email.com)"
>> > > > > > > ]
>> > > > > > > ],
>> > > > > > > "query": [
>> > > > > > > {
>> > > > > > > "link": {"bucket":"user", "tag":"user"}
>> > > > > > > }
>> > > > > > > ,{
>> > > > > > > "map": {
>> > > > > > > "language": "javascript",
>> > > > > > > "source": "function(v) { return[v]; }"
>> > > > > > > }
>> > > > > > > }
>> > > > > > > ]
>> > > > > > > }
>> > > > > > > Is this functionality broken? I am following what it says on the
>> > > > > > > Wiki for the MapRed version of link walking. When I use HTTP
>> > > > > > > link walking,
>> > > > > > > it works correctly.
>> > > > > > > Thanks,
>> > > > > > > Andrew
>> > > > > > > _______________________________________________
>> > > > > > > riak-users mailing list
>> > > > > > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
>> > > > > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> > > > >
>> > > > > _______________________________________________
>> > > > > riak-users mailing list
>> > > > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
>> > > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> > >
>> > > _______________________________________________
>> > > riak-users mailing list
>> > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
>> > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>> >
>> >
>> >
>> > --
>> > Sean Cribbs <s...@basho.com (mailto:s...@basho.com)>
>> > Developer Advocate
>> > Basho Technologies, Inc.
>> > http://www.basho.com/
>
>
>
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
