Hi Doug, That's correct. Do you know the names of all of your keys? If so, it should be pretty trivial to write a script to read the values from the data store, decode, and re-encode using your favorite JSON library.
- Chris -- Christopher Meiklejohn Software Engineer Basho Technologies, Inc. On Thursday, August 29, 2013 at 3:06 AM, Doug Read wrote: > Hi Crhis, > > I just say this, yeah my testing also seems to point to spidermonkey as well. > As far as your suggestion, decode and re-encode, I am not sure I understand > what you mean. Can you provide an example or elaborate? Is this changing the > key values? > > thanks > doug > > > > On Tue, Aug 20, 2013 at 4:07 PM, Chris Meiklejohn <cmeiklej...@basho.com > (mailto:cmeiklej...@basho.com)> wrote: > > Hi Doug, > > > > I've spent a good portion of the day tracing the execution of the Erlang > > port driver with gdb and have definitely determined that Spidermonkey is > > returning an empty response from the function call with a success value > > from the execution. My hunch is that this is definitely related to the > > encoding of the object causing Spidermonkey to fail to execute correctly > > (especially based on our previous testing). > > > > As it's a pretty big undertaking to upgrade erlang_js to a newer version of > > Spidermonkey (or Jagermonkey or V8), I'd definitely recommend writing a > > script to enumerate your keys, decode and re-encode your objects as we've > > already shown that that solution works. > > > > - Chris > > > > > > On Tue, Aug 20, 2013 at 11:27 AM, Chris Meiklejohn <cmeiklej...@basho.com > > (mailto:cmeiklej...@basho.com)> wrote: > > > Hi Doug, > > > > > > While this is certainly possible, the JSON that's being used as the > > > function argument does decode both with Spidermonkey (calling into the > > > JSON decode function with that object directly), and with mochijson2. > > > It's a combination of the function application with that object as the > > > functions argument. > > > > > > - Chris > > > > > > > > > On Mon, Aug 19, 2013 at 11:23 PM, Doug Read <doug.r...@qnary.com > > > (mailto:doug.r...@qnary.com)> wrote: > > > > I noticed the value in some of the json fields have the single quotes > > > > and double quotes, is it possible they aren't being correctly escaped > > > > when being passed to spidermonkey? > > > > > > > > > > > > On Mon, Aug 19, 2013 at 5:33 PM, Chris Meiklejohn > > > > <cmeiklej...@basho.com (mailto:cmeiklej...@basho.com)> wrote: > > > > > Hello Doug, > > > > > > > > > > I've verified that the problem is not related to that particular > > > > > parsing function, as the following JavaScript MapReduce code will > > > > > trigger the exception: > > > > > > > > > > { "map": {"language": "javascript", "source": "function(value, > > > > > keyData, arg) { return [value]; }" } > > > > > > > > > > Specifically, what's happening inside of the erlang_js module is this > > > > > function is being turned into an anonymous immediately invoking > > > > > function expression like such: > > > > > > > > > > (function(value, keyData, arg) { return [value]; })(RIAK_OBJECT_HERE) > > > > > > > > > > This string is then marshalled and called through the Erlang port > > > > > driver to Spidermonkey. The response returned is the following: > > > > > > > > > > {ok, <<>>} > > > > > > > > > > This then causes the js_mochijson2 decode to fail, because it's not > > > > > defined over empty binary objects. > > > > > > > > > > I've verified that this is only the behavior with the object you > > > > > provided, which leads me to believe that something inside of > > > > > Spidermonkey is blowing up, which is triggering the empty binary > > > > > response. > > > > > > > > > > I'm going to continue tracking it down, however, I'd advise moving > > > > > forward to switch to Erlang-based MapReduce jobs, which perform much > > > > > better, as they don't introduce overhead for > > > > > serialization/deserialization and don't require the use of the > > > > > JavaScript virtual machine. > > > > > > > > > > - Chris > > > > > > > > > > > > > > > > > > > > On Mon, Aug 19, 2013 at 4:14 PM, Chris Meiklejohn > > > > > <cmeiklej...@basho.com (mailto:cmeiklej...@basho.com)> wrote: > > > > > > Hi Doug, > > > > > > > > > > > > Can you shoot over the multi get job you are using? > > > > > > > > > > > > - Chris > > > > > > > > > > > > > > > > > > On Mon, Aug 19, 2013 at 4:11 PM, Doug Read <doug.r...@qnary.com > > > > > > (mailto:doug.r...@qnary.com)> wrote: > > > > > > > Hi Chris, > > > > > > > > > > > > > > I appreciate the help. Please let me know if I can help in > > > > > > > anyway. For now I just switched to a "multi" get. > > > > > > > > > > > > > > thanks > > > > > > > doug > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Mon, Aug 19, 2013 at 4:09 PM, Chris Meiklejohn > > > > > > > <cmeiklej...@basho.com (mailto:cmeiklej...@basho.com)> wrote: > > > > > > > > Hi Doug, > > > > > > > > > > > > > > > > After loading your backup, I can trigger the crash through the > > > > > > > > JavaScript map/reduce job. However, if I do the following: > > > > > > > > > > > > > > > > {ok, Client} = riak:local_client(). > > > > > > > > {ok, O} = Client:get(<<"BUCKET">>, <<"KEY">>). > > > > > > > > > > > > > > > > both of these calls succeed: > > > > > > > > > > > > > > > > js_mochijson2:decode(riak_object:get_values(O)). > > > > > > > > mochijson2:decode(riak_object:get_values(O)). > > > > > > > > > > > > > > > > I'm continuing to track down where the crash is occurring, but > > > > > > > > it appears to be something related to erlang_js. > > > > > > > > > > > > > > > > - Chris > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 16, 2013 at 10:45 AM, Doug Read > > > > > > > > <doug.r...@qnary.com (mailto:doug.r...@qnary.com)> wrote: > > > > > > > > > So you are right about the characters in the json, I remove > > > > > > > > > most of the key, just leaving some valid json with the > > > > > > > > > special characters, it produces the same error. > > > > > > > > > > > > > > > > > > A simple node program which reads the json from file and does > > > > > > > > > JSON.parse; this succeeds. > > > > > > > > > > > > > > > > > > var fs = require("fs"); > > > > > > > > > > > > > > > > > > fs.readFile("/tmp/bucket_export/s/e/a/searches_bad.json", > > > > > > > > > function(err, data) { > > > > > > > > > console.log(data); > > > > > > > > > var json = JSON.parse(data); > > > > > > > > > console.log(json); > > > > > > > > > }); > > > > > > > > > > > > > > > > > > > > > > > > > > > So the javascript VM or something before/after it is having > > > > > > > > > trouble parsing it but i am not sure how to debug from here. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, Aug 16, 2013 at 1:58 AM, Christopher Meiklejohn > > > > > > > > > <cmeiklej...@basho.com (mailto:cmeiklej...@basho.com)> wrote: > > > > > > > > > > Hi Doug, > > > > > > > > > > > > > > > > > > > > Just going through your email again; I just noticed that > > > > > > > > > > there is a typo in the bucket name between those two > > > > > > > > > > commands: > > > > > > > > > > > > > > > > > > > > JavaScript: > > > > > > > > > > > > > > > > > > > > curl -XPOST http://localhost:8098/mapred -H 'Content-Type: > > > > > > > > > > application/json' -d '{"inputs":[ [ > > > > > > > > > > "aaaaaaaa-4536-9048-87ef2e48ddda", > > > > > > > > > > "key_5ad26d0d-4d28-40ca-afcb-1c9895cc5c71" ] ], "query":[ { > > > > > > > > > > "map": { "name": "Riak.mapValuesJson", "language": > > > > > > > > > > "javascript" } }, { "reduce": { "name": > > > > > > > > > > "Riak.filterNotFound", "language": "javascript" } } ] }' > > > > > > > > > > > > > > > > > > > > Erlang: > > > > > > > > > > > > > > > > > > > > curl -XPOST http://localhost:8098/mapred -H 'Content-Type: > > > > > > > > > > application/json' -d '{"inputs":[ [ > > > > > > > > > > "aaaaaaaa-50d7-4536-9048-87ef2e48ddda", > > > > > > > > > > "key_5ad26d0d-4d28-40ca-afcb-1c9895cc5c71" ] ], > > > > > > > > > > "query":[{"map":{"language":"erlang","module":"riak_kv_mapreduce","function":"map_object_value"}} > > > > > > > > > > ] }' > > > > > > > > > > > > > > > > > > > > Specifically, 'aaaaaaaa-4536-9048-87ef2e48ddda' in the > > > > > > > > > > failed JavaScript command vs > > > > > > > > > > 'aaaaaaaa-50d7-4536-9048-87ef2e48ddda' in the successful > > > > > > > > > > Erlang command. > > > > > > > > > > > > > > > > > > > > I've verified this is triggering not_founds in the dump you > > > > > > > > > > supplied. > > > > > > > > > > > > > > > > > > > > - Chris > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Christopher Meiklejohn > > > > > > > > > > Software Engineer > > > > > > > > > > Basho Technologies, Inc. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Friday, August 16, 2013 at 1:36 AM, Doug Read wrote: > > > > > > > > > > > > > > > > > > > > > No problem. > > > > > > > > > > > > > > > > > > > > > > Riak 1.3.1 > > > > > > > > > > > Have reproduced the error on 1.3.1 Mac osx and 1.4.1 on > > > > > > > > > > > Ubuntu. I am on my phone now I'll get the build and > > > > > > > > > > > package versions. > > > > > > > > > > > > > > > > > > > > > > That is the erlang mapreduce, that has always worked it > > > > > > > > > > > is the javascript version that fails. The one I > > > > > > > > > > > originally posted but basically any javascript mapreduce > > > > > > > > > > > which takes that bucket key as input. > > > > > > > > > > > > > > > > > > > > > > On Friday, August 16, 2013, Christopher Meiklejohn wrote: > > > > > > > > > > > > Hi Doug, > > > > > > > > > > > > > > > > > > > > > > > > First, my apologies. I confused two email responses I > > > > > > > > > > > > was writing, and that's why my previous e-mail was a > > > > > > > > > > > > bit out of context. > > > > > > > > > > > > > > > > > > > > > > > > I've loaded your backup file into a locally built > > > > > > > > > > > > cluster here off of the Riak 1.2 branch, but I'm still > > > > > > > > > > > > unable to reproduce the issue using the following > > > > > > > > > > > > map/reduce command: > > > > > > > > > > > > > > > > > > > > > > > > curl -XPOST http://localhost:8098/mapred -H > > > > > > > > > > > > 'Content-Type: application/json' -d '{"inputs":[ [ > > > > > > > > > > > > "aaaaaaaa-50d7-4536-9048-87ef2e48ddda", > > > > > > > > > > > > "key_5ad26d0d-4d28-40ca-afcb-1c9895cc5c71" ] ], > > > > > > > > > > > > "query":[{"map":{"language":"erlang","module":"riak_kv_mapreduce","function":"map_object_value"}} > > > > > > > > > > > > ] }' > > > > > > > > > > > > > > > > > > > > > > > > Can you please provide which operating system, build, > > > > > > > > > > > > and package version of Riak you are running? > > > > > > > > > > > > > > > > > > > > > > > > - Chris > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > Christopher Meiklejohn > > > > > > > > > > > > Software Engineer > > > > > > > > > > > > Basho Technologies, Inc. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Friday, August 16, 2013 at 1:23 AM, Christopher > > > > > > > > > > > > Meiklejohn wrote: > > > > > > > > > > > > > > > > > > > > > > > > > Hi Doug, > > > > > > > > > > > > > > > > > > > > > > > > > > Can you provide more information as to how you build > > > > > > > > > > > > > the image with Vagrant so I can try to reproduce it? > > > > > > > > > > > > > The configuration alone isn't going to be enough as > > > > > > > > > > > > > this appears to be a systems related issue. > > > > > > > > > > > > > > > > > > > > > > > > > > - Chris > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > Christopher Meiklejohn > > > > > > > > > > > > > Software Engineer > > > > > > > > > > > > > Basho Technologies, Inc. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Friday, August 16, 2013 at 1:20 AM, Doug Read > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi Chris, > > > > > > > > > > > > > > > > > > > > > > > > > > > > I made a riak-admin backup of the key which > > > > > > > > > > > > > > reproduces the issue. I was wondering if you could > > > > > > > > > > > > > > give me some direction of how to debug the issue. > > > > > > > > > > > > > > > > > > > > > > > > > > > > thanks > > > > > > > > > > > > > > doug > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 15, 2013 at 3:41 PM, Chris Meiklejohn > > > > > > > > > > > > > > <cmeiklej...@basho.com > > > > > > > > > > > > > > (mailto:cmeiklej...@basho.com) > > > > > > > > > > > > > > (mailto:cmeiklej...@basho.com)> wrote: > > > > > > > > > > > > > > > The best guess I have at this point is probably > > > > > > > > > > > > > > > something related to character encoding, but > > > > > > > > > > > > > > > without a reproduction case, I'm not able to > > > > > > > > > > > > > > > debug it any further. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Good luck with the upgrade tonight! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Chris > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 15, 2013 at 3:39 PM, Doug Read > > > > > > > > > > > > > > > <doug.r...@qnary.com (mailto:doug.r...@qnary.com) > > > > > > > > > > > > > > > (mailto:doug.r...@qnary.com)> wrote: > > > > > > > > > > > > > > > > I redirected the output of curl into a file on > > > > > > > > > > > > > > > > an ubuntu box. I am upgrading the cluster to > > > > > > > > > > > > > > > > 1.4.1 tonight. To your point I PUT the value > > > > > > > > > > > > > > > > into the key locally (3 node cluster) and > > > > > > > > > > > > > > > > couldn't reproduce either. Also i am turning on > > > > > > > > > > > > > > > > the java vm logging. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 15, 2013 at 3:33 PM, Chris > > > > > > > > > > > > > > > > Meiklejohn <cmeiklej...@basho.com > > > > > > > > > > > > > > > > (mailto:cmeiklej...@basho.com) > > > > > > > > > > > > > > > > (mailto:cmeiklej...@basho.com)> wrote: > > > > > > > > > > > > > > > > > Hi Doug, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I've configured a Riak 1.2 cluster, and run > > > > > > > > > > > > > > > > > the aforementioned map-reduce job in Erlang > > > > > > > > > > > > > > > > > and I can't trigger the crash. I'm getting > > > > > > > > > > > > > > > > > the expected results of the map/reduce job. > > > > > > > > > > > > > > > > > How did you send me the object that you > > > > > > > > > > > > > > > > > provided off-list? > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Chris > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Aug 15, 2013 at 12:36 PM, Chris > > > > > > > > > > > > > > > > > Meiklejohn <cmeiklej...@basho.com > > > > > > > > > > > > > > > > > (mailto:cmeiklej...@basho.com) > > > > > > > > > > > > > > > > > (mailto:cmeiklej...@basho.com)> wrote: > > > > > > > > > > > > > > > > > > Hi Doug, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Can you provide a sample of the JSON that > > > > > > > > > > > > > > > > > > you're storing in these objects? It appears > > > > > > > > > > > > > > > > > > that mochijson2's tokenizer is crashing > > > > > > > > > > > > > > > > > > because it thinks the JSON is not valid, > > > > > > > > > > > > > > > > > > where the Spidermonkey parsing is > > > > > > > > > > > > > > > > > > succeeding. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > - Chris > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Aug 14, 2013 at 10:58 AM, Doug Read > > > > > > > > > > > > > > > > > > <doug.r...@qnary.com > > > > > > > > > > > > > > > > > > (mailto:doug.r...@qnary.com) > > > > > > > > > > > > > > > > > > (mailto:doug.r...@qnary.com)> wrote: > > > > > > > > > > > > > > > > > > > The following MapReduce job fails using > > > > > > > > > > > > > > > > > > > javascript but succeeds when using erlang. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Riak 1.2.0 2012-0806 Debian x86_64 > > > > > > > > > > > > > > > > > > > 3 nodes, n_val=3 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Riak diag gives large list of > > > > > > > > > > > > > > > > > > > [warning] The following preflists do not > > > > > > > > > > > > > > > > > > > satisfy the n_val: > > > > > > > > > > > > > > > > > > > Not really sure what this means but > > > > > > > > > > > > > > > > > > > thought i would share. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > JAVASCRIPT: > > > > > > > > > > > > > > > > > > > curl -XPOST http://localhost:8098/mapred > > > > > > > > > > > > > > > > > > > -H 'Content-Type: application/json' -d > > > > > > > > > > > > > > > > > > > '{"inputs":[ [ > > > > > > > > > > > > > > > > > > > "aaaaaaaa-4536-9048-87ef2e48ddda", > > > > > > > > > > > > > > > > > > > "key_5ad26d0d-4d28-40ca-afcb-1c9895cc5c71" > > > > > > > > > > > > > > > > > > > ] ], "query":[ { "map": { "name": > > > > > > > > > > > > > > > > > > > "Riak.mapValu > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > _______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
