Riak nodes (potentially) have empty partitions
Hi, We have Riak cluster running 1.1.2. All the nodes own 6.6% of partitions, still memory usage of "beam" process on some nodes is less than the others (significantly). The number of keys on the nodes consuming less memory is also fewer. Then how come they still own the equal number of partitions as the other nodes? Would it be because some partitions are empty? Any suggestions. Cheers Nitish ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: bucket level authentication
Suggest implementing security outside of riak. The interface to applications which use riak for storage should not be riak-dependent. In addition, it would be wise to avoid exposing storage-level details like bucket choice in the security model for your applications. For more details in why riak should not be exposed to the internet, if that was in fact the intention, read this: http://aphyr.com/posts/224-do-not-expose-riak-to-the-internet On Jun 4, 2012, at 10:35 PM, raghwani sohil wrote: > > Hi , > > Is there any way to add bucket level authentication in riak ? > > suppose i have two buckets bucket1 and bucket2 under same riak cluster . I > want to access bucket1 using authentication ( privately) and buckets2 > publicly . Is it possible ? > If yes then how to do this and if no then is there any alternative to do this > . > > thanks , > Sohil Raghwani > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Follow-up: Riak / Map, Reduce - error [preflist_exhausted]
Dear colleagues,
Any idea how to adjust the processing capacity of the Riak JavaScript in
the Map / Reduce process to avoid the error [preflist_exhausted] ?
Checking the user posts, I also see a similar error report sent at 29/05
by Mr. Sati, Hohit.
The Map/Reduce and search capability are interesting features, mainly when
it comes to the analysis of unstructured data.
Thanks in advance for your hints.
Regards,
Claude
- Forwarded by Claude Falbriard/Brazil/IBM on 05/06/2012 10:44 -
From: clau...@br.ibm.com
To: riak-users@lists.basho.com
Date: 24/05/2012 10:01
Subject:Riak / Map, Reduce - error [preflist_exhausted]
Sent by:riak-users-boun...@lists.basho.com
Dear colleagues,
The map reduce process via JavaScript shows an error [preflist_exhausted]
when doing larger queries on unstructured data. Up to to 500 keys, the
queries run OK, but when going to 5000 keys in a bucket the error starts
to apear. My version is kv 1.1.2 (git). I would like to use the
Javascript procedures as it delivers more powerful pattern search behavior
compared to the Lucene like search syntax function.
I've already tried the following adjustments in the app.config file
{map_js_vm_count, 24 },
{reduce_js_vm_count, 18 },
{js_max_vm_mem, 48},
Any hint where to start the tuning to avoid this error?
Regards,
Claude
Claude Falbriard
Certified IT Specialist L2 - Middleware
AMS Hortolândia / SP - Brazil
phone:+55 19 9837 0789
cell: +55 13 8117 3316
e-mail:clau...@br.ibm.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
mochijson2 error from curl mapreduce
I saw something similar from an individual who was using javascript
(http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-July/004843.html),
but I don't think I have the same problem in erlang. Any help is greatly
appreciated.
I'm consistently getting the following error from my curl map reduce calls:
{error,{exit,{json_encode,{bad_term,…..
output dictionary here, then:
[{mochijson2,json_encode,2},
{mochijson2,'-json_encode_array/2-fun-0-',3},
{lists,foldl,3},
{mochijson2,json_encode_array,2},
{riak_kv_wm_mapred,pipe_mapred_nonchunked,5},
{webmachine_resource,resource_call,3},
{webmachine_resource,do,3},
The term that follows the error appears to the correct result of the map reduce.
I'm just trying the mr locally for testing purposes. I'm using eleveldb
backend and riak (1.1.2 2012-04-17) OSX x86_64.
Here's the curl command:
curl -v -X POST http://127.0.0.1:8098/mapred -H "Content_Type:
application/json" -d @query.js
and the query.js:
{
"inputs" : {
"bucket" : "test7",
"index" : "field1_int",
"start" : 0,
"end" : 23
},
"query" : [
{ "map" : {"language" : "erlang", "module" : "impressions", "function"
: "emitStats", "keep" : false} },
{ "reduce" : {"language" : "erlang", "module" : "impressions",
"function" : "reduceStats", "keep" : true} }
],
"timeout" : 60
}
and finally the associated erlang functions:
emitStats(G, undefined, none)->
ObjectJson = riak_object:get_value(G),
{struct, DocPropList} = mochijson:decode(ObjectJson),
I = {proplists:get_value("obj_name",DocPropList,"None"),
proplists:get_value("type",DocPropList,"None"),
proplists:get_value("obj_url",DocPropList,"None"),
"hourly"},
[dict:from_list([{I,1}])].
reduceStats(Gcounts, none)->
[lists:foldr(fun(G, Acc)->dict:merge(fun(_, X, Y) -> X+Y end,G, Acc) end,
dict:new(),
Gcounts)].
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: mochijson2 error from curl mapreduce
On Tue, Jun 5, 2012 at 1:22 PM, John Roy wrote:
> I saw something similar from an individual who was using javascript (
> http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-July/004843.html),
> but
> I don't think I have the same problem in erlang. Any help is greatly
> appreciated.
>
> I'm consistently getting the following error from my curl map reduce calls:
>
> {error,{exit,{json_encode,{bad_term,…..
>
> output dictionary here, then:
>
> [{mochijson2,json_encode,2},
> {mochijson2,'-json_encode_array/2-fun-0-',3},
> {lists,foldl,3},
> {mochijson2,json_encode_array,2},
> {riak_kv_wm_mapred,pipe_mapred_nonchunked,5},
> {webmachine_resource,resource_call,3},
> {webmachine_resource,do,3},
>
> The term that follows the error appears to the correct result of the map
> reduce.
>
> I'm just trying the mr locally for testing purposes. I'm using eleveldb
> backend and riak (1.1.2 2012-04-17) OSX x86_64.
>
> Here's the curl command:
>
> curl -v -X POST http://127.0.0.1:8098/mapred -H "Content_Type:
> application/json" -d @query.js
>
> and the query.js:
>
> {
> "inputs" : {
> "bucket" : "test7",
> "index" : "field1_int",
> "start" : 0,
> "end" : 23
> },
> "query" : [
> { "map" : {"language" : "erlang", "module" : "impressions",
> "function" : "emitStats", "keep" : false} },
> { "reduce" : {"language" : "erlang", "module" : "impressions",
> "function" : "reduceStats", "keep" : true} }
> ],
> "timeout" : 60
> }
>
> and finally the associated erlang functions:
>
> emitStats(G, undefined, none)->
> ObjectJson = riak_object:get_value(G),
> {struct, DocPropList} = mochijson:decode(ObjectJson),
> I = {proplists:get_value("obj_name",DocPropList,"None"),
> proplists:get_value("type",DocPropList,"None"),
> proplists:get_value("obj_url",DocPropList,"None"),
> "hourly"},
> [dict:from_list([{I,1}])].
>
> reduceStats(Gcounts, none)->
> [lists:foldr(fun(G, Acc)->dict:merge(fun(_, X, Y) -> X+Y end,G, Acc)
> end,
> dict:new(),
> Gcounts)].
>
I don't have any experience with Riak's MR implementation, but mochijson2
does not have any native support for serializing dict, and JSON itself
doesn't support keys that are not strings, and I suspect that one or both
of those may be your problem. mochijson2 expects to see {struct, [{Key ::
binary(), Value :: json_term()}]} for JSON objects.
-bob
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: mochijson2 error from curl mapreduce
Excellent -- that helped a great deal.
For those who may search in the future the most important thing is to have the
output of your final step (a reduce in my case) be a list of {Key :: Binary,
Value :: json_term()} as Bob identified below.
In a standard map reduce that means the input to the reduce step and the output
of the map step must also be that same data structure. So by converting the
dictionary in my map step to a string, and doing some from_list, to_list magic
in the reduce I was able to achieve my goal.
Here's my newbie looking (but working!) code for those interested in the
details:
emitStatsFromList(G, undefined, none)->
ObjectJson = riak_object:get_value(G),
{array, ListOfStructs} = mochijson:decode(ObjectJson),
[{getMrKey(I), 1} || I <- ListOfStructs].
getMrKey(StructItem)->
{struct, DocPropList} = StructItem,
SList = [proplists:get_value("obj_name",DocPropList,"None"),
proplists:get_value("type",DocPropList,"None"),
proplists:get_value("obj_url",DocPropList,"None"),
"hourly"],
string:join(SList, ",").
reduceStatsList(Gcounts, none)->
GC = [dict:from_list(Gcounts)],
Update = [lists:foldr(fun(G, Acc)->dict:merge(fun(_, X, Y) -> X+Y end,G,
Acc) end,
dict:new(),
GC)],
[OutputList] = [dict:to_list(I) || I <- Update],
OutputList.
Thanks again,
John
On Jun 5, 2012, at 2:27 PM, Bob Ippolito wrote:
> On Tue, Jun 5, 2012 at 1:22 PM, John Roy wrote:
> I saw something similar from an individual who was using javascript
> (http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-July/004843.html),
> but I don't think I have the same problem in erlang. Any help is greatly
> appreciated.
>
> I'm consistently getting the following error from my curl map reduce calls:
>
> {error,{exit,{json_encode,{bad_term,…..
>
> output dictionary here, then:
>
> [{mochijson2,json_encode,2},
> {mochijson2,'-json_encode_array/2-fun-0-',3},
> {lists,foldl,3},
> {mochijson2,json_encode_array,2},
> {riak_kv_wm_mapred,pipe_mapred_nonchunked,5},
> {webmachine_resource,resource_call,3},
> {webmachine_resource,do,3},
>
> The term that follows the error appears to the correct result of the map
> reduce.
>
> I'm just trying the mr locally for testing purposes. I'm using eleveldb
> backend and riak (1.1.2 2012-04-17) OSX x86_64.
>
> Here's the curl command:
>
> curl -v -X POST http://127.0.0.1:8098/mapred -H "Content_Type:
> application/json" -d @query.js
>
> and the query.js:
>
> {
> "inputs" : {
> "bucket" : "test7",
> "index" : "field1_int",
> "start" : 0,
> "end" : 23
> },
> "query" : [
> { "map" : {"language" : "erlang", "module" : "impressions",
> "function" : "emitStats", "keep" : false} },
> { "reduce" : {"language" : "erlang", "module" : "impressions",
> "function" : "reduceStats", "keep" : true} }
> ],
> "timeout" : 60
> }
>
> and finally the associated erlang functions:
>
> emitStats(G, undefined, none)->
> ObjectJson = riak_object:get_value(G),
> {struct, DocPropList} = mochijson:decode(ObjectJson),
> I = {proplists:get_value("obj_name",DocPropList,"None"),
> proplists:get_value("type",DocPropList,"None"),
> proplists:get_value("obj_url",DocPropList,"None"),
> "hourly"},
> [dict:from_list([{I,1}])].
>
> reduceStats(Gcounts, none)->
> [lists:foldr(fun(G, Acc)->dict:merge(fun(_, X, Y) -> X+Y end,G, Acc) end,
> dict:new(),
> Gcounts)].
>
> I don't have any experience with Riak's MR implementation, but mochijson2
> does not have any native support for serializing dict, and JSON itself
> doesn't support keys that are not strings, and I suspect that one or both of
> those may be your problem. mochijson2 expects to see {struct, [{Key ::
> binary(), Value :: json_term()}]} for JSON objects.
>
> -bob
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
