Re: issue on riak bulk loading---taking huge time
Hey Sangeetha,
at first sight, what strikes me as odd about your bulk import is that it shells
out to curl. That has a significant impact on the time it takes to load the
data into Riak. As a first means to improve script and performance, I'd
recommend looking into using the Riak Erlang client instead [1]. Alternatively
you could also run the Erlang code in the context of a locally running Riak and
use riak:local_client() [2].
Cheers, Mathias
http://riakhandbook.com
[1] https://github.com/basho/riak-erlang-client
[2] https://github.com/basho/riak_kv/blob/master/src/riak_client.erl
On Monday, 14. May 2012 at 07:33, sangeetha.pattabiram...@cognizant.com wrote:
>
>
>
>
>
> From: Pattabi Raman, Sangeetha (Cognizant)
> Sent: Thursday, May 10, 2012 3:25 PM
> To: riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> Subject: issue on riak bulk loading---taking huge time
>
>
>
>
>
>
> Dear team,
>
>
>
>
> FYI:we have a 4 quad core intel processor on each server on 2 node cluster
> with more than 1 TB of storage
>
>
> I Ihave constructed the 2 node physical machine riak cluster with n_val 2 and
> my app.config ,vm.args are attached for your reference..
>
>
>
>
> Please tell me where the bulk inserted data onto riak db gets stored on Local
> file system…its taking huge time to load small size itself…how to tune it to
> perform to large scale since we deal wit hbigdata of in few hungred
> GB’s?
>
>
>
>
> Cmd used:time ./load_data1m Customercalls1m.csv
>
>
>
>
> ./load_data100m CustomerCalls100m(got this error so changed default config of
> app.config…from 8 MB to 3072 MB
>
>
> escript: exception error: no match of right hand side value {error,enoent}
>
>
>
>
>
>
> size
>
>
>
> Load time
>
>
>
> No of mappersonapp.config
>
>
>
> Js-max-vm-mem on app.config
>
>
>
> Js-thread-stack
>
>
>
>
> 100k(10,lakhrows)—5 MB
>
>
>
> 20m39.625 seconds
>
>
>
> 48
>
>
>
> 3 GB 3072MB(changedfromdefault 8MB)since i/p data is large)
>
>
>
> 3 GB 3072MB(changedfromdefault 8MB)since i/p data is large)
>
>
>
>
> 1millionrows---54 MB
>
>
>
> 198m42.375seconds
>
>
>
> 48
>
>
>
> 3 GB 3072MB(changedfromdefault 8MB)since i/p data is large)
>
>
>
> 3 GB 3072MB(changedfromdefault 8MB)since i/p data is large)
>
>
>
>
>
>
> .
>
>
>
>
>
>
> ./load_data script used:
>
>
>
>
> #!/usr/local/bin/escript
>
>
> main([Filename]) ->
>
>
> {ok, Data} = file:read_file(Filename),
>
>
> Lines = tl(re:split(Data, "\r?\n", [{return, binary},trim])),
>
>
> lists:foreach(fun(L) -> LS = re:split(L, ","), format_and_insert(LS) end,
> Lines).
>
>
>
>
> format_and_insert(Line) ->
>
>
> JSON =
> io_lib:format("{\"id\":\"~s\",\"phonenumber\":~s,\"callednumber\":~s,\"starttime\":~s,\"endtime\":~s,\"status\":~s}",
> Line),
>
>
> Command = io_lib:format("curl -X PUT
> http://10.232.5.169:8098/riak/CustomerCalls100k/~s -d '~s' -H 'content-type:
> application/json'", [hd(Line),JSON]),
>
>
> io:format("Inserting: ~s~n", [hd(Line)]),
>
>
> os:cmd(Command).
>
>
>
>
>
>
>
>
> Thanks in advance!!waiting fr the reply…plz anyone help..struck u
> pwit hbulk loading…..and make me clear how riak splits the data and gets
> loaded on cluster
>
>
> Thanks & regards
>
>
> sangeetha
>
>
>
>
> This e-mail and any files transmitted with it are for the sole use of the
> intended recipient(s) and may contain confidential and privileged
> information. If you are not the intended recipient(s), please reply to the
> sender and destroy all copies of the original message. Any unauthorized
> review, use, disclosure, dissemination, forwarding, printing or copying of
> this email, and/or any action taken in reliance on the contents of this
> e-mail is strictly prohibited and may be unlawful.
>
>
>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
> Attachments:
> - app.config.txt
>
> - vm.args.txt
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Reviewers for Riak Handbook Update
Hey! I'm just about wrapping up the changes and new content for the upcoming update of the Riak Handbook. To make sure what I wrote makes (mostly) sense, I'm looking for one or two technical reviewers of the new content. The focus is more on whether everything makes sense, so not necessarily on grammar and spelling, or if there's something that needs further explaining and so forth. It's a bit more than 40 pages (compared to the PDF of the first version), and it'd have to be in the next couple of days, as I want to ship next week. If anyone's got some time to spare and wants to have a go at the new stuff, I'd very much appreciate it. Thanks! Cheers, Mathias http://riakhandbook.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
[ANN] Riak Handbook 1.1 released!
Hey there! Just thought I'd let you guys know that I just released a new version of the Riak Handbook, a free update if you already bought it. It comes with more than 40 pages of new content, focussing primarily on use cases and usage scenarios (data modelling, access patterns, pros and cons), operations (lots of details on monitoring, load-balancing and network placement of nodes), pre- and post-commit, plus a whole slew of little additions and fixes. Also, the book now comes as a man page for easy reading on the command line. You can read all about what's in the release here: http://www.paperplanes.de/2012/5/30/riak-handbook-11-is-out.html To celebrate, the book is 25% off through next Monday: http://riakhandbook.com/ Cheers, Mathias ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: what is content type
Anand, A content type is a cue as to what kind of data you're storing in Riak. The concept is based on Internet media types [1]. The cue may or may not be important to your application to figure out what to do with the data. It's also important for certain Riak features like Riak Search, where you can automatically index data based on it's content type (XML, JSON, etc.). If all you're storing is plain text, you might as well ignore it and store everything as application/octet-stream, but on the other hand, it doesn't hurt to just specify text/plain and have it properly marked as what kind of data it is. Cheers, Mathias http://riakhandbook.com [1] https://en.wikipedia.org/wiki/Internet_media_type On Monday, 25. June 2012 at 09:32, Anand Hegde wrote: > What is content_type and why do I need to bother about this when storing a > blob of text? > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: undefined method content_type
Anand,
I rewrote your script a little bit (you did some odd things in the initialize
method which isn't expected to return anything), and this way it works:
require 'riak'
Riak::Serializers["text/html"] = Riak::Serializers::TextPlain
class RiakClient
def initialize
super
end
def client
@client ||= Riak::Client.new(:nodes => [{:http_port => 8091},{:http_port
=>8092},{:http_port=>8093},{:http_port =>8094}])
end
def get_me(bucket, key)
obj = client.bucket(bucket).get(key)
puts obj.data
end
def put_me(bucket, key, data, content_type)
obj = client.bucket(bucket).get_or_new(key)
obj.content_type = content_type
obj.raw_data=data
obj.store
end
end
if __FILE__ == $0
my_client = RiakClient.new
my_client.put_me("doc", "index.html", "some data goes here", "text/html")
hash = my_client.get_me("doc", "index.html")
end
Cheers, Mathias
http://riakhandbook.com
On Monday, 25. June 2012 at 07:52, Anand Hegde wrote:
> require 'riak'
>
>
>
>
> class RiakClient < Riak::Client
> #attr_accessor :bucket
>
> def initialize(hosts="")
> return Riak::Client.new(:nodes => [{:http_port => 8091},{:http_port
> =>8092},{:http_port=>8093},{:http_port =>8094}])
> end
>
> def get_me(bucket, key)
> obj = self.bucket(bucket).get(key)
> puts obj.data
> end
>
> def put_me(bucket, key, data, content_type)
> obj=self.bucket(bucket).get_or_new(key)
> puts obj.class
> obj.content_type=content_type
> obj.raw_data=data
> obj.store
> end
> end
>
>
>
> if __FILE__ == $0
> my_client=RiakClient.new
> my_client.put_me("doc", "index.html", "some data goes here", "text/html")
> hash=my_client.get_me("doc", "index.html")
> end
>
>
> But I am getting the following errors -
>
> NilClass
> riak_client.rb:32:in `put_me': undefined method `content_type=' for
> nil:NilClass (NoMethodError)
> from riak_client.rb:42:in `'
>
>
> Do I have to import the RiakObject and RiakClient classes and all? and how do
> i do this?
> ___
> riak-users mailing list
> riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: map function->send arguments and store in bucket->key
Venki,
You can specify an argument in your map function and hand the argument's value
over when running the MapReduce request. The part in your JavaScript code is as
simple as adding a second and a third argument to your function like so, with
the third argument being the relevant one:
function map(value, keydata, args) {
}
Whatever ends up in args can be specified via the MapReduce API. Suppose you're
using the Ruby client, you can specify an argument for every map phase you add
to a request. Notice the :arg option in the last line:
results = Riak::MapReduce.new(client).
add("artists","Beatles").
map("function(value, keydata, args) {return [args]'}", :keep => true, :arg =>
{:paul => true).run
The hash in this example ends up as a JSON structure passed into the function,
properly deserialized into a JavaScript object before the call of course.
The same in pure HTTP using curl would look like this:
curl -v -d '{"inputs":[["artists","Beatles"]],
"query":[{"map":{"language":"javascript","source":"function(value, keydata,
args) {return [args];}","arg": {"paul": true}}}]}' -H "Content-Type:
application/json" http://127.0.0.1:8098/mapred
Cheers, Mathias
http://riakhandbook.com
On Tuesday, 3. July 2012 at 09:45, Venki Yedidha wrote:
> Hi All,
> I wrote a map function to retrieve statistics for a particular data.
> However, I want to make this function accessible for many other data
> retrievals. For this, I need to send arguments to my function written in
> javascript. I didn't find examples on sending arguments to map function..
> Please help me on above.
>
> Thanks,
> Venkatesh.Y
> ___
> riak-users mailing list
> riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Pass json as argument to mapreduce
Venki,
You don't have to serialize the argument as a JSON string, it can simply be
specified as a normal JSON data structure, just like the other data in the
MapReduce request:
curl -v -d '{"inputs":[["artists", "Beatles"]],
"query":[{"map":{"language":"javascript","source":"function(v, k, a) {return
[a];}","arg":{"paul": true}}}]}' -H "Content-Type: application/json"
http://127.0.0.1:8098/mapred
Cheers, Mathias
http://riakhandbook.com
On Wednesday, 4. July 2012 at 07:26, Venki Yedidha wrote:
> Hi all,
> I don't know why, I can't pass a json argument to map function.
>
>
> "inputs":"[b1,k1],[b2,k2],[b3,k3]"
> "query":[{map":{"language":"javascript","source":"function(value,keydata,args){}"
>
> for the args parameter I need to pass json of the form:
> {\"name\":\"venkatesh\",\"age\":24}
> (escaped " for json)
> but it showing missing variable name..
> On the other, If I am passing an array, it is not showing the error...
> Please help me on the above.
>
> Thanks,
> Venkatesh
> ___
> riak-users mailing list
> riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Pass json as argument to mapreduce
Venki,
The parameter name is arg not args. Also, you were talking about some sort of
error you're getting, but I couldn't see one in the listing below.
Cheers, Mathias
http://riakhandbook.com
On Thursday, 5. July 2012 at 16:29, Venki Yedidha wrote:
> Thanks,
> I tried the format you sent me earlier but when I tried to access the args
> parameter inside the function, it is showing me the error:
>
> {"inputs":['.$k.'],"query":[{"map":{"language":"javascript","source":"function(value,keydata,"arguments"){
> var data = Riak.mapValuesJson(value)[0];
> var obj = {};
> obj[value.key] = {};
> for(var i=1;i<3;i++)
> {
> var gender = \"G\"+ i;
> obj[value.key][gender] = {};
> for(var j=1;j<=6;j++)
> {
> var agegroup = \"A\" + j;
> obj[value.key][gender][agegroup] = {};
> obj[value.key][gender][agegroup][\"cpc_median\"] =
> data[gender][j][agegroup][\"bid_estimations\"][0][\"cpc_median\"];
> obj[value.key][gender][agegroup][\"users\"] =
> data[gender][j][agegroup][\"users\"];
>
> }
> }
> return [obj];
> }","args":{"gender":"G0,G1","agegroups":"A0,A2","metrics":"users,cpc_median","groupby":"country"},"keep":true}}]}';
>
>
> Please help me on the above.
>
> To clarify my requirement, I would like to access the args parameter (before
> keep parameter) inside javascript function and if possible I would also like
> to know the flow that what is the need for args param after js function.
>
> Thanks,
> Venkatesh.Y
>
> On Wed, Jul 4, 2012 at 2:11 PM, Mathias Meyer (mailto:me...@paperplanes.de)> wrote:
> > Venki,
> >
> > You don't have to serialize the argument as a JSON string, it can simply be
> > specified as a normal JSON data structure, just like the other data in the
> > MapReduce request:
> >
> > curl -v -d '{"inputs":[["artists", "Beatles"]],
> > "query":[{"map":{"language":"javascript","source":"function(v, k, a)
> > {return [a];}","arg":{"paul": true}}}]}' -H "Content-Type:
> > application/json" http://127.0.0.1:8098/mapred
> >
> > Cheers, Mathias
> > http://riakhandbook.com
> >
> >
> >
> >
> >
> > On Wednesday, 4. July 2012 at 07:26, Venki Yedidha wrote:
> >
> > > Hi all,
> > > I don't know why, I can't pass a json argument to map function.
> > >
> > >
> > > "inputs":"[b1,k1],[b2,k2],[b3,k3]"
> > > "query":[{map":{"language":"javascript","source":"function(value,keydata,args){}"
> > >
> > > for the args parameter I need to pass json of the form:
> > > {\"name\":\"venkatesh\",\"age\":24}
> > > (escaped " for json)
> > > but it showing missing variable name..
> > > On the other, If I am passing an array, it is not showing the error...
> > > Please help me on the above.
> > >
> > > Thanks,
> > > Venkatesh
> > > ___
> > > riak-users mailing list
> > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> > > (mailto:riak-users@lists.basho.com)
> > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Pass json as argument to mapreduce
Venki,
>From what I gather you're trying to specify arguments directly as objects in
>the function's argument list? You reference the arg that's passed into the
>MapReduce request by declaring a third parameter variable, not by specifying
>an object: function(value, keydata, arg) {doSomethingWith(arg)}. Then you can
>reference the argument by way of that parameter.
Cheers, Mathias
http://riakhandbook.com
On Friday, 6. July 2012 at 08:40, Venki Yedidha wrote:
> Thanks again,
> I changed args to arg.
> To keep it simple, I just want to use that arg parameter value
> {"gender":"G0,G1","
> agegroups":"A0,A2","metrics":"users,cpc_median","groupby":"country"} inside
> my javascript function.
>
> If I try to assign it to a variable like:
>
> var k = arg;
>
> It is giving undefined variable arg.
>
> On the other hand,
> If I am trying to access this variable passing as a third parameter to
> function(v,k,{"",""})
> It is also giving me error.
>
> Please help me on the above.
>
> Thanks,
> Venkatesh
>
>
> On Thu, Jul 5, 2012 at 9:07 PM, Mathias Meyer (mailto:me...@paperplanes.de)> wrote:
> > Venki,
> >
> > The parameter name is arg not args. Also, you were talking about some sort
> > of error you're getting, but I couldn't see one in the listing below.
> >
> > Cheers, Mathias
> > http://riakhandbook.com
> >
> >
> >
> >
> >
> > On Thursday, 5. July 2012 at 16:29, Venki Yedidha wrote:
> >
> > > Thanks,
> > > I tried the format you sent me earlier but when I tried to access the
> > > args parameter inside the function, it is showing me the error:
> > >
> > > {"inputs":['.$k.'],"query":[{"map":{"language":"javascript","source":"function(value,keydata,"arguments"){
> > > var data = Riak.mapValuesJson(value)[0];
> > > var obj = {};
> > > obj[value.key] = {};
> > > for(var i=1;i<3;i++)
> > > {
> > > var gender = \"G\"+ i;
> > > obj[value.key][gender] = {};
> > > for(var j=1;j<=6;j++)
> > > {
> > > var agegroup = \"A\" + j;
> > > obj[value.key][gender][agegroup] = {};
> > > obj[value.key][gender][agegroup][\"cpc_median\"] =
> > > data[gender][j][agegroup][\"bid_estimations\"][0][\"cpc_median\"];
> > > obj[value.key][gender][agegroup][\"users\"] =
> > > data[gender][j][agegroup][\"users\"];
> > >
> > > }
> > > }
> > > return [obj];
> > > }","args":{"gender":"G0,G1","agegroups":"A0,A2","metrics":"users,cpc_median","groupby":"country"},"keep":true}}]}';
> > >
> > > Please help me on the above.
> > >
> > > To clarify my requirement, I would like to access the args parameter
> > > (before keep parameter) inside javascript function and if possible I
> > > would also like to know the flow that what is the need for args param
> > > after js function.
> > >
> > > Thanks,
> > > Venkatesh.Y
> > >
> > > On Wed, Jul 4, 2012 at 2:11 PM, Mathias Meyer > > (mailto:me...@paperplanes.de) (mailto:me...@paperplanes.de)> wrote:
> > > > Venki,
> > > >
> > > > You don't have to serialize the argument as a JSON string, it can
> > > > simply be specified as a normal JSON data structure, just like the
> > > > other data in the MapReduce request:
> > > >
> > > > curl -v -d '{"inputs":[["artists", "Beatles"]],
> > > > "query":[{"map":{"language":"javascript","source":"function(v, k, a)
> > > > {return [a];}","arg":{"paul": true}}}]}' -H "Content-Type:
> > > > application/json" http://127.0.0.1:8098/mapred
> > > >
> > > > Cheers, Mathias
> > > > http://riakhandbook.com
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wednesday, 4. July 2012 at 07:26, Venki Yedidha wrote:
> > > >
> > > > > Hi all,
> > > > > I don't know why, I can't pass a json argument to map function.
> > > > >
> > > > >
> > > > > "inputs":"[b1,k1],[b2,k2],[b3,k3]"
> > > > > "query":[{map":{"language":"javascript","source":"function(value,keydata,args){}"
> > > > >
> > > > > for the args parameter I need to pass json of the form:
> > > > > {\"name\":\"venkatesh\",\"age\":24}
> > > > > (escaped " for json)
> > > > > but it showing missing variable name..
> > > > > On the other, If I am passing an array, it is not showing the error...
> > > > > Please help me on the above.
> > > > >
> > > > > Thanks,
> > > > > Venkatesh
> > > > > ___
> > > > > riak-users mailing list
> > > > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> > > > > (mailto:riak-users@lists.basho.com)
> > > > > (mailto:riak-users@lists.basho.com)
> > > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> > > >
> > >
> >
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak client for node.js
The drop of PB in the new js branch is for now just temporary. Allows us to focus on getting the JavaScript code base up to speed at least feature-wise and get a feeling for how things could evolve for its API. Help on getting PB support back into that branch is more than welcome! But I'll also be happy to look into it myself once the code in general is up to speed with the newest Riak feature set, for which I unfortunately don't have a specific timeline right now, as I'm still digging through the code, adding tests, completing features, etc. Cheers, Mathias -- http://riakhandbook.com On Wednesday, 26. September 2012 at 20:36, Sebastian Cohnen wrote: > The drop of Protocol Buffers (according to the readme) is a sad thing. AFAIK > there is not a single usable Riak client lib for node that supports PB :-/D > > > On 26.09.2012, at 20:25, Mark Phillips (mailto:m...@basho.com)> wrote: > > > Hi Callixte - > > > > On Wed, Sep 26, 2012 at 12:51 PM, Callixte Cauchois > > mailto:ccauch...@virtuoz.com)> wrote: > > > Hello, > > > > > > what would you guys advise me to use to connect to riak in node.js? > > > Basho seems to push for Voxer's node_riak > > > (https://github.com/mranney/node_riak), but are there better alternatives? > > > > > > > > In addition to node_riak, Mathias Meyer (with some help from Sean > > Cribbs) has been putting in some work on our fork of riak-js (working > > primary to get it working with current Riak and finish Francisco's > > port to JS). > > > > https://github.com/basho-labs/riak-js/tree/js > > > > Perhaps Mathias or Sean can shed some light on when it might be up to > > date with 1.2? > > > > Mark > > > > > > > > Thanks. > > > C. > > > > > > ___ > > > riak-users mailing list > > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > > > > ___ > > riak-users mailing list > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak client for node.js
Callixte,
You run the following:
db.save('bucket', 'key', {index: {indexkey: 'value'}});
Cheers, Mathias
--
http://riakhandbook.com
On Wednesday, 26. September 2012 at 23:51, Callixte Cauchois wrote:
> Having a Basho backed official client for node.js sounds promising. I'll give
> it a try and test it. For now, I have one question:
> I didn't completely get how to set 2i. Do you put something like { indexkey:
> value } in the option field when calling save? it seems from the code that
> the x-riak-index- and the _bin or _int are appended.
>
> Thanks.
> C.
>
> On Wed, Sep 26, 2012 at 2:21 PM, Mathias Meyer (mailto:me...@paperplanes.de)> wrote:
> > The drop of PB in the new js branch is for now just temporary. Allows us to
> > focus on getting the JavaScript code base up to speed at least feature-wise
> > and get a feeling for how things could evolve for its API. Help on getting
> > PB support back into that branch is more than welcome! But I'll also be
> > happy to look into it myself once the code in general is up to speed with
> > the newest Riak feature set, for which I unfortunately don't have a
> > specific timeline right now, as I'm still digging through the code, adding
> > tests, completing features, etc.
> >
> > Cheers, Mathias
> > --
> > http://riakhandbook.com
> >
> >
> >
> >
> >
> > On Wednesday, 26. September 2012 at 20:36, Sebastian Cohnen wrote:
> >
> > > The drop of Protocol Buffers (according to the readme) is a sad thing.
> > > AFAIK there is not a single usable Riak client lib for node that supports
> > > PB :-/D
> > >
> > >
> > > On 26.09.2012, at 20:25, Mark Phillips > > (mailto:m...@basho.com) (mailto:m...@basho.com)> wrote:
> > >
> > > > Hi Callixte -
> > > >
> > > > On Wed, Sep 26, 2012 at 12:51 PM, Callixte Cauchois
> > > > mailto:ccauch...@virtuoz.com)
> > > > (mailto:ccauch...@virtuoz.com)> wrote:
> > > > > Hello,
> > > > >
> > > > > what would you guys advise me to use to connect to riak in node.js?
> > > > > Basho seems to push for Voxer's node_riak
> > > > > (https://github.com/mranney/node_riak), but are there better
> > > > > alternatives?
> > > >
> > > >
> > > >
> > > > In addition to node_riak, Mathias Meyer (with some help from Sean
> > > > Cribbs) has been putting in some work on our fork of riak-js (working
> > > > primary to get it working with current Riak and finish Francisco's
> > > > port to JS).
> > > >
> > > > https://github.com/basho-labs/riak-js/tree/js
> > > >
> > > > Perhaps Mathias or Sean can shed some light on when it might be up to
> > > > date with 1.2?
> > > >
> > > > Mark
> > > >
> > > > >
> > > > > Thanks.
> > > > > C.
> > > > >
> > > > > ___
> > > > > riak-users mailing list
> > > > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> > > > > (mailto:riak-users@lists.basho.com)
> > > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> > > >
> > > >
> > > >
> > > > ___
> > > > riak-users mailing list
> > > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> > > > (mailto:riak-users@lists.basho.com)
> > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> > >
> > >
> > >
> > >
> > > ___
> > > riak-users mailing list
> > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> > > (mailto:riak-users@lists.basho.com)
> > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
> >
> >
> >
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
riak-js pre-release announcement/warning
Hey folks,
I've been putting some work in the riak-js client for Node.js recently, and as
we're getting pretty close to doing a release, I wanted to keep you posted on
changes and especially breaking changes.
About a year ago, Francisco Treacy started rewriting the code (originally
written in CoffeeScript) in pure JavaScript. First up, thanks for all the great
work, Frank!
After he sought a new maintainer for the library earlier this year, the fine
folks of mostlyserious.io took over maintainership [1], and I eventually hopped
on board and started hacking on the new JavaScript implementation too, to
finish off missing bits [2].
Along the way, the domain riakjs.org unfortunately expired, but the website
found a new home at http://riakjs.com, thanks to Vincent Mac [3].
The good news is that things are looking pretty good, and pretty much ready to
be shipped, pending some updates on documentation, which I'll get to this week.
The slightly bad news (only slightly, really) is that there are breaking
changes.
As the client hasn't officially reached v1.0 yet, I figured it's still a good
opportunity to move things around, make things more explicit and cleaner.
The basic APIs, though, are still the same or at least very similar. The most
important changes affect Riak Search and MapReduce.
Riak Search is now fully supported, which means documents (i.e. JavaScript
objects) can be added and removed directly without the need to store them in
Riak. The functionality is namespace in the client, e.g. to find documents, you
can use the following snippet:
db.search.find("users", "email:me...@paperplanes.de")
Same for adding and removing documents:
db.search.add("users", {id: "mathias", email: "me...@paperplanes.de"}) or
db.search.add("users", [{id: "roidrage", email: "me...@paperplanes.de"}, {id:
"mathias", email: "math...@travis-ci.org"}])
db.search.remove("users", [{id: "mathias"}, {query:
'email:"me...@paperplanes.de"'}])
MapReduce was also moved to its own namespace, with the API being pretty much
the same as in the CoffeeScript client:
db.mapreduce.add("users").map("Riak.mapValuesJson").run()
Moving these out into separate namespaces is mostly a matter of clarity and
separating concerns, drawing some inspiration from other Riak clients.
There are other changes, e.g. regarding bucket properties (now db.getBucket()
and db.saveBucket()), Luwak (getFile(), removeFile(), saveFile()), though Luwak
would also be a candidate to be moved into a different namespace and having to
be explicitly enabled because it's been deprecated and is not included in the
standard distribution anymore.
So we're close to doing a release, but I just wanted to make sure information
on what things might break is out before we do. If you rely strongly on
existing APIs, make sure to pin your package.json to 0.4.1.
The next release might still be a bit rough around the edge, and it'll be
missing support for Protocol Buffers, but one step at a time :)
Let me know if you have any suggestions or feedback!
Cheers, Mathias
[1] https://github.com/mostlyserious/riak-js
[2] https://github.com/basho-labs/riak-js
[3] https://github.com/vincentmac/riakjs.com
--
http://riakhandbook.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
ANN: riak-js 0.9.0 released
Hey all, I'm happy to announce the 0.9.0 release of riak-js, the Riak client for Node.js. It's a complete rewrite in plain old JavaScript, bringing some new functionality along the way. You can read all about the fresh release over on the Basho blog [1]. riak-js now has a new home [2] and fully updated documentation [3]. Let me know if you run into any issues or have any questions! npm install riak-js@0.9.0 and off you go! Happy JavaScripting! Cheers, Mathias [1] http://basho.com/blog/technical/2012/11/13/riak-js-fresh-start/ [2] https://github.com/mostlyserious/riak-js [3] http://mostlyserious.github.com/riak-js/ ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: ANN: riak-js 0.9.0 released
I mistyped that indeed. The correct name is indeed riak-js on npmjs. Sorry! Cheers, Mathias On Tuesday, 13. November 2012 at 21:26, Christopher Meiklejohn wrote: > On Tuesday, November 13, 2012 at 3:24 PM, Alexander Sicular wrote: > > Are you published in npm? > > > > npm install riak is for the mranney version. > It appears you mistyped the library name: > > https://npmjs.org/package/riak-js > > npm install riak-js > > - Chris ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
ANN: riak-js 0.9.1 and Riak Handbook 30% off
Hey guys,
A quick one from me. Just shipped riak-js 0.9.1 with some fixes and some neat
additions, most notably request instrumentation, useful to e.g. track metrics
for request times. Here's a simple example that tracks request times per
request method:
var instrument = {
'riak.request.end': function(event) {
var runtime = event.finished_at - event.started_at;
metrics.timing('riak.request.' + event.method.toLowerCase(), runtime);
}
}
db.registerListener(instrument);
Full changelog is available in the README:
https://github.com/mostlyserious/riak-js#091
On a totally unrelated note, the Riak Handbook is 30% off until Monday for the
Thanksgiving Weekend, just use coupon code "gobble" during checkout:
http://riakhandbook.com :)
Have a great weekend, and don't forget to `npm install riak-js@0.9.1`
Cheers, Mathias
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
[ANN] riak-js 0.9.3 and 0.10.0pre1 released
Hey everyone, Just shipped two releases of riak-js today. 0.9.3 [1] is a minor bug fix release. It should now be installable on Windows too, because there's really no reason it shoudn't. The 0.10.0pre1 release brings preliminary support for Protocol Buffers, based on Nathan LaFreniere riakpbc library [2]. It's based on an ongoing pull request [3], which I'm working on completing. While still not fully complete (streaming keys and mapreduce are still pending, but I'm close to having those in there), I wanted to throw it out for testing. So please have at it, and report oddities [4] :) Cheers, Mathias [1] https://github.com/mostlyserious/riak-js/tree/v0.9.3 [2] https://github.com/nathan-lafreniere/riakpbc [3] https://github.com/mostlyserious/riak-js/pull/138 [4] https://github.com/mostlyserious/riak-js/issues ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
[ANN] riak-js v0.10.0
Heya,
Just shipped riak-js 0.10.0, with support for Protocol Buffers:
https://npmjs.org/package/riak-js
To use protobufs, specify a different API option when creating a client:
var riak = require('riak-js').getClient({api: 'protobuf'})
I'd love to have some feedback on how the protobuffs support works with real
workloads, if anyone feels like trying it out. Please report any issues you're
seeing: https://github.com/mostlyserious/riak-js/issues
Currently it's still using a single connection, but I'll work on adding
connection pooling next. Already merged Andrew J. Stone's support for pooled
HTTP connection into the current master, will work off that for protobuffs as
well.
Thanks to Nathan LaFreniere for his awesome riakpbc and protobuf.js libraries,
which helped me a lot!
Cheers, Mathias
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: [ANN] riak-js v0.10.0
Sebastian,
I have yet to look into that, but if you'd be willing to share your hack, maybe
I could integrate that directly into riak-js? :)
Thanks!
Cheers, Mathias
On Friday, 3. May 2013 at 18:00, Sebastian Cohnen wrote:
> Hey Mathias,
>
> this is awesome news, thanks for your work! I'll see what I can do to test
> the new version against production traffic.
>
> One question though: I don't know the work of Andrew, but do you support real
> HTTP keep alive? (not node.js' pseudo keep alive). I had to implement my own
> http agent and inject it into a previous version of riak-js to get useful
> performance. I'd like to drop that hack.
>
>
> Best
>
> Sebastian
>
> On 03.05.2013, at 17:55, Mathias Meyer (mailto:me...@paperplanes.de)> wrote:
>
> > Heya,
> >
> > Just shipped riak-js 0.10.0, with support for Protocol Buffers:
> > https://npmjs.org/package/riak-js
> >
> > To use protobufs, specify a different API option when creating a client:
> >
> > var riak = require('riak-js').getClient({api: 'protobuf'})
> >
> > I'd love to have some feedback on how the protobuffs support works with
> > real workloads, if anyone feels like trying it out. Please report any
> > issues you're seeing: https://github.com/mostlyserious/riak-js/issues
> >
> > Currently it's still using a single connection, but I'll work on adding
> > connection pooling next. Already merged Andrew J. Stone's support for
> > pooled HTTP connection into the current master, will work off that for
> > protobuffs as well.
> >
> > Thanks to Nathan LaFreniere for his awesome riakpbc and protobuf.js
> > libraries, which helped me a lot!
> >
> > Cheers, Mathias
> >
> >
> >
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: [ANN] riak-js v0.10.0
Sebastian,
Is the code that includes the keep alive agent private? Would certainly love to
include one as a default, if you'd like to contribute it.
Cheers, Mathias
On Friday, 3. May 2013 at 18:51, Sebastian Cohnen wrote:
> Sure, see https://gist.github.com/tisba/5511106
>
> The agent is based on the keep alive capable agent included in
> https://github.com/mikeal/request. In a more recent version this was
> extracted to https://github.com/mikeal/forever-agent, but I haven't used it
> since.
>
> On 03.05.2013, at 18:44, Mathias Meyer (mailto:me...@paperplanes.de)> wrote:
> > Sebastian,
> >
> > I have yet to look into that, but if you'd be willing to share your hack,
> > maybe I could integrate that directly into riak-js? :)
> >
> > Thanks!
> >
> > Cheers, Mathias
> >
> >
> >
> >
> > On Friday, 3. May 2013 at 18:00, Sebastian Cohnen wrote:
> >
> > > Hey Mathias,
> > >
> > > this is awesome news, thanks for your work! I'll see what I can do to
> > > test the new version against production traffic.
> > >
> > > One question though: I don't know the work of Andrew, but do you support
> > > real HTTP keep alive? (not node.js' pseudo keep alive). I had to
> > > implement my own http agent and inject it into a previous version of
> > > riak-js to get useful performance. I'd like to drop that hack.
> > >
> > >
> > > Best
> > >
> > > Sebastian
> > >
> > > On 03.05.2013, at 17:55, Mathias Meyer > > (mailto:me...@paperplanes.de)> wrote:
> > >
> > > > Heya,
> > > >
> > > > Just shipped riak-js 0.10.0, with support for Protocol Buffers:
> > > > https://npmjs.org/package/riak-js
> > > >
> > > > To use protobufs, specify a different API option when creating a client:
> > > >
> > > > var riak = require('riak-js').getClient({api: 'protobuf'})
> > > >
> > > > I'd love to have some feedback on how the protobuffs support works with
> > > > real workloads, if anyone feels like trying it out. Please report any
> > > > issues you're seeing: https://github.com/mostlyserious/riak-js/issues
> > > >
> > > > Currently it's still using a single connection, but I'll work on adding
> > > > connection pooling next. Already merged Andrew J. Stone's support for
> > > > pooled HTTP connection into the current master, will work off that for
> > > > protobuffs as well.
> > > >
> > > > Thanks to Nathan LaFreniere for his awesome riakpbc and protobuf.js
> > > > libraries, which helped me a lot!
> > > >
> > > > Cheers, Mathias
> > > >
> > > >
> > > >
> > > > ___
> > > > riak-users mailing list
> > > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> > > >
> > >
> > >
> >
> >
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: [ANN] riak-js v0.10.0
José,
Could you elaborate a bit more on that? Not sure I fully understand what you
mean.
Thanks!
Cheers, Mathias
On Friday, 3. May 2013 at 21:10, José Ghislain Quenum wrote:
> thanks again for the update. the wtf8.node you point at is not there (no
> Release folder). I pointed to the Debug one and it is fine. Maybe you wanna
> fix it
> José
> On May 3, 2013, at 5:55 PM, Mathias Meyer (mailto:me...@paperplanes.de)> wrote:
>
> > Heya,
> >
> > Just shipped riak-js 0.10.0, with support for Protocol Buffers:
> > https://npmjs.org/package/riak-js
> >
> > To use protobufs, specify a different API option when creating a client:
> >
> > var riak = require('riak-js').getClient({api: 'protobuf'})
> >
> > I'd love to have some feedback on how the protobuffs support works with
> > real workloads, if anyone feels like trying it out. Please report any
> > issues you're seeing: https://github.com/mostlyserious/riak-js/issues
> >
> > Currently it's still using a single connection, but I'll work on adding
> > connection pooling next. Already merged Andrew J. Stone's support for
> > pooled HTTP connection into the current master, will work off that for
> > protobuffs as well.
> >
> > Thanks to Nathan LaFreniere for his awesome riakpbc and protobuf.js
> > libraries, which helped me a lot!
> >
> > Cheers, Mathias
> >
> >
> >
> > ___
> > riak-users mailing list
> > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> >
>
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: [ANN] riak-js v0.10.0
José,
Do you use any debug flags for npm of sorts?
I just tried a fresh install, and npm compiled the library into the Release
folder.
Cheers, Mathias
On Saturday, 4. May 2013 at 12:01, José Ghislain Quenum wrote:
> in node_modules in my project inside
> riak-js/niode_modules/protobuf.js/node_modules/wtf8/wtf8.js
>
> var wtf8 = require('./build/Release/wtf8.node');
>
> but there is no Release folder. Only Debug
>
>
> On May 3, 2013, at 9:15 PM, Mathias Meyer (mailto:me...@paperplanes.de)> wrote:
> > José,
> >
> > Could you elaborate a bit more on that? Not sure I fully understand what
> > you mean.
> >
> > Thanks!
> >
> > Cheers, Mathias
> >
> >
> >
> >
> > On Friday, 3. May 2013 at 21:10, José Ghislain Quenum wrote:
> >
> > > thanks again for the update. the wtf8.node you point at is not there (no
> > > Release folder). I pointed to the Debug one and it is fine. Maybe you
> > > wanna fix it
> > > José
> > > On May 3, 2013, at 5:55 PM, Mathias Meyer > > (mailto:me...@paperplanes.de)> wrote:
> > >
> > > > Heya,
> > > >
> > > > Just shipped riak-js 0.10.0, with support for Protocol Buffers:
> > > > https://npmjs.org/package/riak-js
> > > >
> > > > To use protobufs, specify a different API option when creating a client:
> > > >
> > > > var riak = require('riak-js').getClient({api: 'protobuf'})
> > > >
> > > > I'd love to have some feedback on how the protobuffs support works with
> > > > real workloads, if anyone feels like trying it out. Please report any
> > > > issues you're seeing: https://github.com/mostlyserious/riak-js/issues
> > > >
> > > > Currently it's still using a single connection, but I'll work on adding
> > > > connection pooling next. Already merged Andrew J. Stone's support for
> > > > pooled HTTP connection into the current master, will work off that for
> > > > protobuffs as well.
> > > >
> > > > Thanks to Nathan LaFreniere for his awesome riakpbc and protobuf.js
> > > > libraries, which helped me a lot!
> > > >
> > > > Cheers, Mathias
> > > >
> > > >
> > > >
> > > > ___
> > > > riak-users mailing list
> > > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> > > >
> > >
> > >
> >
> >
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: [ANN] riak-js v0.10.0
José,
Could you file an issue [1] describing the problems you've been seeing?
Thank you!
Cheers, Mathias
[1] https://github.com/mostlyserious/riak-js/issues/new
On Sunday, 5. May 2013 at 17:24, José Ghislain Quenum wrote:
> is this just me or has anyone else experienced some weird behavior trying to
> save data into riak using riak-js? My updates are all of a sudden misbehaving.
> I upgraded to 0.10.0 since friday night
>
> On May 4, 2013, at 2:43 PM, José Ghislain Quenum (mailto:jose.que...@googlemail.com)> wrote:
> > Hi Mathias,
> > I didn't use any debug flag. I just did it again and it still created a
> > Debug folder instead of Release. Maybe it's my node_gyp> I need to check.
> > Has anyone else experienced this?
> > BTW I am using OS X Mountain Lion
> > José
> > On May 4, 2013, at 12:24 PM, Mathias Meyer > (mailto:me...@paperplanes.de)> wrote:
> > > José,
> > >
> > > Do you use any debug flags for npm of sorts?
> > >
> > > I just tried a fresh install, and npm compiled the library into the
> > > Release folder.
> > >
> > > Cheers, Mathias
> > >
> > >
> > >
> > >
> > > On Saturday, 4. May 2013 at 12:01, José Ghislain Quenum wrote:
> > >
> > > > in node_modules in my project inside
> > > > riak-js/niode_modules/protobuf.js/node_modules/wtf8/wtf8.js
> > > >
> > > > var wtf8 = require('./build/Release/wtf8.node');
> > > >
> > > > but there is no Release folder. Only Debug
> > > >
> > > >
> > > > On May 3, 2013, at 9:15 PM, Mathias Meyer > > > (mailto:me...@paperplanes.de)> wrote:
> > > > > José,
> > > > >
> > > > > Could you elaborate a bit more on that? Not sure I fully understand
> > > > > what you mean.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Cheers, Mathias
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Friday, 3. May 2013 at 21:10, José Ghislain Quenum wrote:
> > > > >
> > > > > > thanks again for the update. the wtf8.node you point at is not
> > > > > > there (no Release folder). I pointed to the Debug one and it is
> > > > > > fine. Maybe you wanna fix it
> > > > > > José
> > > > > > On May 3, 2013, at 5:55 PM, Mathias Meyer > > > > > (mailto:me...@paperplanes.de)> wrote:
> > > > > >
> > > > > > > Heya,
> > > > > > >
> > > > > > > Just shipped riak-js 0.10.0, with support for Protocol Buffers:
> > > > > > > https://npmjs.org/package/riak-js
> > > > > > >
> > > > > > > To use protobufs, specify a different API option when creating a
> > > > > > > client:
> > > > > > >
> > > > > > > var riak = require('riak-js').getClient({api: 'protobuf'})
> > > > > > >
> > > > > > > I'd love to have some feedback on how the protobuffs support
> > > > > > > works with real workloads, if anyone feels like trying it out.
> > > > > > > Please report any issues you're seeing:
> > > > > > > https://github.com/mostlyserious/riak-js/issues
> > > > > > >
> > > > > > > Currently it's still using a single connection, but I'll work on
> > > > > > > adding connection pooling next. Already merged Andrew J. Stone's
> > > > > > > support for pooled HTTP connection into the current master, will
> > > > > > > work off that for protobuffs as well.
> > > > > > >
> > > > > > > Thanks to Nathan LaFreniere for his awesome riakpbc and
> > > > > > > protobuf.js libraries, which helped me a lot!
> > > > > > >
> > > > > > > Cheers, Mathias
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > ___
> > > > > > > riak-users mailing list
> > > > > > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> > > > > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
[ANN] riak-js v0.10.1
Heya!
Hot on the heels of 0.10.0 I shipped riak-js 0.10.1 over the weekend, which
fixes a bug in the protobuffs code.
More importantly though, it now sports connection pooling/load balancing for
HTTP connections by way of the poolee library, courtesy of Andrew J. Stone.
When creating a client object you can now specify a list of servers to query:
var riak = require('riak-js').getClient({pool: {servers: ['127.0.0.1:8098',
'192.168.1.1:8098']}})
Conveniently, the poolee library also sports a custom agent for keep-alive
connections, which can even be used for single server connections:
var riak = require('riak-js').getClient({pool: {servers: ['127.0.0.1:8098'],
keepalive: true}})
I'll look into adding pooled connection support for Protocol Buffers next.
Cheers, Mathias___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: EC2 and RIAK
. But as always, it depends on your particular use case. Now, after all this realtalk, here's the kicker. Riak's way of replicating data can make both scenarios work. When it's ensured that your data is replicated on more than one node, it can work in both ways. You could use both ephemeral storage and be somewhat safe because data will reside on multiple nodes. The same is true for EBS volumes, as potential variances in I/O or even minutes of total unavailabilities (as seen on the recent Reddit outage) can be recovered a lot easier thanks to handoff and read repairs. You can increase the number of replicas (n_val) to increase your tolerance of instance failure, just make sure that n_val is less than the number of nodes in your cluster. Don't get me wrong, I love EC2 and EBS, being able to spin up servers at any time and to attache more storage to a running instance is extremely powerful, when you can handle the downsides. But if very low latency is what you're looking for, raw iron with lots of memory and SSD as storage device thrown on top is hard to beat. When in doubt, start with a RAID 0 setup on EBS with 4 volumes, and compare it with a RAID 5 in terms of performance. They're known to give a good enough performance in a lot of cases. If you decide to go with a RAID, be sure to add LVM on top for simpler snapshotting, which will be quite painful if not impossible to get consistent snapshots using just EBS snapshots on a bunch of striped volumes. Let us know if you have more questions, there's lots of details involved when you're going under the hood, but this should cover the most important bases. Mathias Meyer Developer Advocate, Basho Technologies [1] http://en.wikipedia.org/wiki/RAID#RAID_10_versus_RAID_5_in_Relational_Databases [2] http://blog.cloudharmony.com/2010/09/benchmarking-of-ec2s-new-cluster.html [3] http://blog.cloudharmony.com/2010/06/disk-io-benchmarking-in-cloud.html [4] http://blog.bioteam.net/2010/07/boot-ephemeral-ebs-storage-performance-on-amazon-cc1-4xlarge-instance-types/ [5] http://blog.bioteam.net/2010/07/local-storage-performance-of-aws-cluster-compute-instances/ [6] http://blog.bioteam.net/2010/07/preliminary-ebs-performance-tests-on-amazon-compute-cluster-cc1-4xlarge-instance-types/ [7] http://victortrac.com/EC2_Ephemeral_Disks_vs_EBS_Volumes On Mittwoch, 30. März 2011 at 18:29, David Dawson wrote: > I am not sure if this has already been discussed, but I am looking at the > feasibility of running RIAK in a EC2 cloud, as we have a requirement that may > require us to scale up and down quite considerably on a month by month basis. > After some initial testing and investigation we have come to the conclusion > that there are 2 solutions although both have their downsides in my opinion: > > 1. Run multiple cluster compute( cc1.4xlarge ) instances ( 23 GB RAM, 10 > Gigabit ethernet, 2 x 845 GB disks running RAID 0 ) > 2. Same as above but using EBS as the storage instead of the local disks. > > The problems I see are as follows with solution 1: > > - A instance failure results in complete loss of data on that machine, as the > disks are ephemeral storage ( e.g. they only exist whilst the machine is up ). > > The problems I see are as follows with solution 2: > > - EBS is slower than the local disks and from what I have read is susceptible > to latency depending on factors out of your control. > - There has been a bit of press lately about availability problems with EBS, > so we would have to use multiple availability zones although there are only 4 > in total and it just seems as though I am over complicating things. > > Has anyone used EC2 and RIAK in production and if so what are their > experiences? > > Otherwise has anyone used RackSpace or Joyent? as these are alternatives > although the Joyent solution seems very expensive, and what are their > experiences? > > Dave > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Strange results from numerical range query in Riak Search
Mike,
you need to make sure the followers field is marked as a numeric field in the
bucket's schema in Riak Search. You can do that by either relying on the
default schema and appending _num to the field name, so it'd be followers_num
instead of followers in the JSON you're generating.
Or you could set your own custom schema for the bucket, telling Riak Search to
treat the followers field as a numeric field. Both is detailed on our wiki:
http://wiki.basho.com/Riak-Search---Schema.html
Mathias Meyer
Developer Advocate, Basho Technologies
On Dienstag, 5. April 2011 at 03:14, Mike Stoddart wrote:
> I'm installing the lakes/celtics twitter data linked through Basho's
> site. I can get some basic text searches to work, but I'm seeing some
> weird results from numerical range queries:
>
> search_query = client.search('stuff', 'followers:[140 TO 150]')
>
> Name:Kevin Shpunt Term:lakers Followers:146
> Name:Carlos Ordonez Term:lakers Followers:15
> Name:Kelvin Shortt Jr Term:lakers,celtics Followers:140
> Name:Stanley T. Davis Term:lakers Followers:15
> Name:John Robertson Term:lakers Followers:141
> Name:Steven Wineglass Term:lakers Followers:1493
> Name:Badral Jambal Term:lakers Followers:1432
>
> Am I doing something wrong?
>
> Thanks
> Mike
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: QUERY: riaksearch
Hi,
Riak Search somewhat denormalizes nested structures in a JSON-ish structure,
which the PHP client creates from the associative array, unless you specify a
different content type. The dot notation below actually dereferences a Riak
Search index called contact1, not the nested property contact1.
You can query the separate fields by using an underscore instead, so you would
run this instead:
$client->search("bucket","contact_phone1:999")->run()
Mathias Meyer
Developer Advocate, Basho Technologies
On Mittwoch, 13. April 2011 at 10:23, khyqo wrote:
> good day everyone.. i am new to riak.. while i am doing some testing, i
> encountered a problem when retrieving data using riaksearch.. i am not
> sure if im doing it the right way.. i am using php (from
> https://github.com/basho/riak-php-client) to test..
>
> sample data:
>
> array(
> "fname"=>"test",
> "lname"=>"yadda",
> "contact"=>array(
> "phone1"=>"999",
> "phone2"=>"888"
> )
> )
>
>
> i was able to retrieve data using
>
> $client->search("bucket","fname:test")->run()
> $client->search("bucket","lname:yadda")->run()
>
> but the problem is that i cant get phone1 and/or phone2 values.. i tried
>
> $client->search("bucket","contact.phone1:999")->run()
> $client->search("bucket","contact->phone1:999")->run()
>
> but to no avail.. suggestions/recommendations would be highly appreciated..
>
> thank you in advance..
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Python feature suggestion - load (binary) content from file
I took the liberty of wrapping this into a pull request. The API corresponds to Ana's original suggestion, allowing bucket.new_binary_from_file(key, filename) Thanks for contributing! Mathias Meyer Developer Advocate, Basho Technologies On Dienstag, 12. April 2011 at 23:21, Ana Nelson wrote: > Hi, > > If Eric or someone has time to do this, that would be great. Otherwise I'll > try to put a pull request together this weekend. > > -Ana > > > On Tue, Apr 12, 2011 at 4:10 PM, Eric Moritz wrote: > > If you don't have this integrated into the Riak Python client, I'll be > > happy to do that for you. > > > > Thanks, > > Eric Moritz. > > > > On Sun, Apr 10, 2011 at 10:49 AM, Ana Nelson wrote: > > > I've just implemented this in a project: > > > > > > binary_data = open(self.filename(), "rb").read() > > > mimetype, encoding = mimetypes.guess_type(self.filename()) > > > if not mimetype: > > > > > > mimetype = 'application/octet-stream' # Riak's own default > > > self.riak_object = self.bucket.new_binary( > > > self.hashstring, > > > binary_data, > > > mimetype > > > > > > ) > > > > > > which writes binary data to a Riak bucket from a file, using Python's > > > mimetypes library to determine the content type from the file's extension. > > > > > > I think it could be convenient to have this be included in the Riak Python > > > client, so perhaps it can implement better buffering. Something like > > > bucket.new_binary_from_file(key, filename) > > > > > > ___ > > > riak-users mailing list > > > riak-users@lists.basho.com > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: RiakSearch + MapReduce integration
Hi there,
this wiki page sums it up pretty nicely:
http://wiki.basho.com/Riak-Search---Querying.html#Querying-Integrated-with-Map-Reduce.
It's just a different kind of input to MapReduce.
In PHP, you could simply do:
$result = $client->search("bucket", "this:that")
->map("function (v) { return [v.key]; }")
->reduce("Riak.reduceSort")
->run();
Mathias Meyer
Developer Advocate, Basho Technologies
On Donnerstag, 14. April 2011 at 20:20, khyqo wrote:
> good day everyone..
>
> i encountered another problem.. i am confused on how to integrate riaksearch
> and mapreduce.. according to
> http://blog.basho.com/2010/07/27/webinar-recap---mapreduce-querying-in-riak/
> (second question), a webinar would be released but i was not able to find
> one.. could anyone give me some points on doing this or any wikis or
> tutorials.. a php based tutorial would be the best..
>
> thank you so much..
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: python-riak-client: 3 unit tests failed
Mikhail,
thanks for bringing it to our attention, I've been noticing that one particular
error (...expected 4) myself, and I'm planning to look into and fix it asap.
As for the other two, they fail because they require Riak Search. You can
disable them by setting the environment variable SKIP_SEARCH=1.
Mathias Meyer
Developer Advocate, Basho Technologies
On Donnerstag, 21. April 2011 at 21:49, Mikhail Sobolev wrote:
Hi,
>
> (I'm not sure if this is a correct list for posting questions about
> python-riak-client. If it's not, please direct me to the correct
> forum.)
>
> I downloaded:
> * latest version of Riak (0.14.0-1 for Debian/32)
> * the latest master of python-riak-client (git clone)
>
> And run 'nosetests -v'. The outcome -- 3 failed test cases -- is a bit
> alarming (see below). Could anyone suggest what to look at?
>
> Kind Regards,
>
> --
> Misha
>
> PS Just in case, it's python 2.6.6 on Ubuntu 10.10.
>
> --- failed test cases ---
> ==
> ERROR: test_search_integration (riak.tests.test_all.RiakHttpTransportTestCase)
> --
> Traceback (most recent call last):
> File "/home/mss/projects/derek/riak-python-client/riak/tests/test_all.py",
> line 395, in test_search_integration
> results = self.client.search("searchbucket", "foo:one OR foo:two").run()
> File "/home/mss/projects/derek/riak-python-client/riak/mapreduce.py", line
> 185, in run
> result = t.mapred(self._inputs, query, timeout)
> File "/home/mss/projects/derek/riak-python-client/riak/transports/http.py",
> line 198, in mapred
> result = json.loads(response[1])
> File "/usr/lib/python2.6/json/__init__.py", line 307, in loads
> return _default_decoder.decode(s)
> File "/usr/lib/python2.6/json/decoder.py", line 319, in decode
> obj, end = self.raw_decode(s, idx=_w(s, 0).end())
> File "/usr/lib/python2.6/json/decoder.py", line 338, in raw_decode
> raise ValueError("No JSON object could be decoded")
> ValueError: No JSON object could be decoded
>
> ==
> ERROR: test_search_integration (riak.tests.test_all.RiakPbcTransportTestCase)
> --
> Traceback (most recent call last):
> File "/home/mss/projects/derek/riak-python-client/riak/tests/test_all.py",
> line 395, in test_search_integration
> results = self.client.search("searchbucket", "foo:one OR foo:two").run()
> File "/home/mss/projects/derek/riak-python-client/riak/mapreduce.py", line
> 185, in run
> result = t.mapred(self._inputs, query, timeout)
> File "/home/mss/projects/derek/riak-python-client/riak/transports/pbc.py",
> line 300, in mapred
> msg_code, resp = self.recv_msg()
> File "/home/mss/projects/derek/riak-python-client/riak/transports/pbc.py",
> line 349, in recv_msg
> self.recv_pkt()
> File "/home/mss/projects/derek/riak-python-client/riak/transports/pbc.py",
> line 390, in recv_pkt
> format(nmsglen))
> RiakError: Socket returned short packet length - expected 4
>
> ==
> ERROR: test_store_of_missing_object
> (riak.tests.test_all.RiakPbcTransportTestCase)
> --
> Traceback (most recent call last):
> File "/home/mss/projects/derek/riak-python-client/riak/tests/test_all.py",
> line 369, in test_store_of_missing_object
> o = o.store()
> File "/home/mss/projects/derek/riak-python-client/riak/riak_object.py", line
> 277, in store
> Result = t.put(self, w, dw, return_body)
> File "/home/mss/projects/derek/riak-python-client/riak/transports/pbc.py",
> line 190, in put
> msg_code, resp = self.recv_msg()
> File "/home/mss/projects/derek/riak-python-client/riak/transports/pbc.py",
> line 349, in recv_msg
> self.recv_pkt()
> File "/home/mss/projects/derek/riak-python-client/riak/transports/pbc.py",
> line 390, in recv_pkt
> format(nmsglen))
> RiakError: Socket returned short packet length - expected 4
>
> --
> Ran 54 tests in 1.526s
>
> FAILED (errors=3)
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Question regarding riak-js getAll() method using where option
Luc, that was indeed a bug in riak-js, but it has been fixed in a recent commit [1], which I just verified to give the expected results. Quickest workaround to get it working locally is to clone the repository, have CoffeeScript installed, and then run the following commands in the repository's directory: $ cake build $ npm install Mathias Meyer Developer Advocate, Basho Technologies [1] https://github.com/frank06/riak-js/commit/d65a2fc3ac227aeb3dc17bc5d7c703a4fcb8c232 On Donnerstag, 28. April 2011 at 17:41, Luc Castera wrote: Hi folks, > > I've encountered an issue while using riak-js today and I was wondering if > someone on the list could help me with it. > > The 'where' option does not seem to work for me. Below is some code that > shows the issue: > https://gist.github.com/947156 > > The second call to getAll here should not return any data since the database > has no memberships with 'mourinho' as the user_id but it does return data > anyway. > > What am I doing wrong here? > > I'll appreciate any help with this. > > Thanks, > > -- > __ > Luc Castera > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Buckets with different expiry_secs value
Dmitry,
the Protocol Buffer interface doesn't yet support setting most of the available
bucket properties. Support for doing that was added just recently though.
For now the easiest workaround is to just use the REST API, for which you'll
need the binaries. Both binaries and atoms work, but for the REST API it's best
to use binaries as it doesn't handle atoms.
Mathias Meyer
Developer Advocate, Basho Technologies
On Dienstag, 17. Mai 2011 at 08:58, Dmitry Demeshchuk wrote:
> Greetings.
>
> I'm looking for a way to set expiry_secs for specific buckets. For
> example, if some buckets need to have object expiration time and some
> buckets need to have unlimited lifetime.
>
> I've tried to do this using multi_backend:
>
> {storage_backend, riak_kv_multi_backend},
> {multi_backend_default, <<"default">>},
> {multi_backend, [
> {<<"default">>, riak_kv_bitcask_backend, []},
> {<<"cache">>, riak_kv_bitcask_backend, [
> {expiry_secs, 60},
> ]}
> ]},
>
> But when I call
>
> riakc_pb_socket:set_bucket(<<"test">>, [{backend, <<"cache">>}]),
>
> this setting isn't applied to the bucket. Tried to set different data
> directories for both backends – with the same result.
>
> Is such thing possible at all, and if it's possible, what am I doing wrong?
>
> Also, looks like riak_kv_multi_backend.erl has documentation a bit
> different with wiki. At wiki backends aliases are binaries, and at the
> module's docs they are atoms. Probably, both of them work, but neither
> of them worked for me :(
>
> --
> Best regards,
> Dmitry Demeshchuk
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: undifend error for js modules
Malka,
the most likely reason is that the JavaScript file is not properly accessible
to Riak on some of the nodes in your cluster.
Have you checked that the file is properly distributed throughout the cluster
and js_source_dir is set accordingly, and Riak is restarted on all nodes?
Mathias Meyer
Developer Advocate, Basho Technologies
On Donnerstag, 19. Mai 2011 at 08:21, Malka Feldman wrote:
> Hi,
> I have some modules with js functions for named MR.
> But for some keys, I get the message that my module is undefined:
> {"lineno":1,"message":"ReferenceError: GetTotal is not
> defined","source":"unknown"}
> If it's defined for some keys why it's not for the others (although they have
> same values)?
>
> Thanks,
> Malka Feldman
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak replication and quorum
Peter, wrote my replies inline. Mathias Meyer Developer Advocate, Basho Technologies On Freitag, 13. Mai 2011 at 20:05, Peter Fales wrote: > Sean, > > Thanks to you and Ben for clarifying how that works. Since that was > so helpful, I'll ask a followup question, and also a question on > a mostly un-related topic... > > 1) When I've removed a couple of nodes and the remaining nodes pick up > the slack, is there any way for me to look under the hood and see that? > I'm using wget to fetch the '.../stats' URL from one of the remaing > live nodes, and under ring_ownership it still lists the original 4 > nodes, each one owning 1/4 or the total partitions. That's part of > reason why I didn't think the data ownership had been moved. > Ring ownership is only affected by nodes explicitly entering and leaving the cluster. Unless you explicitly tell the cluster to remove a node, or explicitly tell that node to leave the cluster, ownership will remain the same even in case of a failure on one or more nodes. Data ownership is moved around implicitly in case of failure. By looking at the preference list, the coordinating node simply picks the next node(s) to pick up the slack for the failed one(s). The only way to find out if a handoff is currently happening between any two nodes is to look at the logs. They'll indicate beginning and end of a transfer. The cluster state and therefore the stats don't take re-partitioning or handoff into account yet. > 2) My test involves sending a large number of read/write requests to the > cluster from multiple client connections and timing how long each request > takes. I find that the vast majority of the requests are processed > quickly (a few milliseconds to 10s of milliseconds). However, every once > in while, the server seems to "hang" for a while. When that happens > the response can take several hundred milliseconds or even several > seconds. Is this something that is known and/or expected? There > doesn't seem to be any pattern to how often it happens -- typically > I'll see it a "few" times during a 10-minute test run. Sometimes > it will go for several minutes without a problem. I haven't ruled > out a problem with my test client, but it's fairly simple-minded C++ > program using the protocol buffers interface, so I don't think there > is too much that can go wrong on that end. > Easiest to find out if the problem is something stalling is to look at the stats and the percentiles for put and get fsms, which are responsible for taking care of reads and writes. Look for the JSON keys node_get_fsm_time_* and node_put_fsm_time_*. If anything jumps out here during and shortly after your benchmark run, something on the Riak or EC2 end is probably waiting for something else. Are you using EBS in any way for storing Riak's data? If so, what kind of setup do you have, single volume or RAID? ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Clarification on key filters
Jeremiah,
I sure hope you have not been drinking mouthwas.
The wiki is indeed showing single key filters, and that's confusing. In
hindsight, it confused me too when I worked with key filters from the wiki page
for the first time. I'll make sure we put some proper examples on that page to
clarify how they should end up looking when multiple are put together.
The bottom line is that Ripple does produce proper key filter code with
conditions and that you are absolutely correct in bringing up this slight
confusion.
Mathias Meyer
Developer Advocate, Basho Technologies
On Donnerstag, 26. Mai 2011 at 03:34, Jeremiah Peschka wrote:
> An MR job has 0 or more key filters. Adding a few transforms generates a
> lovely list of lists:
> { "inputs":{ "bucket":"invoices" "key_filters":[["tokenize", "-", 1],
> ["to_lower"], ["matches", "solutions"]] }, // ... }
> That makes sense: we've got a list of key filters. And a key filter is, in
> effect, a list of arguments.
>
> The complex predicates are throwing me off. Looking at the spec from Ripple
> in filter_builder_spec.rb [1] and it's showing that
>
> subject.OR do
> starts_with "foo"
> ends_with "bar"
> end
>
>
> becomes
>
> [[:or, [[:starts_with, "foo"],[:ends_with, "bar"
>
> Which is not at all the same as what the wiki says an OR should look like:
>
> ["or", [["eq", "google"]], [["less_than", "g"]]]
>
> Apart from the obvious difference in syntax, have I been drinking mouthwash
> or did the wiki suddenly switch from showing a complete key filter condition
> to showing an individual key filter command for AND, OR, and NOT?
>
> Jeremiah
> ___
> riak-users mailing list
> riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Old values on put call
Anthony,
currently there's no simple way to achieve what you're after. The API is
modelled around a GET before you PUT model.
You can specify the option return_body when putting new data into Riak, like so:
{ok, RObj} = riak_pb_socket:put (.., NewObject, [return_body]),
That returns the freshly written data in the result. It's not exactly what you
want, but could be used to achieve what you're after by deliberately creating
siblings, by e.g. leaving out the vector clock. Specifying the return_body flag
then will return the list of siblings created with this requests.
Be aware that abusing this technique with too many writes and not properly
reconciling siblings may cause your Riak objects to grow in an unhealthy
manner, and it's not exactly a recommended way of doing things.
It's an option but certainly not as simple as the one you're after, unless
you're prepared to deal with the potential conflicts, and e.g. handle siblings
immediately after you reconciled the differences between two objects in your
compare() function, see [1] for more details.
Mathias Meyer
Developer Advocate, Basho Technologies
[1] http://wiki.basho.com/Vector-Clocks.html
On Mittwoch, 18. Mai 2011 at 23:28, Anthony Molinaro wrote:
> Hi,
>
> I'm working on an application where I am storing an erlang external term
> format in riak. The entire structure gets updated at once, but I'd like to
> see what is changing, so I have something like this (in pseudo code).
>
> NewObject = construct_new(...),
> OldObject = riak_pb_socket:get(...),
> ok = riak_pb_socket:put (.., NewObject),
> compare (OldObject, NewObject),
>
> The idea being that I am updating the object everytime, but I would like
> to have a general idea what has changed.
>
> So I was wondering if there are any options for put to return the previous
> value? That would allow me to remove the call to get and simply do something
> like.
>
> NewObject = construct_new(...),
> OldObject = riak_pb_socket:put (.., NewObject),
> compare (OldObject, NewObject),
>
> Now I assume what I get back would depend a lot on what the w value is, but
> in most cases, I tend to use the defaults. Also, I would think the old
> values could be returned as a list in some cases where there was disagreement.
>
> Anyway, would something like this be hard to implement in riak itself
> (it's sort of a specialized use case, but I could see it being useful
> in cases like mine where you always want a put to succeed, but you might
> want to check what changed for tracking reasons, and I do understand that
> you won't be absolutely accurate all the time, but I mostly am looking for
> something scalable and mostly accurate).
>
> -Anthony
>
> --
>
> Anthony Molinaro (mailto:antho...@alumni.caltech.edu)>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: riaksearch: using index docs in place of real objects
Greg, Riak Search stores indexed documents in Riak KV too, as serialized Erlang terms. You can easily verify that by requesting a document from http://riak.host:8098/riak/_rsid_/key. So whenever you query something through the Solr interface the documents you get back are fetched from these buckets, and therefore the same distribution and consistency properties apply to them as to objects stored directly in Riak KV. Bottom line is there's nothing wrong with just using them instead of fetching them again from Riak KV. Mathias Meyer Developer Advocate, Basho Technologies On Mittwoch, 25. Mai 2011 at 00:34, Greg Pascale wrote: > Hi, > > In our data model, our riak objects are flat JSON objects, and thus their > corresponding index documents are nearly identical - the only difference is > that a few fields which are ints in the riak objects are strings in the index > doc. > > Since they are so similar, we are directly using the index docs returned from > our search call, skipping the second step of doing gets on the returned keys > to retrieve the real objects. > > Is this advisable? Are there any circumstances under which we might run into > consistency issues? > > Thanks, > -Greg > > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: riaksearch: using index docs in place of real objects
This behavior is specific to the Solr interface. It first fetches document IDs matching the criteria and then fetches the documents from Riak KV. Using the Erlang interface you can fetch just the IDs. It would certainly make sense to add an option like that, but it'd be inconsistent with Solr's interface. As far as I know (and I tried to confirm this in the dark depths of the Solr wiki) you can tell Solr to store specifc indexed fields, but not specify an option at query time to omit stored fields. Correct me if I'm wrong though, would be more than reason (and probably also simple) enough to add something like that to the Riak Search Solr API. Mathias Meyer Developer Advocate, Basho Technologies On Donnerstag, 26. Mai 2011 at 20:50, Greg Pascale wrote: > Thanks Mathias, > > We'll continue to do that then. > > It seems to me, though, that in the common case you aren't interested in the > index docs when you do a search, so it's needlessly inefficient to retrieve > them. Might it make sense to add a search option to not return the index docs > if you don't care about them? > > -Greg > > On Thu, May 26, 2011 at 6:42 AM, Mathias Meyer (mailto:math...@basho.com)> wrote: > > Greg, > > > > Riak Search stores indexed documents in Riak KV too, as serialized Erlang > > terms. You can easily verify that by requesting a document from > > http://riak.host:8098/riak/_rsid_/key. > > > > So whenever you query something through the Solr interface the documents > > you get back are fetched from these buckets, and therefore the same > > distribution and consistency properties apply to them as to objects stored > > directly in Riak KV. Bottom line is there's nothing wrong with just using > > them instead of fetching them again from Riak KV. > > > > Mathias Meyer > > Developer Advocate, Basho Technologies > > > > > > On Mittwoch, 25. Mai 2011 at 00:34, Greg Pascale wrote: > > > > > Hi, > > > > > > In our data model, our riak objects are flat JSON objects, and thus their > > > corresponding index documents are nearly identical - the only difference > > > is that a few fields which are ints in the riak objects are strings in > > > the index doc. > > > > > > Since they are so similar, we are directly using the index docs returned > > > from our search call, skipping the second step of doing gets on the > > > returned keys to retrieve the real objects. > > > > > > Is this advisable? Are there any circumstances under which we might run > > > into consistency issues? > > > > > > Thanks, > > > -Greg > > > > > > ___ > > > riak-users mailing list > > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > > > (mailto:riak-users@lists.basho.com) > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: riaksearch: using index docs in place of real objects
That is correct, Greg. It's either determined by the key used to store the object in Riak KV (given the precommit hook is used), or by a key specified when indexing directly into Riak Search, using e.g. the Solr or the Erlang API. There'll always be a key required, and that'll be used to look up the serialized document in Riak KV. Mathias Meyer Developer Advocate, Basho Technologies On Donnerstag, 26. Mai 2011 at 21:56, Greg Pascale wrote: > Eric, I believe the key is the document id, which will be the same as the key > of the corresponding object in . > > -Greg > > On Thu, May 26, 2011 at 12:41 PM, Eric Moritz (mailto:e...@themoritzfamily.com)> wrote: > > Out of curiosity what is the key in this URL? > > http://riak.host:8098/riak/_rsid_/key > > > > On Thu, May 26, 2011 at 9:42 AM, Mathias Meyer > (mailto:math...@basho.com)> wrote: > > > Greg, > > > > > > Riak Search stores indexed documents in Riak KV too, as serialized Erlang > > > terms. You can easily verify that by requesting a document from > > > http://riak.host:8098/riak/_rsid_/key. > > > > > > So whenever you query something through the Solr interface the documents > > > you get back are fetched from these buckets, and therefore the same > > > distribution and consistency properties apply to them as to objects > > > stored directly in Riak KV. Bottom line is there's nothing wrong with > > > just using them instead of fetching them again from Riak KV. > > > > > > Mathias Meyer > > > Developer Advocate, Basho Technologies > > > > > > > > > On Mittwoch, 25. Mai 2011 at 00:34, Greg Pascale wrote: > > > > > > > Hi, > > > > > > > > In our data model, our riak objects are flat JSON objects, and thus > > > > their corresponding index documents are nearly identical - the only > > > > difference is that a few fields which are ints in the riak objects are > > > > strings in the index doc. > > > > > > > > Since they are so similar, we are directly using the index docs > > > > returned from our search call, skipping the second step of doing gets > > > > on the returned keys to retrieve the real objects. > > > > > > > > Is this advisable? Are there any circumstances under which we might run > > > > into consistency issues? > > > > > > > > Thanks, > > > > -Greg > > > > > > > > ___ > > > > riak-users mailing list > > > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > > > > (mailto:riak-users@lists.basho.com) > > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > > > > > > > > > ___ > > > riak-users mailing list > > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > ___ > > riak-users mailing list > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Search with Erlang API
Hagbard,
what the Protobufs client (which I assume is what you're using) does to get
search results is to run a MapReduce job with the search as input.
You can easily use this to your advantage to achieve what you're after with a
custom map function, example to follow. When Riak Search results are fed into
the map phase, it receives the score in the keydata parameter, so you can just
use an anonymous function to return both the Riak object and the key data, or
extract the just score, as you see fit for your use case.
MapObjectKeydata = fun(RiakObject, KeyData, none) ->
[RiakObject, KeyData]
end.
Timeout = 10.riakc_pb_socket:search(Pid, "bucket", "query", [{map, {qfun,
MapObjectKeydata}, none, true}], Timeout).
Mathias Meyer
Developer Advocate, Basho Technologies
On Montag, 30. Mai 2011 at 12:37, Hagbard Celine wrote:
> Hi,
> im experimenting a little bit with Riak Search and want to query my node
> via Erlang API. But instead of the command line interface i got only a list
> of bucket/key pairs. Is there a way to get scoring information via Erlang
> API?
>
> Thanks
> Hagbard
> --
> Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir
> belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: The JSON of death
Sylvain, Bob is indeed correct. This list contains the character 153 which is a control character in Unicode, hence mochijson:decode() giving an error, causing the MapReduce process to crash. I couldn't reproduce a complete unavailability locally though, Riak (also 0.14.2) didn't seem to be bothered by that crash, the process was simply restarted. If you could send me some logs to peek at I could investigate the issue further. You need to make sure that the string you're storing in Riak is properly encoded for Unicode (i.e. resulting in \u1234 character sets) if your application and MapReduce code rely on that. If you need to enforce this before storing data inside Riak, you can use a pre-commit hook to validate the JSON using e.g. mochijson2:decode() and have your application respond accordingly, see [1] for an example. [1] https://gist.github.com/1031311 Mathias Meyer Developer Advocate, Basho Technologies On Freitag, 17. Juni 2011 at 03:09, Sylvain Niles wrote: > So we've been encountering some input that has json which is parsed > just fine by ruby but causes riak map reduce to die a horrible death. > > Running Riak 0.14.2 with json2.js updated to the latest from github > (thanks for that suggestion Dan) > > Steps to reproduce: > > bin/riak attach > > Badstuff = <<123,34,97,100,100,114,101,115, > 115,34,58,34,84,117,100,111, > 114,32,80,108,97,99,101,32,72, > 105,115,116,111,114,105,99,32, > 72,111,117,115,101,32,97,110, > 100,32,71,97,114,100,101,110, > 44,32,49,54,52,52,32,51,49,115, > 116,32,83,116,114,101,101,116, > 44,32,78,87,44,32,87,97,115, > 104,105,110,103,116,111,110,34, > 44,34,112,114,105,99,101,34,58, > 56,46,51,56,44,34,100,117,114, > 97,116,105,111,110,34,58,55,50, > 48,48,46,48,44,34,99,104,97, > 114,105,116,121,34,58,110,117, > 108,108,44,34,116,104,105,115, > 95,119,101,101,107,34,58,116, > 114,117,101,44,34,116,111,100, > 97,121,34,58,102,97,108,115, > 101,44,34,99,114,101,97,116, > 101,100,95,97,116,34,58,34,84, > 104,117,32,74,117,110,32,49,54, > 32,49,53,58,49,49,58,49,54,32, > 45,48,53,48,48,32,50,48,49,49, > 34,44,34,116,105,116,108,101, > 34,58,34,70,97,116,104,101,114, > 153,115,32,68,97,121,32,71,105, > 108,100,105,110,103,32,87,111, > 114,107,115,104,111,112,34,44, > 34,99,117,108,116,117,114,101, > 100,34,58,110,117,108,108,44, > 34,117,112,100,97,116,101,100, > 95,97,116,34,58,34,84,104,117, > 32,74,117,110,32,49,54,32,49, > 53,58,49,49,58,49,55,32,45,48, > 53,48,48,32,50,48,49,49,34,44, > 34,114,101,99,117,114,114,105, > 110,103,95,111,110,34,58,110, > 117,108,108,44,34,117,115,101, > 114,110,97,109,101,34,58,34, > 101,118,101,110,116,98,114,105, > 116,101,34,44,34,117,114,108, > 34,58,34,104,116,116,112,58,47, > 47,116,117,100,111,114,112,108, > 97,99,101,102,97,116,104,101, > 114,115,100,97,121,46,101,118, > 101,110,116,98,114,105,116,101, > 46,99,111,109,34,44,34,108,101, > 97,114,110,34,58,110,117,108, > 108,44,34,95,116,121,112,101, > 34,58,34,69,118,101,110,116,34, > 44,34,116,97,103,115,34,58,91, > 34,102,97,116,104,101,114,39, > 115,32,100,97,121,34,44,34,32, > 103,105,108,100,105,110,103,34, > 44,34,32,97,114,116,115,32,38, > 32,99,114,97,102,116,115,34,44, > 34,32,111,116,104,101,114,34, > 93,44,34,108,110,103,34,58,45, > 55,55,46,48,54,50,53,56,49,44, > 34,100,114,105,110,107,115,34, > 58,110,117,108,108,44,34,110, > 101,120,116,95,119,101,101,107, > 34,58,102,97,108,115,101,44,34, > 102,97,109,105,108,121,34,58, > 110,117,108,108,44,34,99,104, > 101,97,112,34,58,110,117,108, > 108,44,34,114,101,99,117,114, > 114,105,110,103,95,117,110,116, > 105,108,34,58,110,117,108,108, > 44,34,116,111,109,111,114,114, > 111,119,34,58,102,97,108,115, > 101,44,34,116,111,110,105,103, > 104,116,34,58,102,97,108,115, > 101,44,34,116,104,105,115,95, > 119,101,101,107,101,110,100,34, > 58,116,114,117,101,44,34,116, > 105,109,101,95,122,111,110,101, > 34,58,45,49,56,48,48,48,46,48, > 44,34,100,101,115,99,114,105, > 112,116,105,111,110,34,58,34, > 83,97,116,117,114,100,97,121, > 44,32,74,117,110,101,32,49,56, > 32,124,32,49,48,58,51,48,32,97, > 46,109,46,32,32,92,114,92,110, > 32,67,101,108,101,98,114,97, > 116,101,32,70,97,116,104,101, > 114,226,128,153,115,32,68,97, > 121,32,119,105,116,104,32,65, > 109,101,114,105,99,97,226,128, > 153,115,32,70,111,117,110,100, > 105,110,103,32,70,97,116,104, > 101,114,44,32,71,101,111,114, > 103,101,32,87,97,115,104,105, > 110,103,116,111,110,33,32,70, > 97,109,105,108,105,101,115,32, > 101,110,106,111,121,32,97,110, > 32,105,110,116,101,114,97,99, > 116,105,118,101,3
Re: Riak crash on 0.14.2 riak_kv_stat terminating
Jeremy,
looks like you're hitting the timeout for the MapReduce job you're running, so
the issue isn't memory-related. You could either increase the timeout for the
whole job, or split up the single MapReduce request in multiple, ensuring a
shorter runtime of a single job, then collecting the results on your
application's side.
Mathias Meyer
Developer Advocate, Basho Technologies
On Dienstag, 21. Juni 2011 at 14:25, Jeremy Raymond wrote:
> I increased the memory to 3GB on the VMs I'm using for Riak and also replaced
> a JavaScript reduce function I had missed converting to Erlang with the
> Erlang version. Monitoring the memory on the machines indicates that Riak is
> not running out of memory. There is lots of disk space on the machines (~30GB
> free).
>
> Riak crashed again this morning with a {error, flow_timeout} (below). My use
> case is I have ~75 map/reduce jobs that run about every 10 minutes over about
> 8700 items in a bucket. Once an hour the 8700 items get updated with a write
> to each item.
>
> =SUPERVISOR REPORT 21-Jun-2011::02:34:34 ===
> Supervisor: {local,luke_phase_sup}
> Context: child_terminated
> Reason: {error,flow_timeout}
> Offender: [{pid,<0.7243.26>},{name,undefined},{mfa,{luke_phase,start_lin\
> k,[riak_kv_reduce_phase,3,[accumulate,converge],undefined,<0.7241.26>,66000,[{e\
> rlang,{reduce, ...
>
>
> =ERROR REPORT 21-Jun-2011::02:34:34 ===
> ** State machine <0.7274.26> terminating
> ** Last message in was {'EXIT',<0.7241.26>,{error,flow_timeout}}
> ** When State == waiting_kl
> ** Data == {state,<0.7241.26>,mapred,<<>>,
> [[{639406966332270026714112114313373821099470487552,
> 'riak@x.x.x.x (mailto:riak@x.x.x.x)'},
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: speeding up riaksearch precommit indexing
Les, maybe it's worth looking into Beetle [1] which is a HA messaging solution built on RabbitMQ and Redis. It supports multiple brokers and message de-duplication, using Redis. It's written in Ruby, but should either way give you some inspiration on how something like this could be achieved. On a different note, you could draw some inspiration from existing implementations that build on top of Riak Core like riak_zab [2] or riak_id [3]. You'd benefit from everything Riak itself is built on and could fashion your own implementation on top of that, sort of faking a queue system and using Riak KV as a storage backend. As a final thought, there's a plugin for RabbitMQ [4] to store message directly into Riak, again benefitting from the fault-tolerance of Riak, but basically baked right into your messaging system. You could run multiple Rabbits, all writing messages directly to Riak, and then resort to failover should one of the Rabbits go down. Mathias Meyer Developer Advocate, Basho Technologies [1] http://xing.github.com/beetle/ [2] https://github.com/jtuple/riak_zab [3] https://github.com/seancribbs/riak_id [4] https://github.com/jbrisbin/riak-exchange On Mittwoch, 22. Juni 2011 at 00:07, Les Mikesell wrote: > I'd like to have fully redundant feeds with no single point of failure, > but avoid the work of indexing the duplicate copy and having it written > to a bitcask even if it would eventually be cleaned up. > > > On 6/21/2011 4:43 PM, Sylvain Niles wrote: > > Why not write to a queue bucket with a timestamp and have a queue > > processor move writes to the "final" bucket once they're over a > > certain age? It can dedup/validate at that point too. > > > > > > On Tue, Jun 21, 2011 at 2:26 PM, Les Mikesell > (mailto:lesmikes...@gmail.com)> wrote: > > > Where can I find the redis hacks that get close to clustering? Would > > > membase work with syncronous replication on a pair of nodes for a reliable > > > atomic 'check and set' operation to dedup redundant data before writing to > > > riak? Conceptually I like the 'smart client' fault tolerance of > > > memcache/membase and restricting it to a pair of machines would keep the > > > client configuration reasonable. > > > > > > -Les > > > > > > > > > On 6/18/2011 6:54 PM, John D. Rowell wrote: > > > > > > > > The "real" queues like HornetQ and others can take care of this without > > > > a single point of failure but it's a pain (in my opinion) to set them up > > > > that way, and usually with all the cluster and failover features active > > > > they get quite slow for writes.We use Redis for this because it's > > > > simpler and lightweight. The problem is that there is no real clustering > > > > option for Redis today, even thought there are some hacks that get > > > > close. When we cannot afford a single point of failure or any downtime, > > > > we tend to use MongoDB for simple queues. It has full cluster support > > > > and the performance is pretty close to what you get with Redis in this > > > > use case. > > > > > > > > OTOH you could keep it all Riak and setup a separate small cluster with > > > > a RAM backend and use that as a queue, probably with similar > > > > performance. The idea here is that you can scale these clusters (the > > > > "queue" and the indexed production data) independently in response to > > > > your load patterns, and have optimum hardware and I/O specs for the > > > > different cluster nodes. > > > > > > > > -jd > > > > > > > > 2011/6/18 Les Mikesell > > > (mailto:lesmikes...@gmail.com) > > > > <mailto:lesmikes...@gmail.com>> > > > > > > > > Is there a good way to handle something like this with redundancy > > > > all the way through? On simple key/value items you could have two > > > > readers write the same things to riak and let bitcask cleanup > > > > eventually discard one, but with indexing you probably need to use > > > > some sort of failover approach up front. Do any of those queue > > > > managers handle that without adding their own single point of > > > > failure? Assuming there are unique identifiers in the items being > > > > written, you might use the CAS feature of redis to arbitrate writes > > > > into its queue, but what happens when the redis node fails? > > > > > > > > -Les > > >
Re: Concurrency hazard when updating links
Manuel, what you're seeing is not specific to links, it's generally how concurrent updates to a single object are handled in Riak, links are no exception. If you want to handle that properly you need to enable the allow_mult property on the bucket in question. Now, whenever two clients update the same object based on the same origin (i.e. vector clock) they'll cause siblings to be created. The next time you fetch the object you'll get both siblings and you can then reconcile the difference, e.g. merging all links together, checking if they're still valid, etc., and then store back the updated value using the vector clock you got when requesting the object including the siblings. We have wiki page [1] dedicated to vector clocks and conflict resolution, explaining the process in more detail. Mathias Meyer Developer Advocate, Basho Technologies [1] On Mittwoch, 22. Juni 2011 at 11:14, Manuel Astudillo wrote: > > Hello, > > > Probably I have some miss understating on whats the best way to update links > in Riak, but I am wondering if you know about this hazard when updating links: > > > client A: getLinks(X) > client B: getLinks(X) > > > clientB:updateLinks(X) > clientA:updateLinks(X) > > > where the update made by B is cleaned by A, and it seems like there is not > even any way to get a notification about it or detect it somehow... > > > > Manuel Astudillo, M.Sc.C.E > Chief Executive Officer, founder. > > Optimal Bits > Skolskeppsgatan 26 > SE-23443, Lomma, Sweden. > mobile:+46 (0) 733 329536 > @optimalbits > > www.optimalbits.com (http://www.optimalbits.com) > > > > > > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Concurrency hazard when updating links
And here's the link I neatly forgot to include: http://wiki.basho.com/Vector-Clocks.html Mathias Meyer Developer Advocate, Basho Technologies On Mittwoch, 22. Juni 2011 at 17:18, Mathias Meyer wrote: > Manuel, > > what you're seeing is not specific to links, it's generally how concurrent > updates to a single object are handled in Riak, links are no exception. If > you want to handle that properly you need to enable the allow_mult property > on the bucket in question. > > Now, whenever two clients update the same object based on the same origin > (i.e. vector clock) they'll cause siblings to be created. The next time you > fetch the object you'll get both siblings and you can then reconcile the > difference, e.g. merging all links together, checking if they're still valid, > etc., and then store back the updated value using the vector clock you got > when requesting the object including the siblings. > > We have wiki page [1] dedicated to vector clocks and conflict resolution, > explaining the process in more detail. > > Mathias Meyer > Developer Advocate, Basho Technologies > > [1] > > On Mittwoch, 22. Juni 2011 at 11:14, Manuel Astudillo wrote: > > > > > Hello, > > > > > > Probably I have some miss understating on whats the best way to update > > links in Riak, but I am wondering if you know about this hazard when > > updating links: > > > > > > client A: getLinks(X) > > client B: getLinks(X) > > > > > > clientB:updateLinks(X) > > clientA:updateLinks(X) > > > > > > where the update made by B is cleaned by A, and it seems like there is not > > even any way to get a notification about it or detect it somehow... > > > > > > > > Manuel Astudillo, M.Sc.C.E > > Chief Executive Officer, founder. > > > > Optimal Bits > > Skolskeppsgatan 26 > > SE-23443, Lomma, Sweden. > > mobile:+46 (0) 733 329536 > > @optimalbits > > > > www.optimalbits.com (http://www.optimalbits.com) > > > > > > > > > > > > ___ > > riak-users mailing list > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Link Walking via Map Reduce
Andrew,
you're indeed hitting a JSON encoding problem here. BERT is binary data, and
won't make the JSON parser happy when trying to deserialize it, before handing
it into the map phase. You have two options here, and none of them will involve
JavaScript as the MapReduce language.
1.) Use the Protobuff API, use Erlang functions to return the value or object
(e.g. riak_mapreduce:map_object_value or riak_kv_mapreduce:map_identity), and
then run MapReduce queries with the content type 'application/x-erlang-binary'.
However, you're constrained by client libraries here, e.g. Ruby and Python
don't support this content type for MapReduce on the Protobuffs interface yet,
so you'd either implement something custom, or resort to a client that does,
riak-erlang-client comes to mind, though it was proven to be possible using the
Java client too, thanks to Russell. See [1] and [2]
2.) Convert the result from BERT into a JSON-parseable structure inside an
Erlang map function, before it's returned to the client.
The second approach certainly is less restrictive in terms of API usage, but
certainly involves some overhead inside of the MapReduce request itself, but is
less prone to encoding/decoding issues with JSON.
Mathias Meyer
Developer Advocate, Basho Technologies
[1]
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-June/004447.html
[2]
http://lists.basho.com/pipermail/riak-users_lists.basho.com/2011-June/004485.html
On Donnerstag, 23. Juni 2011 at 07:59, Andrew Berman wrote:
> Hey Ryan,
>
> Here is the error from the sasl log. It looks like some sort of
> encoding error. Any thoughts on how to fix this? I am storing the
> data as BERT encoded binary and I set the content-type as
> application/octet-stream.
>
> Thanks for your help!
>
> Andrew
>
> ERROR REPORT 9-Jun-2011::21:37:05 ===
> ** Generic server <0.5996.21> terminating
> ** Last message in was {batch_dispatch,
> {map,
> {jsanon,<<"function(value) {return [value];}">>},
> [{struct,
> [{<<"bucket">>,<<"user">>},
> {<<"key">>,<<"LikiWUPJSFuxtrhCYpsPfg">>},
> {<<"vclock">>,
>
> <<"a85hYGBgzGDKBVIsLKaZdzOYEhnzWBmes6Yd58sCAA==">>},
> {<<"values">>,
> [{struct,
> [{<<"metadata">>,
> {struct,
> [{<<"X-Riak-VTag">>,
> <<"1KnL9Dlma9Yg4eMhRuhwtx">>},
> {<<"X-Riak-Last-Modified">>,
> <<"Fri, 10 Jun 2011 03:05:11 GMT">>}]}},
> {<<"data">>,
>
> <<131,108,0,0,0,18,104,2,100,0,6,114,...>>}]}]}]},
> <<"user">>,none]}}
> ** When Server state == {state,<0.143.0>,riak_kv_js_map,#Port<0.92614>,true}
> ** Reason for termination ==
> ** {function_clause,[{js_driver,eval_js,
> [#Port<0.92614>,{error,bad_encoding},5000]},
> {riak_kv_js_vm,invoke_js,2},
> {riak_kv_js_vm,define_invoke_anon_js,3},
> {riak_kv_js_vm,handle_call,3},
> {gen_server,handle_msg,5},
> {proc_lib,init_p_do_apply,3}]}
>
> =CRASH REPORT 9-Jun-2011::21:37:05 ===
> crasher:
> initial call: riak_kv_js_vm:init/1
> pid: <0.5996.21>
> registered_name: []
> exception exit:
> {function_clause,[{js_driver,eval_js,[#Port<0.92614>,{error,bad_encoding},5000]},{riak_kv_js_vm,invoke_js,2},{riak_kv_js_vm,define_invoke_anon_js,3},{riak_kv_js_vm,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
> in function gen_server:terminate/6
> in call from proc_lib:init_p_do_apply/3
> ancestors: [riak_kv_js_sup,riak_kv_sup,<0.128.0>]
> messages: []
> links: [<0.142.0>,<0.6009.21>]
> dictionary: []
> trap_exit: false
> status: running
> heap_size: 4181
> stack_size: 24
> reductions: 2586
> neighbours:
> neighbour:
> [{pid,<0.6009.21>},{registered_name,[]},{initial_call,{riak_kv_mapper,init,[Argument__1]}},{current_function,{gen,do_call,4}},{ancestors,[riak_kv_mapper_sup,riak_kv_sup,<0.128.0>]},{messages,[]},{links,[<0.5996.21>,<12337.6227.21>,<0.162.0>]},{dictionary,[]},{trap_exit,false},{status,waiting},{heap_size,987},{stack_size,53},{reductions,1043}]
> =SUPERVISOR REPORT 9-Jun-2011::21:37:05 ===
> Supervisor: {local,riak_kv_js_sup}
> Context: child_terminated
> Reason:
> {function_clause,[{js_driver,eval_js,[#Port<0.92614>,{error,bad_encoding},5000]},{riak_kv_js_vm,invoke_js,2},{riak_kv_js_vm,define_invoke_anon_js,3},{riak_kv_js_vm,handle_call,3},{gen_server,handle_msg,5},{proc_lib,init_p_do_apply,3}]}
> Offender:
> [{pid,<0.5996.21>},{name,unde
Re: Link Walking via Map Reduce
Andrew,
the data looks like JSON, but it's not valid JSON. Have a look at the list
that's in the data section (which is your BERT encoded data), the first
character in that list is 131, which is not a valid UTF-8 character, and JSON
only allows valid UTF-8 characters. With a binary-encoded format, there's
always a chance for a control character like that to blow up the JSON generated
before and after the MapReduce code is executed. With JSON, content agnosticism
only goes as far as the set of legal characters allows.
On a side note, if the data were a valid representation of a string, you would
see it as a string in the log file as well, not just as a list of numbers.
Mathias Meyer
Developer Advocate, Basho Technologies
On Donnerstag, 23. Juni 2011 at 17:31, Andrew Berman wrote:
> But isn't the value itself JSON? Meaning this part:
>
> {struct,
> [{<<"bucket">>,<<"user">>},
> {<<"key">>,<<"LikiWUPJSFuxtrhCYpsPfg">>},
> {<<"vclock">>,
>
> <<"a85hYGBgzGDKBVIsLKaZdzOYEhnzWBmes6Yd58sCAA==">>},
> {<<"values">>,
> [{struct,
> [{<<"metadata">>,
> {struct,
> [{<<"X-Riak-VTag">>,
> <<"1KnL9Dlma9Yg4eMhRuhwtx">>},
> {<<"X-Riak-Last-Modified">>,
> <<"Fri, 10 Jun 2011 03:05:11 GMT">>}]}},
> {<<"data">>,
>
> <<131,108,0,0,0,18,104,2,100,0,6,114,...>>}]}]}
>
> So the only thing that is not JSON is the data itself, but when I get
> the value, shouldn't I be getting the all the info above which is JSON
> encoded?
>
> Thank you all for your help,
>
> Andrew
>
> On Thu, Jun 23, 2011 at 8:17 AM, Sean Cribbs (mailto:s...@basho.com)> wrote:
> > The object has to be JSON-encoded to be marshalled into the Javascript VM,
> > and also on the way out if the Accept header indicates application/json. So
> > you have two places where it needs to be encodable into JSON.
> > On Thu, Jun 23, 2011 at 11:14 AM, Andrew Berman > (mailto:rexx...@gmail.com)> wrote:
> > >
> > > Mathias,
> > >
> > > I thought Riak was content agnostic when it came to the data being
> > > stored? The map phase is not running Riak.mapValuesJson, so why is
> > > the data itself going through the JSON parser? The JSON value
> > > returned by v with all the info is valid and I see the struct atom in
> > > there so mochijson2 can parse it properly, but I'm not clear why
> > > mochijson2 would be coughing at the data part.
> > >
> > > --Andrew
> > >
> > > On Thu, Jun 23, 2011 at 5:32 AM, Mathias Meyer > > (mailto:math...@basho.com)> wrote:
> > > > Andrew,
> > > >
> > > > you're indeed hitting a JSON encoding problem here. BERT is binary data,
> > > > and won't make the JSON parser happy when trying to deserialize it,
> > > > before
> > > > handing it into the map phase. You have two options here, and none of
> > > > them
> > > > will involve JavaScript as the MapReduce language.
> > > >
> > > > 1.) Use the Protobuff API, use Erlang functions to return the value or
> > > > object (e.g. riak_mapreduce:map_object_value or
> > > > riak_kv_mapreduce:map_identity), and then run MapReduce queries with the
> > > > content type 'application/x-erlang-binary'. However, you're constrained
> > > > by
> > > > client libraries here, e.g. Ruby and Python don't support this content
> > > > type
> > > > for MapReduce on the Protobuffs interface yet, so you'd either implement
> > > > something custom, or resort to a client that does, riak-erlang-client
> > > > comes
> > > > to mind, though it was proven to be possible using the Java client too,
> > > > thanks to Russell. See [1] and [2]
> > > >
> > > > 2.) Convert the result from BERT into a JSON-parseable structure inside
> > > > an Erlang map function, before it's returned to the client.
> > > >
> > > > The second approach certainly is less restrictive in terms of API usage,
> > > > but certainly involves some overhead inside of the MapReduce request
> > > > itself,
> > > > but is less prone to encoding/decoding issues with JSON.
> > > >
> > > > Mathias Meyer
> > > > Developer Advocate, Basho Technologies
&g
[ANN] Riak Python Client 1.2.2
I'm happy to announce a freshly squeezed release of the Python client for Riak [1]. Other than a couple of bugfixes it includes some neat new features contributed by the community. Here's a handy list of the noteworthy ones: * A nicer API for using key filters with MapReduce, check out the README for examples [2] (Eric Moritz) * Support storing new objects without having to specify a key (Mark Erdmann) * Handy function aliases for the Riak built-in MapReduce functions (Eric Moritz) * Add a convenience method to create Riak object directly from file (Ana Nelson) * Support return_body parameter when creating a new object (Stefan Praszalowicz, Andy Gross) It includes a nice bunch of bug fixes and smaller improvements as well, they're all listed in the release notes [3]. Thanks to everyone who contributed to this release, keep those pull requests coming! [1] https://github.com/basho/riak-python-client [2] https://github.com/basho/riak-python-client/blob/riak-python-client-1.2.2/README.rst [3] https://github.com/basho/riak-python-client/blob/riak-python-client-1.2.2/RELEASE_NOTES.md Mathias Meyer Developer Advocate, Basho Technologies ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Mysterious JavaScript MapReduce crashes and timeouts with 0.14.2
David,
given your description of the error behaviour and the MapReduce job shown
below, here's an outline of what's happening in your cluster when you fire that
job 208 times in quick succession:
* All nodes in the cluster go through their keys using the filter you supplied,
that's 208 times of going through all 75k keys.
* Listing keys (which key filters still technically do) is a blocking
operation, blocking the vnode responsible for a particular partition, and
therefore, set of keys. It's an expensive operation, and again, you're running
hundreds of jobs in quick succession, enforcing 208 key listings in a very
short period of time.
* Then 208 MapReduce jobs have to be spread across the cluster to the
respective nodes having the data, each job is queued to be run by one of the
JavaScript VM processes.
* Then all nodes with keys matching your criteria load all the objects matching
the key filter from their respective disks, again, 208 times multiplied by the
number of objects that match the criteria.
* The reduce phase is run only on one node, in your case riak01, that's 208
times again. This is only CPU bound obviously, but requires all the data to be
gathered on that particular node from all the jobs that just ran the map
functions, again, 208 times.
The intention here is certainly not to excuse Riak for the behaviour it's
showing. It's to give you an idea that you're putting a considerable load on
your cluster. The problem is that you're also not running an SMP-enabled
version of Erlang, therefore everything utilizes just one CPU/core, slowing
things down even more. The process you're seeing should be beam.smp and not
just beam. Make sure your Erlang is compiled with --enable-smp if you're on a
multi-core/multi-CPU system.
Other things you should/could try to reduce the risk of crashing Riak:
* Instead of rapid-fire, limit the amount of concurrent MapReduce jobs.
* Ditch JavaScript, use Erlang instead. From looking at your code, the logic
seams to be straight forward and should not be that hard to implement in
Erlang. This is not a must, but you'll gain tremendously by not relying on the
JavaScript VM, which involves lots of JSON serialization and call-outs to
external libraries.
* Reduce the need for key filters, or simply avoid them entirely. If you need
range queries or queries looking for values starting with a particular string,
which seems to be the case here, consider using Riak Search instead.
* Spread the request more evenly across your cluster, reducing the coordination
load on just one of the nodes. This won't help with the entire pain though, as
it only reduces the load of the last step in your MapReduce job, so it's more
of a general practice.
Mathias Meyer
Developer Advocate, Basho Technologies
On Freitag, 24. Juni 2011 at 20:43, David Mitchell wrote:
> I am doing 208 MapReduce jobs in rapid-fire succession using anonymous
> JavaScript functions. I am sending the MapReduce jobs to a single node,
> riak01. There are about 75,000 keys in the bucket.
> Erlang: R13B04
> Riak: 0.14.2
>
> When I had my MapReduce timeout set to 120,000 ("timeout":12), I was
> getting
> mapexec_error, {error,timeout}
> This first timeout wrote to the error log after seven seconds. The second and
> third wrote to the error log after five seconds. The four timeout wrote the
> error log after eight seconds. The beam process never crashed.
>
> So, I increased the value to 30,000,000 ("timeout":3000). In the first
> run, all MapReduce jobs completed without error, each one taking about 1 to 3
> seconds to complete.
> The CPU usage on riak01 was about 50 percent for all 208 jobs.
> Below is a sample output from iostat -x
> avg-cpu: %user %nice %system %iowait %steal %idle
> 51.00 0.00 5.01 0.10 0.00 43.89
>
> Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm
> %util
> hda 0.00 8.22 0.00 3.21 0.00 91.38 28.50 0.01 2.62 2.12 0.68
>
> In the second run, on the 53rd MapReduce job, the job was still waiting to
> complete after 10 minutes. So, there was never a timeout, and nothing was
> written to the error logs. However, the beam process obviously crashed. On
> raik01, I executed the following commands:
> ./riak-admin status
> Node is not running!
> ./riak ping
> Node 'riak@10.0.60.208 (mailto:riak@10.0.60.208)' not responding to pings.
> ./riak attach
> Node is not running!
>
> However, ps and top showed the process running.
> ps output:
> 1003 31807 1.0 8.7 172080 132584 pts/1 Rsl+ Jun22 28:53
> /home/DMitchell/riak2/riak/rel/riak/erts-5.7.5/bin/beam -K true -A 64 --
> -root /home/DMitchell/riak2/riak/rel/riak -progname riak -- -home
> /home/DMitchell -- -boot
> /home/DMitchell/riak2/riak/rel/ri
Re: mr_queue gone wild
Sylvain, you should not be using riak HEAD for anything that's close to your production environment. Development in Riak is in big flux right now, and it'll be hard for us to help you find the specific problem. Could you please install Riak Search 0.14.2, best with a fresh installation, and try running this setup again to see if you get the same erroneous results? If you do, some more details on your data and the MapReduce jobs you're running would be great to reproduce and figure out the problem. Mathias Meyer Developer Advocate, Basho Technologies On Mittwoch, 29. Juni 2011 at 00:41, Sylvain Niles wrote: > We recently started trying to move our production environment over to > riak and we're seeing some weird behavior that's preventing us from > doing so: > > Running: riak HEAD from github as of last Friday, riak_search turned > on with indexing of the problem bucket "events". > When we turn on our processes that start creating objects in the > bucket "events", the mr_queue directory starts growing massively and > the riak process starts spending most of its time in io_wait. With a > total of about 7000 objects (each is a ripple document that's got > maybe 10-20 lines of text in it) in the events bucket out bitcask dir > was ~240MB and the mr_queue dir was 1.2GB. Something is going horribly > wrong.. Our logs only show flow_timeouts for normal requests trying to > do simple map/reduce lookups. Any ideas where to look? > > Thanks in advance, > Sylvain > > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Mysterious JavaScript MapReduce crashes and timeouts with 0.14.2
On Samstag, 25. Juni 2011 at 00:23, David Mitchell wrote: > Hi Mathias, > > Thank you for responding, and give me things to try. > > I am testing Raik using a hypervisor, so each node only has one CPU and 1.5 > GB of RAM. I have 222,526 total keys stored. > I'd suggest testing your setup on something other than a hypervisor'd setup of virtual machines to get a good picture of performance, especially at this memory and machine size. > Correct me if I am mistaken, but if we want to use Riak for data processing > via MapReduce, we should abandon JavaScript and opt for Erlang. This is my > first take away. Second, use Riak Search rather than key filters to do range > queries. > Considering your key filter setup, yes, you should be using Riak Search instead to prefilter your MapReduce inputs. As for MapReduce itself, it's well worth looking into Erlang, because it avoids a lot of overhead involved with JavaScript MapReduce. I wouldn't hold it generally true to always go for Erlang for more intensive MapReducing, it just depends on how often you run your queries and what execution speed expectations you have of them. In your case I'd say it'd be well worth looking into Erlang as an alternative. Mathias Meyer Developer Advocate, Basho Technologies ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Map/Red Function Cache Corruption?
Eric,
are you by any chance still running Riak 0.14.1? There was a bug showing the
same symptoms you're describing, which was fixed in the recent 0.14.2 release.
Mathias Meyer
Developer Advocate, Basho Technologies
On Samstag, 2. Juli 2011 at 14:40, Eric Stevens wrote:
> I've been struggling with this for a few days now. I have a pair of related
> Map/Reduce queries. Executed individually they produce the expected results,
> but when executed in parallel (simultaneous XHR calls from the browser),
> their results tend to get mixed up with each other; I see values from both
> mixed together in a single result.
>
> Once that corruption happens, even the calls executed individually can end up
> with cross-pollinated results. It seems to be related to the caching of the
> anonymous javascript functions used, as adding a space to the function
> declaration clears the issue up. Taking that space back out causes the
> problem again.
>
> I can confirm that the corruption can become persistent by using a line such
> as (PHP client):
> ejsLog('/tmp/mapred/reduce_" . join('.', explode('\\', get_class($this))) .
> ".log', JSON.stringify(values));
>
> After the corruption happens, if I clear the log path, execute just the one
> map/red operation, I will see log entries appear for both classes (the class
> name being encoded as part of the function declaration). The problem seems to
> happen for both map and reduce operations, though I have not specifically
> noticed cross-contamination between map and reduce functions, just
> map-for-map and reduce-for-reduce.
>
> Adding this as the first line of the function declaration seems to solve the
> problem, but this is obviously something intended to defeat caching, so I'm
> sure there's a performance cost to this (note, just get_class($this) is not
> sufficient, cross-contamination can still happen, each call needs to be
> unique call-to-call, thus the mt_rand()).
> "//".get_class($this).' '.mt_rand()."
>
>
> I noticed Francisco Treacy had this same problem back in September:
> http://riak-users.197444.n3.nabble.com/reduce-headaches-td1524860.html but I
> didn't see any resolution there. Is there a potential longer term cost to my
> cachebusting, such as memory exhaustion (i.e. is there any garbage collection
> on the function cache)?
>
> Any help would be sincerely appreciated!
>
> -Eric
> ___
> riak-users mailing list
> riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: db.save() vs. db.update()
Claus,
it looks you're mixing up metadata objects with actual data objects when
calling update. update does a get/put cycle, where it fetches an object, merges
its attributes with the ones you hand over to update() and save the result
back. Any change you accidentally pass in the meta object from a different part
of your code that's fetching the object initially?
Mathias Meyer
Developer Advocate, Basho Technologies
On Mittwoch, 6. Juli 2011 at 00:30, Claus Guttesen wrote:
> Hi.
>
> When saving a new record using db.update() getting the record using
> curl has alot more info:
>
> curl "localhost:8098/riak/users/RQC6seqRIKkBNKQt65QMko3W66"
> {"props":{"w":"quorum","r":"quorum","big_vclock":50,"old_vclock":86400,"n_val":3,"postcommit":[],"last_write_wins":false,"small_vclock":10,"allow_mult":false,"young_vclock":20,"rw":"quorum","pw":0,"dw":"quorum","linkfun":{"mod":"riak_kv_wm_link_walker","fun":"mapreduce_linkfun"},"name":"users","chash_keyfun":{"mod":"riak_core_util","fun":"chash_std_keyfun"},"basic_quorum":true,"notfound_ok":false,"pr":0,"precommit":[]},"club":"BBIL","mname":"","year":"1967","gender":"male","fname":"Claus","lname":"Guttesen"}%
>
> When doing a db.save() with the same data I get:
>
> curl "localhost:8098/riak/users/RQC6seqRIKkBNKQt65QMko3W66"
> {"fname":"Claus","mname":"","lname":"Guttesen","year":"1967","club":"BBIL","gender":"male"}
>
> If I do a db.save() first and then a db.update() the information is
> also terse. Why is db.update() more verbose?
>
> I'm using riak-js v. 0.4.0.
>
> My idea was to add more information in addtion to name, club, year
> born, gender. But when I edit the user and save it the previous data
> gets overwritten. So I changed db.save() to db.update(), saved a new
> user and noticed alot more info.
>
> --
> regards
> Claus
>
> When lenity and cruelty play for a kingdom,
> the gentler gamester is the soonest winner.
>
> Shakespeare
>
> twitter.com/kometen (http://twitter.com/kometen)
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: riak mapreduce failure
Matt,
in your JS function, you return value, where instead you must return a list of
values, so changing it to
return [value];
fixes the problem.
Mathias Meyer
Developer Advocate, Basho Technologies
On Donnerstag, 7. Juli 2011 at 05:53, Matt Graham wrote:
> Hi,
> I'm trying to test out riak's mapreduce for just a simple case but I
> can't get it to work. Maybe someone can help with what I'm doing
> wrong. I have the latest build of Riak available for Ubuntu
> installed.
>
> I add an object to riak, that works. I can then fetch it. Then I try
> to run a mapreduce on that bucket and it fails.
>
> mdg@maple:~/tmp$ curl -i -d 6 http://127.0.0.1:8098/riak/test/2
> HTTP/1.1 204 No Content
> Vary: Accept-Encoding
> Server: MochiWeb/1.1 WebMachine/1.7.3 (participate in the frantic)
> Date: Thu, 07 Jul 2011 03:47:00 GMT
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 0
>
> mdg@maple:~/tmp$ curl -i http://127.0.0.1:8098/riak/test/2
> HTTP/1.1 200 OK
> X-Riak-Vclock: a85hYGBgzGDKBVIs7JvMrDOYEhnzWBlMVnQe58sCAA==
> Vary: Accept-Encoding
> Server: MochiWeb/1.1 WebMachine/1.7.3 (participate in the frantic)
> Link: ; rel="up"
> Last-Modified: Tue, 05 Jul 2011 09:33:16 GMT
> ETag: "4fhdlMvPAPNw4SegwaTNMv"
> Date: Thu, 07 Jul 2011 03:47:09 GMT
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 1
>
> 6
> mdg@maple:~/tmp$curl -i -d
> '{"inputs":"test","query":[{"map":{"language":"javascript","source":"function(value)
> { return value; }","keep":true}}]}' -H "Content-Type:
> application/json" http://127.0.0.1:8098/mapred
> HTTP/1.1 500 Internal Server Error
> Server: MochiWeb/1.1 WebMachine/1.7.3 (participate in the frantic)
> Date: Thu, 07 Jul 2011 03:47:22 GMT
> Content-Type: text/html
> Content-Length: 692
>
> 500 Internal Server
> ErrorInternal Server ErrorThe server
> encountered an error while processing this
> request:{error,{exit,{json_encode,{bad_term,{<<"bucket">>,<<"test">>}}},
> [{mochijson2,json_encode,2},
> {mochijson2,'-json_encode_array/2-fun-0-',3},
> {lists,foldl,3},
> {mochijson2,json_encode_array,2},
> {riak_kv_wm_mapred,process_post,2},
> {webmachine_resource,resource_call,3},
> {webmachine_resource,do,3},
>
> {webmachine_decision_core,resource_call,1}]}}mochiweb+webmachine
> web server
>
>
> I get that json_encode error every time. Any ideas on how to not get
> that or how to work around it?
>
> Thanks,
> Matt
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Quering Riaksearch on Second Node
Muhammad,
can you see the merge_index directory fill with data on the second machine
that's not responding to queries? Anything unusual showing up in the log?
Can you access the normal data you're indexing through Riak KV on both machines?
Mathias Meyer
Developer Advocate, Basho Technologies
On Donnerstag, 7. Juli 2011 at 20:28, Muhammad Yousaf wrote:
> Hi,
>
> I have 2 node Riaksearch cluster. My first node is working perfectly with
> that stats
>
> curl -H "Accept: text/plain" http://192.168.1.182:8098/stats
> {
> "vnode_gets": 0,
> "vnode_puts": 0,
> "read_repairs": 0,
> "vnode_gets_total": 480294,
> "vnode_puts_total": 106564,
> "node_gets": 0,
> "node_gets_total": 341810,
> "node_get_fsm_time_mean": "undefined",
> "node_get_fsm_time_median": "undefined",
> "node_get_fsm_time_95": "undefined",
> "node_get_fsm_time_99": "undefined",
> "node_get_fsm_time_100": "undefined",
> "node_puts": 0,
> "node_puts_total": 34671,
> "node_put_fsm_time_mean": "undefined",
> "node_put_fsm_time_median": "undefined",
> "node_put_fsm_time_95": "undefined",
> "node_put_fsm_time_99": "undefined",
> "node_put_fsm_time_100": "undefined",
> "read_repairs_total": 1711,
> "cpu_nprocs": 472,
> "cpu_avg1": 374,
> "cpu_avg5": 479,
> "cpu_avg15": 502,
> "mem_total": 4002107392.0,
> "mem_allocated": 3946246144.0,
> "nodename": "riaksearch@192.168.1.182",
> "connected_nodes": [
> "riaksearch@192.168.1.167"
> ],
> "sys_driver_version": "1.5",
> "sys_global_heaps_size": 0,
> "sys_heap_type": "private",
> "sys_logical_processors": 4,
> "sys_otp_release": "R14B01",
> "sys_process_count": 387,
> "sys_smp_support": true,
> "sys_system_version": "Erlang R14B01 (erts-5.8.2) [source] [64-bit]
> [smp:4:4] [rq:4] [async-threads:64] [hipe] [kernel-poll:true]",
> "sys_system_architecture": "x86_64-unknown-linux-gnu",
> "sys_threads_enabled": true,
> "sys_thread_pool_size": 64,
> "sys_wordsize": 8,
> "ring_members": [
> "riaksearch@192.168.1.167",
> "riaksearch@192.168.1.182"
> ],
> "ring_num_partitions": 64,
> "ring_ownership":
> "[{'riaksearch@192.168.1.182',32},{'riaksearch@192.168.1.167',32}]",
> "ring_creation_size": 64,
> "storage_backend": "riak_kv_bitcask_backend",
> "pbc_connects_total": 505,
> "pbc_connects": 0,
> "pbc_active": 0,
> "riak_err_version": "1.0.0",
> "runtime_tools_version": "1.8.4.1",
> "riak_search_version": "0.14.0",
> "riak_search_core_version": "0.14.0",
> "qilr_version": "0.14.0",
> "riak_solr_version": "0.14.0",
> "merge_index_version": "0.14.0",
> "luwak_version": "1.0.0",
> "skerl_version": "1.0.0",
> "riak_kv_version": "0.14.0",
> "bitcask_version": "1.1.5",
> "riak_core_version": "0.14.0",
> "luke_version": "0.2.3",
> "erlang_js_version": "0.5.0",
> "mochiweb_version": "1.7.1",
> "webmachine_version": "1.8.0",
> "crypto_version": "2.0.2",
> "os_mon_version": "2.2.5",
> "cluster_info_version": "1.0.0",
> "sasl_version": "2.1.9.2",
> "stdlib_version": "1.17.2",
> "kernel_version": "2.14.2",
> "executing_mappers": 0
>
> my query from first node works fine
>
> (riak1@127.0.0.1 (mailto:riak1@127.0.0.1))3> {ok,Client}
> =riakc_pb_socket:start_link('192.168.1.182',8087).
> {ok,<0.46.0>}
> (riak1@127.0.0.1 (mailto:riak1@127.0.0.1))4> riakc_pb_socket:search(Client,
> <<"player">>, "playername:yousaf").
> {ok,[[<<"player">>,<<"yousaf">>]]}
>
> also
>
> curl http://192.168.1.182:8098/solr/player/select?q=playername:yousaf
>
>
>
> 0
> 2
>
> on
> 0
> playername:yousaf
> or
> value
> standard
> 1.1
> 1
>
>
>
>
> yousaf
>
> 1
>
> 20110705
>
> muhammad
>
> 1
>
> snowball
>
> 100
>
> yousaf
>
> yousaf
>
> world1
>
> 100
>
>
>
>
>
>
>
> but on my second node i can get list of keys but cannot search it
>
> (riak1@127.0.0.1)6> {ok,Client}
> =riakc_pb_socket:start_link('192.168.1.167',8087).
> {ok,<0.53.0>}
> (riak1@127.0.0.1)7> riakc_pb_socket:search(Client, <<"player">>,
> "playername:yousaf").
> {ok,[]}
>
> and
>
> curl http://192.168.1.167:8098/solr/player/select?q=playername:yousaf.
>
>
>
> 0
> 1
>
> on
> 0
> playername:yousaf.
> or
> value
> standard
> 1.1
> 0
>
>
>
>
>
>
> but data is replicating properly. Any idea's how to fix it only searching is
> causing problem??
>
>
>
> Regards,
>
> Muhammad Yousaf
> ___
> riak-users mailing list
> riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: /stats on Riak server
Jon, The erldocs already pointed to by Jeff are a good start to find out what the values represent, and I made a note to improve our documentation to go into more detail on monitoring and the stats too. the most important stats to look out for are usually the fsm times and their percentiles (node_get_fsm* and node_put_fsm*), as they'll immediately point you towards increased latency scenarios, even for shorter bursts. To get an idea of the number of requests Riak is handling, gather the number of requests in the last minute too (vnode_gets and vnode_puts) or use a counter for the total number of requests (vnode_gets_total and vnode_puts_total). Mathias Meyer Developer Advocate, Basho Technologies On Samstag, 9. Juli 2011 at 06:07, Jon Baer wrote: > Hi, > > I am wondering if there is a clearly explanation or docs somewhere that > explains a little bit more on what some of the /stats represent. > > Also which are the ones you would want your sysadmin to watch / monitor > closely. > > Thanks. > > - Jon > > > > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Two question in re pre/post commit hooks
Jeffrey, 1.) a given key/value data pair is owned by multiple nodes, depending on your replication level. Operations on a particular key/value pair happen on all nodes that are primary replicas, but requests leading to these operations can be coordinated by any node in the cluster. So when you write a piece of data, N nodes will be involved in the process. But your commit hooks will only be run on the coordinating node, the one you sent the request to. Ownership can change at any time when nodes go down or the cluster changes due to new nodes entering oder old nodes leaving. 2.) The key/value pair is not locked. As per above, this would require a distributed lock across all replicas of the key/value pair, which is not how Riak works. While the update itself is coordinated by a single process on the coordinating, at any time, a different client could come along and update the same piece of data through a different path, so e.g. through a different or even the same node. The original updater doesn't have any knowledge of that, if it's a conflicting update, it will simply create a sibling (given you enabled sibling creations on that particular buckets). You may want to look into Statebox [1] as an alternative way, or try to serialize writes through a messaging system like RabbitMQ to ensure atomicity to a certain extent. [1] https://github.com/mochi/statebox Mathias Meyer Developer Advocate, Basho Technologies On Freitag, 8. Juli 2011 at 19:14, Jeffrey Kesselman wrote: > (1) AIUI a given key/value data pair is "owned" by a given node at one time > and operations on it happen on that node and are then replicated out. Is this > correct? > > (2) Is the key/value pair "locked" between pre update and post update? > > The motivation for this question is this. > > I need to do the following operations on update: > > (1) In the pre-update hook, get the existing value (lets call it A) > (2) Compare some data in the update (lets call it A') with A > (3) Validate or reject the update based on that comparison > (4) If validated, write A' out as the new value of A, if not, return a > validation error. > > I need 1-4 to happen in a consistent fashion, If someone else can come in and > update A after (1) but before (4), the scheme wont work. > > I *could* do my own locking in (1) and then add an unlock step in (5) > assuming that all updates will take place on the same node, but I dislike > injecting locking into a system that has unknown other expectations of the > operation. > -- > It's always darkest just before you are eaten by a grue. > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Introduction question
Anton, 1.) You can include custom headers stored with Riak objects, using the X-Riak-Meta-. These are specific for every Riak object. 2.) To allow just GET requests you'll be better off putting an HTTP aware filter or reverse proxy in front of your Riak cluster, which explicitly only enables GET requests. There's no way of disabling certain access methods for HTTP in Riak directly, at least as far as configuration goes. As for your general question, it's hard to give a good answer without knowing more details about what your data looks like and, what read/write patterns you have. Providing us with some more details would help giving you an answer here. If you only access all data for a particular zip code in one go though, the answer would probably be yes. Mathias Meyer Developer Advocate, Basho Technologies On Dienstag, 12. Juli 2011 at 15:04, Anton Podviaznikov wrote: > Hi, > > I just started with learning documentation and playing with Riak so I have > few questions: > > 1. Can I set custom HTTP headers for HTTP API? > 2. Can I enable e.x. just HTTP GET requests? > > > And general question: > e.x. I have list of zip codes and want to expose that data via REST(just for > reading). > > Is it good idea to use riak for such task? > > Best regards, > Anton Podviaznikov > > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Quering Riaksearch on Second Node
You don't necessarily have to reindex your data. Copying over the data
directories from the old Riak instances should do.
Mathias Meyer
Developer Advocate, Basho Technologies
On Montag, 11. Juli 2011 at 14:58, Muhammad Yousaf wrote:
>
> Thanks Mathias & Sylvain,
> Everything works fine on second node except riakc_pb_socket:search command.
> Yes, Normal data can be accessed.
>
> Upgrading riaksearch to 0.14.2 solves that problem. 1 Question though,
> Copying Data folder which includes bitcask,merge_index,mr_queue and ring to
> new version
> of riak search Data folder will it work or not ?. If yes, do i have to
> reindex my data ?
>
>
>
> Regards,
>
> Muhammad Yousaf
>
> > Date: Mon, 11 Jul 2011 09:59:09 +0200
> > From: math...@basho.com (mailto:math...@basho.com)
> > To: muhammad.you...@live.com (mailto:muhammad.you...@live.com)
> > CC: riak-users@lists.basho.com (mailto:riak-users@lists.basho.com)
> > Subject: Re: Quering Riaksearch on Second Node
> >
> > Muhammad,
> >
> > can you see the merge_index directory fill with data on the second machine
> > that's not responding to queries? Anything unusual showing up in the log?
> >
> > Can you access the normal data you're indexing through Riak KV on both
> > machines?
> >
> > Mathias Meyer
> > Developer Advocate, Basho Technologies
> >
> >
> > On Donnerstag, 7. Juli 2011 at 20:28, Muhammad Yousaf wrote:
> >
> > > Hi,
> > >
> > > I have 2 node Riaksearch cluster. My first node is working perfectly with
> > > that stats
> > >
> > > curl -H "Accept: text/plain" http://192.168.1.182:8098/stats
> > > {
> > > "vnode_gets": 0,
> > > "vnode_puts": 0,
> > > "read_repairs": 0,
> > > "vnode_gets_total": 480294,
> > > "vnode_puts_total": 106564,
> > > "node_gets": 0,
> > > "node_gets_total": 341810,
> > > "node_get_fsm_time_mean": "undefined",
> > > "node_get_fsm_time_median": "undefined",
> > > "node_get_fsm_time_95": "undefined",
> > > "node_get_fsm_time_99": "undefined",
> > > "node_get_fsm_time_100": "undefined",
> > > "node_puts": 0,
> > > "node_puts_total": 34671,
> > > "node_put_fsm_time_mean": "undefined",
> > > "node_put_fsm_time_median": "undefined",
> > > "node_put_fsm_time_95": "undefined",
> > > "node_put_fsm_time_99": "undefined",
> > > "node_put_fsm_time_100": "undefined",
> > > "read_repairs_total": 1711,
> > > "cpu_nprocs": 472,
> > > "cpu_avg1": 374,
> > > "cpu_avg5": 479,
> > > "cpu_avg15": 502,
> > > "mem_total": 4002107392.0,
> > > "mem_allocated": 3946246144.0,
> > > "nodename": "riaksearch@192.168.1.182 (mailto:riaksearch@192.168.1.182)",
> > > "connected_nodes": [
> > > "riaksearch@192.168.1.167 (mailto:riaksearch@192.168.1.167)"
> > > ],
> > > "sys_driver_version": "1.5",
> > > "sys_global_heaps_size": 0,
> > > "sys_heap_type": "private",
> > > "sys_logical_processors": 4,
> > > "sys_otp_release": "R14B01",
> > > "sys_process_count": 387,
> > > "sys_smp_support": true,
> > > "sys_system_version": "Erlang R14B01 (erts-5.8.2) [source] [64-bit]
> > > [smp:4:4] [rq:4] [async-threads:64] [hipe] [kernel-poll:true]",
> > > "sys_system_architecture": "x86_64-unknown-linux-gnu",
> > > "sys_threads_enabled": true,
> > > "sys_thread_pool_size": 64,
> > > "sys_wordsize": 8,
> > > "ring_members": [
> > > "riaksearch@192.168.1.167 (mailto:riaksearch@192.168.1.167)",
> > > "riaksearch@192.168.1.182 (mailto:riaksearch@192.168.1.182)"
> > > ],
> > > "ring_num_partitions": 64,
> > > "ring_ownership": "[{'riaksearch@192.168.1.182
> > > (mailto:riaksearch@192.168.1.182)',32},{'riaksearch@192.168.1.167
> > > (mailto:riaksearch@192.168.1.167)',32}]",
> > > "ring_creation_s
Re: Riak in a common test suite
Mikhail, that certainly sounds possible, but would extend the run time of your test suite quite considerably, e.g. when you start a fresh set of virtualized instances on every run. An alternative would be to always leave a Riak cluster running for testing purposes and reset it after or before every complete test run, i.e. stop the Riak processes, wipe the data, and start them up again. Ripple (Ruby client for Riak) and riak-js (Node.js client) both include a test server that runs a Riak instance with an in-memory backend, maybe that would be an alternative to go with? Mathias Meyer Developer Advocate, Basho Technologies On Dienstag, 12. Juli 2011 at 18:08, Mikhail Sobolev wrote: > Hi, > > As part of our test suite I'd like to start a three node cluster for the > testing purposes. > > Could anyone [please! :)] comment on feasibility/possibility of such a > thing? > > -- > Misha > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Recap for July 27 - 28
It's possible to update objects from Erlang MapReduce, but that doesn't conflict with the answer to 4) because it's still a similar workflow of fetching (through MapReduce) and then storing the object back. The original question refered to partial updates, e.g. appending a new entry to a list stored e.g. as JSON under a specific key in an atomic fashion, which is not possible in Riak either way. Mathias Meyer Developer Advocate, Basho Technologies On Samstag, 30. Juli 2011 at 11:47, Jonathan Langevin wrote: > Re: 4) - This is regarding to updating the data that belongs to a key, > correct? > I've read elsewhere that it's possible to modify & save an object via a MR > call, is that correct? (seems it would conflict with the answer to point 4) > > > Jonathan Langevin > Systems Administrator > Loom Inc. > Wilmington, NC: (910) 241-0433 - jlange...@loomlearning.com > (mailto:jlange...@loomlearning.com) - www.loomlearning.com > (http://www.loomlearning.com/) - Skype: intel352 > > > On Fri, Jul 29, 2011 at 7:39 PM, Mark Phillips (mailto:m...@basho.com)> wrote: > > Afternoon, Evening, Morning to All - > > > > Short Recap for today. > > > > Enjoy and have a great weekend. > > > > Mark > > > > Community Manager > > Basho Technologies > > wiki.basho.com (http://wiki.basho.com) > > twitter.com/pharkmillups (http://twitter.com/pharkmillups) > > --- > > > > Riak Recap for July 27 - 28 > > > > > > 1) Last reminder to register for the "Riak and Python" webinar > > happening on August 2nd. (If you don't register, you run the risk of > > hurting Mathias' feelings). > > > > * Details here ---> > > http://blog.basho.com/2011/07/19/Riak-and-Python-Webinar-Happening-August-2/ > > > > 2) Kresten Krab Thorup's talk from Erlang Factory London about his > > Riak Mobile project is now available in video form. > > > > * Watch here ---> > > http://www.erlang-factory.com/conference/London2011/speakers/KrestenKrabThorup > > > > 3) Looks like Riak will be one of the topics at the next Ruby Meetup > > happening in Colima, Mexico. > > > > * Details here ---> http://www.meetup.com/rubyco/events/24982511/ > > > > 4) Q --- Is it possible to add data to a key in riak? (from doki_pen via > > #riak) > > > > A --- No. You'll always have to fetch, update, and store the object. > > > > ___ > > riak-users mailing list > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Ann: Release Python client 1.3.0
Happy Eunit day everyone! Hot on the heels of this week's webinar [1] on Riak and Python, I'm happy to announce a fresh release of the Python client for Riak, now at version 1.3.0 [2], which includes several new features as outlined in the webinar. Most noteworthy features are support for the Solr interface Riak Search provides, support for Luwak, an in-memory test server, and support for caching protocol buffers and reusing/pooling HTTP connections. To give each new feature the attention it deserves I summed up the most noteworthy changes in a blog post. For a list of changes, please consult the release notes [4]. [1] http://blog.basho.com/2011/08/03/Follow-Up-To-Riak-And-Python-Webinar/ [2] https://github.com/basho/riak-python-client [3] http://blog.basho.com/2011/08/04/Whats-New-In-Riaks-Python-Client/ [4] https://github.com/basho/riak-python-client/blob/riak-python-client-1.3.0/RELEASE_NOTES.md Mathias Meyer Developer Advocate, Basho Technologies ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Error compiling Riak from source
The wiki refers to the current stable release of Riak, which is 0.14.2, and which still relies on Erlang R13B04. To compile the current development master, e.g. to try out secondary indexes, you need to use Erlang R14B03 instead. Mathias Meyer Developer Advocate, Basho Technologies On Dienstag, 9. August 2011 at 12:22, Morten Siebuhr wrote: > Hi all, > > I've tried to compile Riak from the Github sources (for testing > secondary indices), but to no avail. I installed Erlang R14B04 per the > wiki instructions, but when I use that to compile Riak itself, I get > the following error: > > $ make rel > ... > ERROR: OTP release R13B04 does not match required regex R14B0[23] > make: *** [rel] Error 1 > > (The regex is found in rebar.config.) > > Is this a bug or, more likely, am I missing something important > between installing R14B04 and compiling Riak? > > -- > Morten Siebuhr > > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Python Riak Search Returning 'NoneType' object
Which interface for search are you using? Are you using the Solr search interface, or do you go through MapReduce? The former would using client.solr().query(), the latter client.search(). Should you use the latter, do you call obj.get() on every result object before calling get_data(). The result set by default is empty unless you feed it into a Map phase to extract the object. Cheers, Mathias On Dienstag, 13. September 2011 at 06:47, idmartin wrote: > this problem has completely halted progress on my project, if anyone has any > input it would be greatly appreciated. > > -- > View this message in context: > http://riak-users.197444.n3.nabble.com/Python-Riak-Search-Returning-NoneType-object-tp3317538p3331764.html > Sent from the Riak Users mailing list archive at Nabble.com > (http://Nabble.com). > > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: [RFC] Python client: move to properties
The short answer: yes, we can and we should. I had that on my radar for a while too, because it felt un-Pythonic. As for deprecation, there's no specific rule for the Python client yet. I'm happy to accept a patch for it for e.g. a version of the client 1.4.0 with an announcement that support for these getters/setters will be removed in said version. I'm not a fan of removing things in patch versions myself, but that's certainly up for discussion. The Python client is quite old and has come a long way, so I'm all for getting rid of the cruft that came with it. Cheers, Mathias On Mittwoch, 14. September 2011 at 13:30, Greg Stein wrote: > Hi all, > > There are some non-Pythonic patterns in riak-python-client that should > be pretty easy to switch. Things like client.get_r() and > client.set_r() are kinda silly. Python long ago moved past the > getter/setter paradigm, with the notion of directly exposing instance > attributes. As Guido has said numerous times, "we're all adults here", > so we don't need to wrap simple attribute access/change in methods > which do exactly that anyways. Just allow apps to directly change it, > since the developers doing that *are* adults and know what they're > doing. > > For something like bucket.get_r()... first: it should not have a > damned argument that is returned if you pass in a value. That is > nonsense. Don't call the function if you're going to pass an argument! > The remaining logic looks at a local value, or defers to the client > default value. We can make "bucket.r" a property, and create a getter > that does the right thing. Applications can then use "bucket.r" and > they will get the right logic, rather than the messier > "bucket.get_r()". > > There are similar changes throughout (eg. get_transport). > > But... this goes back to a question that I've raised earlier: what > kinds of API guarantees are being made on the interface? Can we simply > get rid of the .get_r() method? If we choose to do so, is there a > deprecation policy? With a policy in place, then it would be easier > for me (and others) to provide patches, knowing that they conform to > compatibility requirements. As it stands, I'd be happy to code a > patch, but am wary that my effort would be rejected per some > (unstated) policy. > > I don't know how much of a compatibility policy lies with Basho or the > community. Dunno how to help there. > > And back to the start: can we get the code simplified, and move > towards properties? > > Cheers, > -g > > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Python Riak Search Returning 'NoneType' object
Is it possible at all you indexed the documents using the Solr interface, and you're now trying to use them in a MapReduce query? If so, that won't work, because the get() call expects the objects with their respective bucket and key to exist in Riak KV, which means you'd have to index via a pre-commit hook and storing data in Riak directly. Cheers, Mathias On Dienstag, 13. September 2011 at 17:20, idmartin wrote: > matt, > > I am using client.search() would the other method be better? > > Here is a snippet of the code. > > def search( self, bu, query ): > > if not self.bucket[bu].search_enabled(): > self.bucket[bu].enable_search() > > query = self.client.search(bu , query).run() > > data = '' > > for r in query: > > s = r.get().get_data() > > data += s['post'] > > return data > > It fails when trying to append data with a "nonetype" object, ive tried it > serveral other ways in the code itself, i always get the same result. > > last night i boiled it down to get() returing a riakobject but get_data() is > returning None. > > init() on the riakobject sets self._data = None. alls get_data() does is > return self._data. Somewhere along the line i assume the RiakObject is > supposed to populate the data but it doesnt seem to be happening. > > -- > View this message in context: > http://riak-users.197444.n3.nabble.com/Python-Riak-Search-Returning-NoneType-object-tp3317538p3332906.html > Sent from the Riak Users mailing list archive at Nabble.com > (http://Nabble.com). > > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Can't PUT to a Bucket
Your URL is pointing to a non-existing endpoint. Change it to http://markson.hk:8098/buckets/test/keys/1234 (http://markson.hk:8098/buckets/test/1234) (note the "keys" URL component before the actual key), and you should be good to go. Cheers, Mathias http://nosqlhandbook.com On Freitag, 18. November 2011 at 13:22, Markson wrote: > First I'm pretty sure the riak is setting ok by using the command: riak-admin > status > > But I can't PUT in into a bucket test: > curl -v -X PUT -d 'This is really cool' -H "Content-Type: text/plain" > http://markson.hk:8098/buckets/test/1234 > > < HTTP/1.1 404 Object Not Found > < Server: MochiWeb/1.1 WebMachine/1.9.0 (participate in the frantic) > < Date: Fri, 18 Nov 2011 12:13:03 GMT > < Content-Type: text/html > < Content-Length: 193 > > Does the error due to the new bucket test? Should I create it first? > > > Thanks > > > > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Ripple + riak confguration
Vamsi, there's nothing wrong with your cluster setup. Support for multiple nodes in Ripple is being worked on in the master branch, you can have a lookie here at the documentation update on how to use it: https://github.com/seancribbs/ripple/commit/88fed2bdb1900ccc26fd292b3607d66cbcbe82c4 Cheers, Mathias On Montag, 28. November 2011 at 11:43, Vamsi Krishna wrote: > Hi, > > I was able to set up a riak cluster with 4 nodes. I am using ripple as an orm > for riak db, but I can only configure with one node. I am getting data > replicated in other nodes but, the issue is if the configured node with > ripple is down I was not able to make use of the other nodes. Is there a way > that I can configure ripple with all those nodes? Or can is there something > wrong with the cluster setup ? I followed wiki.basho.com > (http://wiki.basho.com) for the cluster setup. > > Regards, > Vamsi. > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: discourraging reaction about riak
If you read it thoroughly, he doesn't recommend against using Riak. He recommends to start out with a relational database like MySQL or Postgres if you don't know what Riak is and how you'd benefit from it, or how you data will evolve over time. Start out with one of them, add Riak to the mix later, when the need arises. Which I think is not unreasonable suggest, even if he recommends to use MySQL and Postgres in a K-V mode. My head spins from storing an EAV model in MySQL, but it's been done before and will be done again. Correct me if I'm wrong, but that's how I read what he says, at least from the somewhat mixed results of the translation. Cheers, Mathias http://twitter.com/riakhandbook On Friday, 9. December 2011 at 09:42, Lyes zaiko wrote: > Hi everybody > > I've read an article of an erlang developper talking about his experience > with riak, and I'm a bit afraid on his conclusions because in the final, he > recommanded to use sql dtabases in kv-mode instead of riak!!! > > here is the original russian link > http://lionet.livejournal.com/98362.html?utm_source=feedburner, and the > English Google translated link > http://translate.google.com/translate?hl=en&sl=ru&tl=en&u=http%3A%2F%2Flionet.livejournal.com%2F98362.html%3Futm_source%3Dfeedburner > > > What do you think about that? > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
[ANN] The Riak Handbook
Hey everyone, I'm happy to announce the public release of the Riak Handbook, the most definitive guide on Riak. The most comprehensive book on all things Riak, Riak Search, Riak 2i, MapReduce and data modeling. I could go on and on about all the things it covers, but you should go and see for yourself: http://riakhandbook.com/ Cheers, Mathias ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Ruby eventmachine async client
Hey Sean, for a while I hacked on an EM-based Riak client but didn't find the time to investigate it further. So you can do several things with riak-ruby-client an EM: 1) Just use the client with EM.defer, putting it into other threads. That arguably voids the purpose of using EM, but at least defers the blocking I/O work to a pool of threads where it can be handled without blocking the reactor. 2) Write a backend for riak-ruby-client that utilizes em-synchrony and em-http to turn things into a more procedural workflow. That should be simple enough based on the net/http backend [1]. Be aware of the limitations of using fibers though, their stacks only fit 4K so keep your stacks small :) 3) You could also just use em-http-request, but that way you'd give up a lot of functionality that's already in riak-ruby-client. I'd recommend looking at #2, it looks like an easy path and should make for a great addition to the riak-ruby-client too. In my ideal world, there'd be a backend based on Faraday [2], so that the Ruby client wouldn't even have to worry about the underlying implementation, but you need to punch Sean Cribbs (very nicely) about that. Or you could just use Faraday instead of em-synchrony all along, since Faraday has a backend for em-http/synchrony already. Hope this helps. Cheers, Mathias http://riakhandbook.com [1]: https://github.com/basho/riak-ruby-client/blob/master/lib/riak/client/net_http_backend.rb [2]: https://github.com/technoweenie/faraday On Friday, 16. March 2012 at 05:39, Sean McKibben wrote: > Given Mathias Meyer's talk at Scotland Ruby Conference about eventmachine > programming (while wearing a Riak t-shirt!), I was hoping to see a little bit > more in terms of eventmachine clients and Riak. > > Has anyone used EventMachine and/or em-synchrony with Riak and could give me > some advice? I'm using ruby-riak-client at this point and going the route of > trying to wrap my workflow in fibers so riak client plays nice with it. > > Am I better off just using a HTTP client like EM::HttpRequest or > EM::Synchrony::Multi, or is there some good way to use ruby-riak-client or > ripple with eventmachine that requires less manual intervention? > > Sorry if this has been covered somewhere else but I haven't had much luck > finding anyone else using EM with Riak. > > Thanks, > Sean McKibben > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Adoption - What can we do better?
Thanks for the praise of my book. I'm curious though, what does "advanced" entail for you guys? I'm continuously working on updates for the book, and I'm happy to look at things that you think are missing, but it'd be great to have some more concrete examples of what you think a book on advanced Riak usage should cover. Thanks! Cheers, Mathias On Thursday, 19. April 2012 at 19:57, Alexander Sicular wrote: > +1 for Mathias' book. It rocks. But I still think that an even more advanced > book will only get so much of an audience. > > The masses like to be shown. They don't like to read. Frankly, that means > screencasts. > > > @siculars on twitter > http://siculars.posterous.com > > Sent from my iRotaryPhone > > On Apr 19, 2012, at 13:54, "Jason J. W. Williams" (mailto:jasonjwwilli...@gmail.com)> wrote: > > > On Thu, Apr 19, 2012 at 11:37 AM, Zabrane Mickael > (mailto:zabra...@gmail.com)> wrote: > > > +1 for a solid book on advanced Riak usage. > > > > > > > > Better marketing of Mathias' excellent Riak Handbook would definitely > > help...would be great if it showed up on Amazon at least if not > > bn.com (http://bn.com). > > > > Someday replication making it into the community version would be > > really nice...it's one of the few areas where Cassandra has a leg up. > > Makes it easier to sell/stand up a resilient proof-of-concept across > > availability zones and then buy support when the installation is grown > > to support more production load. > > > > -J > > > > ___ > > riak-users mailing list > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > > ___ > riak-users mailing list > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
