Is Riak right for me?
Riak is very appealing for several reasons; scalability, durability, open-source, performance etc. I'm currently using PostgreSQL for all my storage needs, but I'm investigating nosql (can I use that name?) solutions for scalability and to experiment with map/reduce functionality for statistics and reporting. I have a few requirements that nosql solutions might not be able to meet. 1) Every second I take a snapshot of my data and store it in the database in one record. Each recorded snapshot includes the timestamp it was taken. 2) I have a playback feature that lets me retrieve historical data. During playback, the browser requests a recorded snapshot every second: 2011-01-01 08:00:00 2011-01-01 08:00:01 2011-01-01 08:00:02 2011-01-01 08:00:03 ... Currently it takes less than 75ms for the server to retrieve the data from PostgreSQL and to return it to the browser. Some processing is done before the response is sent. 3) Every second each client's browser requests the current data snapshot (i.e. not in playback mode). The same comment for timing and processing applies from 2). 4) Every minute I retrieve statistics and a report for a specific type of data to present on the browser. Currently with PostgreSQL this takes about 2-3s for the web server to retrieve the data, process it and return it to the browser. The only primary key I use is a serial integer, only because that's the default. I don't see anything in my data that would be useful as a key when using a key/value database. My data is a good fit for storing as a 'document' though. I know there might not be enough information here but do you think Riak is a good fit? Thanks Mike ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Is Riak right for me?
Cool - some very useful information. Much appreciated. I need to digest this a bit before replying with more questions. I'm still trying to understand map/reduce so I'm struggling to get my head around how to apply it to my requirements. I'm obviously used to indexing. :) Thanks Mike On Fri, Feb 11, 2011 at 9:23 AM, Sean Cribbs wrote: > Mike, > > Sounds like an interesting project. Here are some things to think about > (corresponding to your bullet points): > > 1) What seems fairly natural and obvious for key choice is the timestamp, > since so many of your operations are time-oriented. > 2) For playback of historical data, consider using MapReduce to grab more > than just single seconds of data... maybe 30 at a time. You could even put > some of your preprocessing into the map or reduce phase. Make sure to > generate the key list (since you'll know them) instead of trying to do a > full-bucket query with filtering. > 3) Beware of race-conditions and the possibility of not all clients seeing > the data right away. This can be somewhat alleviated by using DW=W=quorum > when writing, but you're still talking about dogpiling a bunch of requests on > the same key. An in-memory write-through cache of the "latest second" might > be what you need here. > 4) This is another case where you could use MapReduce to crunch the data. 60 > items is not very much, so I think you'll have good results here. The > internal MapReduce cache will also reduce the pain of multiple computations > on the same data. > > All in all, I think Riak will be a good fit for your application, with the > possible exception of the polling-every-second thing. A couple of tips to > make sure you have a good experience with Riak: > > First, benchmark your usage pattern as best you can to make sure that Riak > will meet your performance needs. For example, I might create some > basho_bench tests with appropriate key and value generators that have: > > a) 1 write per second (the snapshot data) > b) X reads per second (where X is the number of expected clients) > c) 1-5 historical replays per second (via MapReduce) > d) X roll-up reports per minute (X = number of clients again) > > I'd then run them concurrently, and in different combinations to simulate the > load. > > Second, make sure you start with at least 3 nodes (even in your local > developer setup). Because Riak is designed to be distributed, there are > certain things that are sub-optimal when the number of nodes is less than the > replication factor (N value, default 3). > > Let us know if there's anything else we can help you with. > > Sean Cribbs > Developer Advocate > Basho Technologies, Inc. > http://basho.com/ > > On Feb 11, 2011, at 8:49 AM, Mike Stoddart wrote: > >> Riak is very appealing for several reasons; scalability, durability, >> open-source, performance etc. I'm currently using PostgreSQL for all >> my storage needs, but I'm investigating nosql (can I use that name?) >> solutions for scalability and to experiment with map/reduce >> functionality for statistics and reporting. >> >> I have a few requirements that nosql solutions might not be able to meet. >> >> 1) Every second I take a snapshot of my data and store it in the >> database in one record. Each recorded snapshot includes the timestamp >> it was taken. >> >> 2) I have a playback feature that lets me retrieve historical data. >> During playback, the browser requests a recorded snapshot every >> second: >> >> 2011-01-01 08:00:00 >> 2011-01-01 08:00:01 >> 2011-01-01 08:00:02 >> 2011-01-01 08:00:03 ... >> >> Currently it takes less than 75ms for the server to retrieve the data >> from PostgreSQL and to return it to the browser. Some processing is >> done before the response is sent. >> >> 3) Every second each client's browser requests the current data >> snapshot (i.e. not in playback mode). The same comment for timing and >> processing applies from 2). >> >> 4) Every minute I retrieve statistics and a report for a specific type >> of data to present on the browser. Currently with PostgreSQL this >> takes about 2-3s for the web server to retrieve the data, process it >> and return it to the browser. >> >> The only primary key I use is a serial integer, only because that's >> the default. I don't see anything in my data that would be useful as a >> key when using a key/value database. My data is a good fit for storing >> as a 'document' though. >> >> I know there might not be enough information here but do you think >> Riak is a good fit? >> >> Thanks >> Mike >> >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak compile error on Fedora 12 (32 bit)
I'm trying to compile Riak on Fedora 12 (32 bit) but I'm getting an error: make rel fatal: Not a git repository (or any of the parent directories): .git ./rebar get-deps fatal: Not a git repository (or any of the parent directories): .git fatal: Not a git repository (or any of the parent directories): .git fatal: Not a git repository (or any of the parent directories): .git ==> cluster_info (get-deps) ==> skerl (get-deps) ==> protobuffs (get-deps) ==> mochiweb (get-deps) ==> webmachine (get-deps) ==> riak_core (get-deps) ==> riakc (get-deps) ==> luke (get-deps) ==> erlang_js (get-deps) ==> ebloom (get-deps) ==> bitcask (get-deps) ==> eper (get-deps) ==> riak_kv (get-deps) ==> luwak (get-deps) ==> riak_err (get-deps) ==> rel (get-deps) ==> riak-0.14.0 (get-deps) ./rebar compile generate fatal: Not a git repository (or any of the parent directories): .git fatal: Not a git repository (or any of the parent directories): .git fatal: Not a git repository (or any of the parent directories): .git ==> cluster_info (compile) ==> skerl (compile) ==> protobuffs (compile) ==> mochiweb (compile) ==> webmachine (compile) ==> riak_core (compile) ==> riakc (compile) ==> luke (compile) ==> erlang_js (compile) Running make -C c_src make[1]: Entering directory `/home/mike/Downloads/riak-0.14.0/deps/erlang_js/c_src' make[1]: Nothing to be done for `js'. make[1]: Leaving directory `/home/mike/Downloads/riak-0.14.0/deps/erlang_js/c_src' ==> ebloom (compile) ==> bitcask (compile) ==> eper (compile) ==> riak_kv (compile) ==> luwak (compile) ==> riak_err (compile) ==> rel (compile) ==> riak-0.14.0 (compile) ==> rel (generate) ERROR: Unable to generate spec: Undefined applications: [inets] I'm stumped - any suggestions? Thanks Mike ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak compile error on Fedora 12 (32 bit)
Thanks David, I already had erlang-inets installed. I even tried reinstalling it but it didn't make any difference. Mike On Fri, Feb 11, 2011 at 9:39 PM, David Smith wrote: > On Fri, Feb 11, 2011 at 6:05 PM, Mike Stoddart wrote: >> I'm trying to compile Riak on Fedora 12 (32 bit) but I'm getting an error: >> ==> rel (generate) >> ERROR: Unable to generate spec: Undefined applications: [inets] > > Try installing the erlang-inets package via yum. > > It's unfortunate they break up Erlang into so many tiny packages -- > I'm not sure what the rationale is for it (particularly in a > loosely-coupled language like Erlang). > > D. > > -- > Dave Smith > Engineering Manager > Basho Technologies, Inc. > diz...@basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak compile error on Fedora 12 (32 bit)
Thanks. It's an FC13 RPM so I don't know if it will work on FC12 but I'll try it tomorrow night. Thanks Mike -- Sent from my Palm Pre On 12 Feb 2011 12:33 a.m., David Smith <diz...@basho.com> wrote: It might not be perfect, but I spun up a VM + buildslave for fedora and build an RPM this evening: http://dl.dropbox.com/u/2596651/riak-0.14.0.18.gce2ab2c-1.fc13.i686.rpm Maybe that works better? Or is at least functional? :) D. On Fri, Feb 11, 2011 at 7:43 PM, Mike Stoddart <sto...@gmail.com> wrote: > Thanks David, I already had erlang-inets installed. I even tried > reinstalling it but it didn't make any difference. > > Mike > > On Fri, Feb 11, 2011 at 9:39 PM, David Smith <diz...@basho.com> wrote: >> On Fri, Feb 11, 2011 at 6:05 PM, Mike Stoddart <sto...@gmail.com> wrote: >>> I'm trying to compile Riak on Fedora 12 (32 bit) but I'm getting an error: >>> ==> rel (generate) >>> ERROR: Unable to generate spec: Undefined applications: [inets] >> >> Try installing the erlang-inets package via yum. >> >> It's unfortunate they break up Erlang into so many tiny packages -- >> I'm not sure what the rationale is for it (particularly in a >> loosely-coupled language like Erlang). >> >> D. >> >> -- >> Dave Smith >> Engineering Manager >> Basho Technologies, Inc. >> diz...@basho.com >> > -- Dave Smith Engineering Manager Basho Technologies, Inc. diz...@basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak compile error on Fedora 12 (32 bit)
Good to know! I assumed library diiferences would fail the install. Thanks Mike -- Sent from my Palm Pre On 12 Feb 2011 9:37 a.m., David Smith <diz...@basho.com> wrote: I tried out a FC13 on 12 the other day and it _seemed_ to work fine. But that was hardly a rigorous test... :) D. On Sat, Feb 12, 2011 at 7:33 AM, Mike Stoddart <sto...@gmail.com> wrote: > Thanks. It's an FC13 RPM so I don't know if it will work on FC12 but I'll > try it tomorrow night. > > Thanks > Mike > > > > -- Sent from my Palm Pre > > On 12 Feb 2011 12:33 a.m., David Smith <diz...@basho.com> wrote: > > It might not be perfect, but I spun up a VM + buildslave for fedora > and build an RPM this evening: > > http://dl.dropbox.com/u/2596651/riak-0.14.0.18.gce2ab2c-1.fc13.i686.rpm > > Maybe that works better? Or is at least functional? :) > > D. > > On Fri, Feb 11, 2011 at 7:43 PM, Mike Stoddart <sto...@gmail.com> wrote: >> Thanks David, I already had erlang-inets installed. I even tried >> reinstalling it but it didn't make any difference. >> >> Mike >> >> On Fri, Feb 11, 2011 at 9:39 PM, David Smith <diz...@basho.com> wrote: >>> On Fri, Feb 11, 2011 at 6:05 PM, Mike Stoddart <sto...@gmail.com> wrote: >>>> I'm trying to compile Riak on Fedora 12 (32 bit) but I'm getting an >>>> error: >>>> ==> rel (generate) >>>> ERROR: Unable to generate spec: Undefined applications: [inets] >>> >>> Try installing the erlang-inets package via yum. >>> >>> It's unfortunate they break up Erlang into so many tiny packages -- >>> I'm not sure what the rationale is for it (particularly in a >>> loosely-coupled language like Erlang). >>> >>> D. >>> >>> -- >>> Dave Smith >>> Engineering Manager >>> Basho Technologies, Inc. >>> diz...@basho.com >>> >> > > > > -- > Dave Smith > Engineering Manager > Basho Technologies, Inc. > diz...@basho.com > -- Dave Smith Engineering Manager Basho Technologies, Inc. diz...@basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak compile error on Fedora 12 (32 bit)
Well I'll be it installed and service riak start works. So far so good! Thanks Mike On Sat, Feb 12, 2011 at 9:37 AM, David Smith wrote: > I tried out a FC13 on 12 the other day and it _seemed_ to work fine. > But that was hardly a rigorous test... :) > > D. > > On Sat, Feb 12, 2011 at 7:33 AM, Mike Stoddart wrote: >> Thanks. It's an FC13 RPM so I don't know if it will work on FC12 but I'll >> try it tomorrow night. >> >> Thanks >> Mike >> >> >> >> -- Sent from my Palm Pre >> >> On 12 Feb 2011 12:33 a.m., David Smith wrote: >> >> It might not be perfect, but I spun up a VM + buildslave for fedora >> and build an RPM this evening: >> >> http://dl.dropbox.com/u/2596651/riak-0.14.0.18.gce2ab2c-1.fc13.i686.rpm >> >> Maybe that works better? Or is at least functional? :) >> >> D. >> >> On Fri, Feb 11, 2011 at 7:43 PM, Mike Stoddart wrote: >>> Thanks David, I already had erlang-inets installed. I even tried >>> reinstalling it but it didn't make any difference. >>> >>> Mike >>> >>> On Fri, Feb 11, 2011 at 9:39 PM, David Smith wrote: >>>> On Fri, Feb 11, 2011 at 6:05 PM, Mike Stoddart wrote: >>>>> I'm trying to compile Riak on Fedora 12 (32 bit) but I'm getting an >>>>> error: >>>>> ==> rel (generate) >>>>> ERROR: Unable to generate spec: Undefined applications: [inets] >>>> >>>> Try installing the erlang-inets package via yum. >>>> >>>> It's unfortunate they break up Erlang into so many tiny packages -- >>>> I'm not sure what the rationale is for it (particularly in a >>>> loosely-coupled language like Erlang). >>>> >>>> D. >>>> >>>> -- >>>> Dave Smith >>>> Engineering Manager >>>> Basho Technologies, Inc. >>>> diz...@basho.com >>>> >>> >> >> >> >> -- >> Dave Smith >> Engineering Manager >> Basho Technologies, Inc. >> diz...@basho.com >> > > > > -- > Dave Smith > Engineering Manager > Basho Technologies, Inc. > diz...@basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Python client performance issue
I added some code to my system to test writing data into Riak. I'm
using the Python client library with protocol buffers. I'm writing a
snapshot of my current data, which is one json object containing on
average 60 individual json sub-objects. Each sub object contains about
22 values.
# Archived entry. ts is a formatted timestamp.
entry = self._bucket.new(ts, data=data)
entry.store()
# Now write the current entry.
entry = self._bucket.new("current", data=data)
entry.store()
I'm writing the same data twice; the archived copy and the current
copy, which I can easily retrieve later. Performance is lower than
expected; top is showing a constant cpu usage of 10-12%.
I haven't decided to use Riak; this is to help me decide. But for now
are there any optimisations I can do here? A similiar test with
MongoDB shows a steady cpu usage of 1%. The cpu usages are for my
client, not Riak's own processes. The only differences in my test app
is the code that writes the data to the database. Otherwise all other
code is 100% the same between these two test apps.
Any suggestions appreciated.
Thanks
Mike
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Python client performance issue
Will do when I get time. Would the REST API be any faster?
Thanks
Mike
On Mon, Feb 14, 2011 at 7:01 PM, Thomas Burdick
wrote:
> I would highly recommend looking in to the cProfile and pstat module and
> profile the code that is going slow. If your using the protocol buffer
> client it could possibly be related to the fact that python protocol buffers
> is extraordinarily slow and is well known to be slow. Profile until proven
> guilty though.
> Tom Burdick
>
> On Mon, Feb 14, 2011 at 7:09 AM, Mike Stoddart wrote:
>>
>> I added some code to my system to test writing data into Riak. I'm
>> using the Python client library with protocol buffers. I'm writing a
>> snapshot of my current data, which is one json object containing on
>> average 60 individual json sub-objects. Each sub object contains about
>> 22 values.
>>
>> # Archived entry. ts is a formatted timestamp.
>> entry = self._bucket.new(ts, data=data)
>> entry.store()
>>
>> # Now write the current entry.
>> entry = self._bucket.new("current", data=data)
>> entry.store()
>>
>> I'm writing the same data twice; the archived copy and the current
>> copy, which I can easily retrieve later. Performance is lower than
>> expected; top is showing a constant cpu usage of 10-12%.
>>
>> I haven't decided to use Riak; this is to help me decide. But for now
>> are there any optimisations I can do here? A similiar test with
>> MongoDB shows a steady cpu usage of 1%. The cpu usages are for my
>> client, not Riak's own processes. The only differences in my test app
>> is the code that writes the data to the database. Otherwise all other
>> code is 100% the same between these two test apps.
>>
>> Any suggestions appreciated.
>> Thanks
>> Mike
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak instead of memcached?
I was thinking of using memcached for cheap, fast storage to share between servers. I need to store stuff like user preferences, permissions, session etc. But memcached (I think) requires you to define which servers you want to use for storage. Ideally I don't care. I want my core/auth server to write preferences and permissions to the cache and let tornado on other servers pull those keys out. I think Riak could perform this in theory, but is Riak fast enough? ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak instead of memcached?
Thanks. I like Riak's flexibility. Just throw another server into the mix and Riak adjusts accordingly. With memcached I'd have to change code or configuration and restart to see the new machine. I could locate a small Riak database on servers that need permissions/preferences/etc. I don't think I need sessions. I guess I could memcached Riak on each machine for reads! Haha, that kind of stuff just makes me chuckle... :) Thanks Mike On Wed, Feb 16, 2011 at 7:48 PM, Jason J. W. Williams wrote: > My benchmarks with Riak against Redis showed Riak is fast, but it's > never going to be as fast for caching as an in-memory only store that > doesn't have to do coordination. That said if your data set is larger > than RAM, Riak will likely be faster as it will scale to handle that > better. > > -J > > On Wed, Feb 16, 2011 at 5:33 PM, Mike Stoddart wrote: >> I was thinking of using memcached for cheap, fast storage to share >> between servers. I need to store stuff like user preferences, >> permissions, session etc. But memcached (I think) requires you to >> define which servers you want to use for storage. Ideally I don't >> care. I want my core/auth server to write preferences and permissions >> to the cache and let tornado on other servers pull those keys out. I >> think Riak could perform this in theory, but is Riak fast enough? >> >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak instead of memcached?
I'm not familiar with those - will do some reading. Thanks Mike On Wed, Feb 16, 2011 at 8:38 PM, Joshua Partogi wrote: > Why don't you use Membase/Couchbase instead? It provides the same > capability as memcached but adds clustering capability on top of it. > > On Thu, Feb 17, 2011 at 11:33 AM, Mike Stoddart wrote: >> I was thinking of using memcached for cheap, fast storage to share >> between servers. I need to store stuff like user preferences, >> permissions, session etc. But memcached (I think) requires you to >> define which servers you want to use for storage. Ideally I don't >> care. I want my core/auth server to write preferences and permissions >> to the cache and let tornado on other servers pull those keys out. I >> think Riak could perform this in theory, but is Riak fast enough? > > > > -- > http://twitter.com/jpartogi > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
riak-search build assumes access to github using 'git' protocol
I'm trying to build riak-serach at work but our corporate firewall blocks the 'git' protocol. I can only download via http or https. I've tried changing rebar.config to use the URLs for https instead but I get errors: fatal: Not a git repository (or any of the parent directories): .git Any ideas how to get this to build on Fedora 12 64bit? Thanks ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: riak-search build assumes access to github using 'git' protocol
I was running make rel using the tar ball but it kept complaining
about git as I mentioned. I tried ./rebar compile but I get:
==> erlang_js (compile)
Compiled src/js_drv_comm.erl
Compiled src/js_benchmark.erl
Compiled src/js_memory.erl
Compiled src/erlang_js.erl
Compiled src/erlang_js_sup.erl
Compiled src/js.erl
Compiled src/js_mochinum.erl
Compiled src/js_cache.erl
Compiled src/js_driver.erl
Compiled src/js_mochijson2.erl
ERROR: compile failed while processing
/home/mike/download/riak_search-0.14.0/deps/erlang_js:
{'EXIT',{function_clause,[{lists,thing_to_list,[{error,bad_name}]},
{lists,flatmap,2},
{lists,flatmap,2},
{rebar_port_compiler,default_env,0},
{rebar_port_compiler,compile,2},
{rebar_core,run_modules,4},
{rebar_core,execute,4},
{rebar_core,process_dir,4}]}}
Thanks
Mike
On Thu, Feb 17, 2011 at 12:03 PM, Jon Meredith wrote:
> Did you try downloading the tarball from riak_search-0.14.0-1.tar.gz and
> running make?
> If that fails you could try running
> ./rebar compile
> ./rebar rel
> And that should build the release in rel/riak
> BR, Jon.
>
> On Thu, Feb 17, 2011 at 9:57 AM, Mike Stoddart wrote:
>>
>> I'm trying to build riak-serach at work but our corporate firewall
>> blocks the 'git' protocol. I can only download via http or https. I've
>> tried changing rebar.config to use the URLs for https instead but I
>> get errors:
>>
>> fatal: Not a git repository (or any of the parent directories): .git
>>
>> Any ideas how to get this to build on Fedora 12 64bit?
>>
>> Thanks
>>
>> ___
>> riak-users mailing list
>> riak-users@lists.basho.com
>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak compile error on Fedora 12 (32 bit)
David, would you mind building an RPM for Riak Search as well? That would be most appreciated. Could you add the FC13 RPMs to the official download page or are older systems frowned upon? Thanks Mike On Sat, Feb 12, 2011 at 9:37 AM, David Smith wrote: > I tried out a FC13 on 12 the other day and it _seemed_ to work fine. > But that was hardly a rigorous test... :) > > D. > > On Sat, Feb 12, 2011 at 7:33 AM, Mike Stoddart wrote: >> Thanks. It's an FC13 RPM so I don't know if it will work on FC12 but I'll >> try it tomorrow night. >> >> Thanks >> Mike >> >> >> >> -- Sent from my Palm Pre >> >> On 12 Feb 2011 12:33 a.m., David Smith wrote: >> >> It might not be perfect, but I spun up a VM + buildslave for fedora >> and build an RPM this evening: >> >> http://dl.dropbox.com/u/2596651/riak-0.14.0.18.gce2ab2c-1.fc13.i686.rpm >> >> Maybe that works better? Or is at least functional? :) >> >> D. >> >> On Fri, Feb 11, 2011 at 7:43 PM, Mike Stoddart wrote: >>> Thanks David, I already had erlang-inets installed. I even tried >>> reinstalling it but it didn't make any difference. >>> >>> Mike >>> >>> On Fri, Feb 11, 2011 at 9:39 PM, David Smith wrote: >>>> On Fri, Feb 11, 2011 at 6:05 PM, Mike Stoddart wrote: >>>>> I'm trying to compile Riak on Fedora 12 (32 bit) but I'm getting an >>>>> error: >>>>> ==> rel (generate) >>>>> ERROR: Unable to generate spec: Undefined applications: [inets] >>>> >>>> Try installing the erlang-inets package via yum. >>>> >>>> It's unfortunate they break up Erlang into so many tiny packages -- >>>> I'm not sure what the rationale is for it (particularly in a >>>> loosely-coupled language like Erlang). >>>> >>>> D. >>>> >>>> -- >>>> Dave Smith >>>> Engineering Manager >>>> Basho Technologies, Inc. >>>> diz...@basho.com >>>> >>> >> >> >> >> -- >> Dave Smith >> Engineering Manager >> Basho Technologies, Inc. >> diz...@basho.com >> > > > > -- > Dave Smith > Engineering Manager > Basho Technologies, Inc. > diz...@basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Python client performance issue
I have a sneaky suspicion it's the json encoder. I used json myself to
encode the data before passing it to bucket.new() (not even sure if
this is valid!) and the cpu usage dropped to 1%.
On Tue, Feb 15, 2011 at 1:05 PM, Andy Gross wrote:
>
>
> Sorry, I should have been more clear. The Python client depends on
> "protobuf>=2.3.0" in setup.py, so people are already most likely using
> protobufs-2.3.0.
> - Andy
>
> On Tuesday, February 15, 2011 at 3:09 AM, Nico Meyer wrote:
>
> Hi Andy.
>
> I am not quite sure what you mean, is the protobuf library included with
> riak-python-client? Or are you talking about the version of the protobuf
> compiler that was used to create riakclient_pb2.py from
> riakclient.proto?
>
> Cheers,
> Nico
>
> Am Dienstag, den 15.02.2011, 02:23 -0800 schrieb Andy Gross:
>
> python-riak-client already uses version 2.3.0. Adventurous types
> might want to check out https://github.com/Greplin/fast-python-pb,
> which wraps the C/C++ protocol buffers library.
>
> --
> Andy Gross
> Principal Architect
> Basho Technologies, Inc.
>
>
> On Tuesday, February 15, 2011 at 1:46 AM, Nico Meyer wrote:
>
> Hi Mike,
>
> perhaps you can try to upgrade the protocol buffers library to at
> least
> version 2.3.0. This is from the changelog for that version:
>
> Python
> * 10-25 times faster than 2.2.0, still pure-Python.
>
>
> Cheers,
> Nico
>
> Am Montag, den 14.02.2011, 19:35 -0500 schrieb Mike Stoddart:
>
> Will do when I get time. Would the REST API be any faster?
>
> Thanks
> Mike
>
> On Mon, Feb 14, 2011 at 7:01 PM, Thomas Burdick
> wrote:
>
> I would highly recommend looking in to the cProfile and pstat
> module and
> profile the code that is going slow. If your using the protocol
> buffer
> client it could possibly be related to the fact that python
> protocol buffers
> is extraordinarily slow and is well known to be slow. Profile
> until proven
> guilty though.
> Tom Burdick
>
> On Mon, Feb 14, 2011 at 7:09 AM, Mike Stoddart
> wrote:
>
> I added some code to my system to test writing data into Riak.
> I'm
> using the Python client library with protocol buffers. I'm
> writing a
> snapshot of my current data, which is one json object
> containing on
> average 60 individual json sub-objects. Each sub object
> contains about
> 22 values.
>
> # Archived entry. ts is a formatted timestamp.
> entry = self._bucket.new(ts, data=data)
> entry.store()
>
> # Now write the current entry.
> entry = self._bucket.new("current", data=data)
> entry.store()
>
> I'm writing the same data twice; the archived copy and the
> current
> copy, which I can easily retrieve later. Performance is lower
> than
> expected; top is showing a constant cpu usage of 10-12%.
>
> I haven't decided to use Riak; this is to help me decide. But
> for now
> are there any optimisations I can do here? A similiar test
> with
> MongoDB shows a steady cpu usage of 1%. The cpu usages are for
> my
> client, not Riak's own processes. The only differences in my
> test app
> is the code that writes the data to the database. Otherwise
> all other
> code is 100% the same between these two test apps.
>
> Any suggestions appreciated.
> Thanks
> Mike
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
> ___
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak compile error on Fedora 12 (32 bit)
Great thanks! Any chance you could put up a 64 bit riak search for FC12/13 as well please? Sorry to be a nuisance! Thanks Mike On Thu, Feb 17, 2011 at 4:15 PM, David Smith wrote: > http://downloads.basho.com/riak-search/CURRENT/riak-search-0.14.0-1.fc13.i686.rpm > > FWIW, we will have Fedora 32/64 bit builds moving forward. > > D. > > On Thu, Feb 17, 2011 at 12:17 PM, Mike Stoddart wrote: >> David, would you mind building an RPM for Riak Search as well? That >> would be most appreciated. Could you add the FC13 RPMs to the official >> download page or are older systems frowned upon? >> >> Thanks >> Mike >> >> On Sat, Feb 12, 2011 at 9:37 AM, David Smith wrote: >>> I tried out a FC13 on 12 the other day and it _seemed_ to work fine. >>> But that was hardly a rigorous test... :) >>> >>> D. >>> >>> On Sat, Feb 12, 2011 at 7:33 AM, Mike Stoddart wrote: >>>> Thanks. It's an FC13 RPM so I don't know if it will work on FC12 but I'll >>>> try it tomorrow night. >>>> >>>> Thanks >>>> Mike >>>> >>>> >>>> >>>> -- Sent from my Palm Pre >>>> >>>> On 12 Feb 2011 12:33 a.m., David Smith wrote: >>>> >>>> It might not be perfect, but I spun up a VM + buildslave for fedora >>>> and build an RPM this evening: >>>> >>>> http://dl.dropbox.com/u/2596651/riak-0.14.0.18.gce2ab2c-1.fc13.i686.rpm >>>> >>>> Maybe that works better? Or is at least functional? :) >>>> >>>> D. >>>> >>>> On Fri, Feb 11, 2011 at 7:43 PM, Mike Stoddart wrote: >>>>> Thanks David, I already had erlang-inets installed. I even tried >>>>> reinstalling it but it didn't make any difference. >>>>> >>>>> Mike >>>>> >>>>> On Fri, Feb 11, 2011 at 9:39 PM, David Smith wrote: >>>>>> On Fri, Feb 11, 2011 at 6:05 PM, Mike Stoddart wrote: >>>>>>> I'm trying to compile Riak on Fedora 12 (32 bit) but I'm getting an >>>>>>> error: >>>>>>> ==> rel (generate) >>>>>>> ERROR: Unable to generate spec: Undefined applications: [inets] >>>>>> >>>>>> Try installing the erlang-inets package via yum. >>>>>> >>>>>> It's unfortunate they break up Erlang into so many tiny packages -- >>>>>> I'm not sure what the rationale is for it (particularly in a >>>>>> loosely-coupled language like Erlang). >>>>>> >>>>>> D. >>>>>> >>>>>> -- >>>>>> Dave Smith >>>>>> Engineering Manager >>>>>> Basho Technologies, Inc. >>>>>> diz...@basho.com >>>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Dave Smith >>>> Engineering Manager >>>> Basho Technologies, Inc. >>>> diz...@basho.com >>>> >>> >>> >>> >>> -- >>> Dave Smith >>> Engineering Manager >>> Basho Technologies, Inc. >>> diz...@basho.com >>> >> > > > > -- > Dave Smith > Engineering Manager > Basho Technologies, Inc. > diz...@basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Python client performance issue
Much better - using simplejson and return_body=False reduces CPU usage
to 0.5-0.7%. Which is more consistent with other DBs.
Thanks for your help!
Mike
On Mon, Feb 14, 2011 at 8:09 AM, Mike Stoddart wrote:
> I added some code to my system to test writing data into Riak. I'm
> using the Python client library with protocol buffers. I'm writing a
> snapshot of my current data, which is one json object containing on
> average 60 individual json sub-objects. Each sub object contains about
> 22 values.
>
> # Archived entry. ts is a formatted timestamp.
> entry = self._bucket.new(ts, data=data)
> entry.store()
>
> # Now write the current entry.
> entry = self._bucket.new("current", data=data)
> entry.store()
>
> I'm writing the same data twice; the archived copy and the current
> copy, which I can easily retrieve later. Performance is lower than
> expected; top is showing a constant cpu usage of 10-12%.
>
> I haven't decided to use Riak; this is to help me decide. But for now
> are there any optimisations I can do here? A similiar test with
> MongoDB shows a steady cpu usage of 1%. The cpu usages are for my
> client, not Riak's own processes. The only differences in my test app
> is the code that writes the data to the database. Otherwise all other
> code is 100% the same between these two test apps.
>
> Any suggestions appreciated.
> Thanks
> Mike
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Can't start Riak Search
I installed the new FC13 Riak Search 64bit RPM (onto FC12) but I'm
having issues. I uninstalled Riak and deleted the old database files.
I had to manually create the riaksearch user/group as well as the
/var/lib/riaksearch/bitcask and /var/lib/riaksearch/ring directories.
=INFO REPORT 18-Feb-2011::09:34:13 ===
Spidermonkey VM (thread stack: 8MB, max heap: 8MB, pool:
riak_kv_js_hook) host starting (<0.168.0>)
=ERROR REPORT 18-Feb-2011::09:34:13 ===
** Generic server riak_kv_vnode_master terminating
** Last message in was {0,get_vnode}
** When Server state == {state,12308,[],undefined,riak_kv_vnode,
riak_kv_legacy_vnode}
** Reason for termination ==
** {{badmatch,{error,{{badmatch,{error,eacces}},
[{bitcask,open,2},
{riak_kv_bitcask_backend,start,2},
{riak_kv_vnode,init,1},
{riak_core_vnode,init,1},
{gen_fsm,init_it,6},
{proc_lib,init_p_do_apply,3}]}}},
[{riak_core_vnode_master,get_vnode,2},
{riak_core_vnode_master,handle_call,3},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}
=ERROR REPORT 18-Feb-2011::09:34:13 ===
** gen_event handler riak_core_ring_handler crashed.
** Was installed in riak_core_ring_events
** Last event was: {ring_update,
{chstate,'riak@127.0.0.1',[],
{64,
[{0,'riak@127.0.0.1'},
{22835963083295358096932575511191922182123945984,
'riak@127.0.0.1'},
{45671926166590716193865151022383844364247891968,
'riak@127.0.0.1'},
{68507889249886074290797726533575766546371837952,
'riak@127.0.0.1'},
{91343852333181432387730302044767688728495783936,
'riak@127.0.0.1'},
{114179815416476790484662877555959610910619729920,
'riak@127.0.0.1'},
{137015778499772148581595453067151533092743675904,
'riak@127.0.0.1'},
{159851741583067506678528028578343455274867621888,
'riak@127.0.0.1'},
{182687704666362864775460604089535377456991567872,
'riak@127.0.0.1'},
{205523667749658222872393179600727299639115513856,
'riak@127.0.0.1'},
{228359630832953580969325755111919221821239459840,
'riak@127.0.0.1'},
{25119559391624893906625833062344003363405824,
'riak@127.0.0.1'},
{274031556999544297163190906134303066185487351808,
'riak@127.0.0.1'},
{296867520082839655260123481645494988367611297792,
'riak@127.0.0.1'},
{319703483166135013357056057156686910549735243776,
'riak@127.0.0.1'},
{342539446249430371453988632667878832731859189760,
'riak@127.0.0.1'},
{365375409332725729550921208179070754913983135744,
'riak@127.0.0.1'},
{388211372416021087647853783690262677096107081728,
'riak@127.0.0.1'},
{411047335499316445744786359201454599278231027712,
'riak@127.0.0.1'},
{433883298582611803841718934712646521460354973696,
'riak@127.0.0.1'},
{456719261665907161938651510223838443642478919680,
'riak@127.0.0.1'},
{479555224749202520035584085735030365824602865664,
'riak@127.0.0.1'},
{50239118783249787813251666124688006726811648,
'riak@127.0.0.1'},
{525227150915793236229449236757414210188850757632,
'riak@127.0.0.1'},
{548063113999088594326381812268606132370974703616,
'riak@127.0.0.1'},
{570899077082383952423314387779798054553098649600,
'riak@127.0.0.1'},
{593735040165679310520246963290989976735222595584,
'riak@127.0.0.1'},
{616571003248974668617179538802181898917346541568,
'riak@127.0.0.1'},
Re: Can't start Riak Search
Riak Search initially complained that the riakuser didn't exist so I created
it. There's no riak user either so I'll create and check the config.
Thanks
Mike
-- Sent from my Palm Pre
On 18 Feb 2011 9:43 a.m., Sean Cribbs <s...@basho.com> wrote:
The "eaccess" error comes from either lacking file permissions or the proper
directory structure. In this case, it looks like bitcask could not access the
data directory. Make sure those directories exist and belong to the "riak"
user (there's no "riaksearch" user).
Sean Cribbs <s...@basho.com>
Developer Advocate
Basho Technologies, Inc.
http://basho.com/
On Feb 18, 2011, at 9:38 AM, Mike Stoddart wrote:
> I installed the new FC13 Riak Search 64bit RPM (onto FC12) but I'm
> having issues. I uninstalled Riak and deleted the old database files.
> I had to manually create the riaksearch user/group as well as the
> /var/lib/riaksearch/bitcask and /var/lib/riaksearch/ring directories.
>
> =INFO REPORT 18-Feb-2011::09:34:13 ===
> Spidermonkey VM (thread stack: 8MB, max heap: 8MB, pool:
> riak_kv_js_hook) host starting (<0.168.0>)
>
> =ERROR REPORT 18-Feb-2011::09:34:13 ===
> ** Generic server riak_kv_vnode_master terminating
> ** Last message in was {0,get_vnode}
> ** When Server state == {state,12308,[],undefined,riak_kv_vnode,
> riak_kv_legacy_vnode}
> ** Reason for termination ==
> ** {{badmatch,{error,{{badmatch,{error,eacces}},
> [{bitcask,open,2},
> {riak_kv_bitcask_backend,start,2},
> {riak_kv_vnode,init,1},
> {riak_core_vnode,init,1},
> {gen_fsm,init_it,6},
> {proc_lib,init_p_do_apply,3}]}}},
>[{riak_core_vnode_master,get_vnode,2},
> {riak_core_vnode_master,handle_call,3},
> {gen_server,handle_msg,5},
> {proc_lib,init_p_do_apply,3}]}
>
> =ERROR REPORT 18-Feb-2011::09:34:13 ===
> ** gen_event handler riak_core_ring_handler crashed.
> ** Was installed in riak_core_ring_events
> ** Last event was: {ring_update,
> {chstate,'riak@127.0.0.1',[],
> {64,
>[{0,'riak@127.0.0.1'},
>
{22835963083295358096932575511191922182123945984,
> 'riak@127.0.0.1'},
>
{45671926166590716193865151022383844364247891968,
> 'riak@127.0.0.1'},
>
{68507889249886074290797726533575766546371837952,
> 'riak@127.0.0.1'},
>
{91343852333181432387730302044767688728495783936,
> 'riak@127.0.0.1'},
>
{114179815416476790484662877555959610910619729920,
> 'riak@127.0.0.1'},
>
{137015778499772148581595453067151533092743675904,
> 'riak@127.0.0.1'},
>
{159851741583067506678528028578343455274867621888,
> 'riak@127.0.0.1'},
>
{182687704666362864775460604089535377456991567872,
> 'riak@127.0.0.1'},
>
{205523667749658222872393179600727299639115513856,
> 'riak@127.0.0.1'},
>
{228359630832953580969325755111919221821239459840,
> 'riak@127.0.0.1'},
>
{25119559391624893906625833062344003363405824,
> 'riak@127.0.0.1'},
>
{274031556999544297163190906134303066185487351808,
> 'riak@127.0.0.1'},
>
{296867520082839655260123481645494988367611297792,
> 'riak@127.0.0.1'},
>
{319703483166135013357056057156686910549735243776,
> 'riak@127.0.0.1'},
>
{342539446249430371453988632667878832731859189760,
> 'riak@127.0.0.1'},
>
{365375409332725729550921208179070754913983135744,
> 'riak@127.0.0.1'},
>
{388211372416021087647853783690262677096107081
Re: Can't start Riak Search
No worries - I fully understand and appreciate it all the same. :)
On Fri, Feb 18, 2011 at 10:24 AM, David Smith wrote:
> I should note that we've not had a chance to validate these fedora
> packages to the degree I'd like -- this was simply an
> early-morning-coffee-sipping-I'm-gonna-try-and-build-packages
> approach. :) When 0.14.1 lands, we should have sorted out these
> issues.
>
> Thanks,
>
> D.
>
> On Fri, Feb 18, 2011 at 8:22 AM, Mike Stoddart wrote:
>> Riak Search initially complained that the riakuser didn't exist so I created
>> it. There's no riak user either so I'll create and check the config.
>>
>> Thanks
>> Mike
>>
>>
>>
>> -- Sent from my Palm Pre
>>
>> On 18 Feb 2011 9:43 a.m., Sean Cribbs wrote:
>>
>> The "eaccess" error comes from either lacking file permissions or the proper
>> directory structure. In this case, it looks like bitcask could not access
>> the data directory. Make sure those directories exist and belong to the
>> "riak" user (there's no "riaksearch" user).
>>
>> Sean Cribbs
>> Developer Advocate
>> Basho Technologies, Inc.
>> http://basho.com/
>>
>> On Feb 18, 2011, at 9:38 AM, Mike Stoddart wrote:
>>
>>> I installed the new FC13 Riak Search 64bit RPM (onto FC12) but I'm
>>> having issues. I uninstalled Riak and deleted the old database files.
>>> I had to manually create the riaksearch user/group as well as the
>>> /var/lib/riaksearch/bitcask and /var/lib/riaksearch/ring directories.
>>>
>>> =INFO REPORT 18-Feb-2011::09:34:13 ===
>>> Spidermonkey VM (thread stack: 8MB, max heap: 8MB, pool:
>>> riak_kv_js_hook) host starting (<0.168.0>)
>>>
>>> =ERROR REPORT 18-Feb-2011::09:34:13 ===
>>> ** Generic server riak_kv_vnode_master terminating
>>> ** Last message in was {0,get_vnode}
>>> ** When Server state == {state,12308,[],undefined,riak_kv_vnode,
>>> riak_kv_legacy_vnode}
>>> ** Reason for termination ==
>>> ** {{badmatch,{error,{{badmatch,{error,eacces}},
>>> [{bitcask,open,2},
>>> {riak_kv_bitcask_backend,start,2},
>>> {riak_kv_vnode,init,1},
>>> {riak_core_vnode,init,1},
>>> {gen_fsm,init_it,6},
>>> {proc_lib,init_p_do_apply,3}]}}},
>>> [{riak_core_vnode_master,get_vnode,2},
>>> {riak_core_vnode_master,handle_call,3},
>>> {gen_server,handle_msg,5},
>>> {proc_lib,init_p_do_apply,3}]}
>>>
>>> =ERROR REPORT 18-Feb-2011::09:34:13 ===
>>> ** gen_event handler riak_core_ring_handler crashed.
>>> ** Was installed in riak_core_ring_events
>>> ** Last event was: {ring_update,
>>> {chstate,'riak@127.0.0.1',[],
>>> {64,
>>> [{0,'riak@127.0.0.1'},
>>> {22835963083295358096932575511191922182123945984,
>>> 'riak@127.0.0.1'},
>>> {45671926166590716193865151022383844364247891968,
>>> 'riak@127.0.0.1'},
>>> {68507889249886074290797726533575766546371837952,
>>> 'riak@127.0.0.1'},
>>> {91343852333181432387730302044767688728495783936,
>>> 'riak@127.0.0.1'},
>>> {114179815416476790484662877555959610910619729920,
>>> 'riak@127.0.0.1'},
>>> {137015778499772148581595453067151533092743675904,
>>> 'riak@127.0.0.1'},
>>> {159851741583067506678528028578343455274867621888,
>>> 'riak@127.0.0.1'},
>>> {182687704666362864775460604089535377456991567872,
>>> 'riak@127.0.0.1'},
>>> {205523667749658222872393179600727299639115513856,
>>> 'riak@127.0.0.1'},
>>> {228359630832953580969325755111919221821239459840,
>>> 'riak@127.0.0.1'},
>>> {25119559391624893906625833062344003363405824,
>>> 'riak@127.0.0.1'},
>>> {274031556999544297163190906134303066185487351808,
>>> 'riak@127.0.0.1'},
>>> {296867520082839655260123481645494988367611297792,
>>> 'riak@127.0.0.1'},
>>> {319703483166135013357056057156686910549735243776,
>>> 'riak@127.0.0.1'},
>>> {342539446249430371453988632667878832731859189760,
>>> 'riak@127.0.0.1'},
>>> {365375409332725729550921208179070754913983135744,
>>> 'riak@127.0.0.1'},
>>> {388211372416021087647853783690262677096107081728,
>>> 'riak@127.0.0.1'},
>>> {4110473354993164457
Riak Search - querying numerical fields (No JSON object could be decoded)
My documents are structured like this:
{
timestamp: '2011-02-02 08:00:00',
count: 40,
data {
}
}
I want to query all documents where count > 40 so in Python I did:
search_query = client.search('bucket', 'count:[40:*]')
search_query.run()
I got an error:
python test_get.py
Traceback (most recent call last):
File "test_get.py", line 11, in
search_query.run()
File "build/bdist.linux-x86_64/egg/riak/mapreduce.py", line 184, in run
File "build/bdist.linux-x86_64/egg/riak/transports/http.py", line
194, in mapred
File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.6/json/decoder.py", line 338, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Is this a problem with how I'm storing my data or is something not
configured properly?
Thanks
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Search - querying numerical fields (No JSON object could be decoded)
My mistake - my query format was wrong, [40 TO 100] works fine. Except
run() returns a bunch of lists. Each list contains the bucket name and
the timestamp. More reading required.
Thanks
Mike
On Fri, Feb 18, 2011 at 11:51 AM, Mike Stoddart wrote:
> My documents are structured like this:
>
> {
> timestamp: '2011-02-02 08:00:00',
> count: 40,
> data {
>
>
> }
> }
>
> I want to query all documents where count > 40 so in Python I did:
>
> search_query = client.search('bucket', 'count:[40:*]')
> search_query.run()
>
> I got an error:
>
> python test_get.py
> Traceback (most recent call last):
> File "test_get.py", line 11, in
> search_query.run()
> File "build/bdist.linux-x86_64/egg/riak/mapreduce.py", line 184, in run
> File "build/bdist.linux-x86_64/egg/riak/transports/http.py", line
> 194, in mapred
> File "/usr/lib64/python2.6/json/__init__.py", line 307, in loads
> return _default_decoder.decode(s)
> File "/usr/lib64/python2.6/json/decoder.py", line 319, in decode
> obj, end = self.raw_decode(s, idx=_w(s, 0).end())
> File "/usr/lib64/python2.6/json/decoder.py", line 338, in raw_decode
> raise ValueError("No JSON object could be decoded")
> ValueError: No JSON object could be decoded
>
> Is this a problem with how I'm storing my data or is something not
> configured properly?
>
> Thanks
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak search question (Python)
I'm experimenting with Riak Search; I'm storing tweets and then trying
to query them. My code to store a tweet is:
tweet = bucket.new(str(uuid.uuid1()), data={
'name': s.user.name,
'text': s.text,
})
tweet.store()
My code to search the tweets is:
search_query = client.search('tweets', 'name:[andy TO john]')
I've tried many different queries including:
search_query = client.search('tweets', 'text:and')
But I never get any hits. Am I doing something fundamentally wrong? I
print out the tweet as I add it to the database so I know what terms I
can search on. But nothing works. I did remember to do:
serach-cmd install tweets
Any suggestions appreciated.
Thanks
Mike
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
[no subject]
The Python client connects to one specific node. The downside to this is that the client has to have knowledge of the node and the node has to be up. If that pre-configured node goes down the client is stuck. What options are there for letting a client connect to the cluster through any node? Thanks Mike -- Sent from my Palm Pre ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak search timestamps (date/time)
Does Riak Search support timestamps? The majority of my data is queried using timestamps; e.g. the last three hours of data or the snapshot from Jan 20th at 08:00:01 etc. Thanks Mike ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: about riak search
I found this confusing as well but eventually worked it out. Will the two merge someday? Thanks Mike On Tue, Feb 22, 2011 at 9:08 AM, Sean Cribbs wrote: > Wilson, > > You only need to run one of them. Riak Search includes all of the > functionality of Riak KV. > > Sean Cribbs > Developer Advocate > Basho Technologies, Inc. > http://basho.com/ > > On Feb 22, 2011, at 9:03 AM, wilson Tuladhar wrote: > >> Hi, >> >> I have a problem with the riak search functionality. >> >> The problem is I can start riak and riaksearch independently but i cannot >> run them simultaneously. If i start riak db first then i cannot start riak >> search and vice-versa. Is it a problem or am i doing something wrong??? >> >> /Wilson >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > > ___ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak & Timestamps (Python)
Just want to quickly clarify how to store timestamps in Riak Search using Python. My code: dt = datetime.datetime.strptime(data['Time(PDT)'], "%b %d, %Y %H:%M:%S") ut = time.mktime(dt.timetuple()) data['Time(PDT)'] = int(ut) # Save the object to Riak. json = simplejson.dumps(data) obj = bucket.new(data['Time(PDT)'], data=json) obj.store(return_body=False) This is giving me: TypeError: 1275598830 has type , but expected one of: (,) Am I misunderstanding something fundamental here? Thanks Mike ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak & Timestamps (Python)
Ah thanks - I thought it was complaining about the value. On Sat, Apr 2, 2011 at 9:23 PM, Rexxe wrote: > Bucket keys are strings and you're passing an int (data['Time(PDT)'] which > you set to int(ut)) > > On Sat, Apr 2, 2011 at 6:13 PM, Mike Stoddart wrote: >> >> Just want to quickly clarify how to store timestamps in Riak Search >> using Python. My code: >> >> dt = datetime.datetime.strptime(data['Time(PDT)'], "%b %d, >> %Y %H:%M:%S") >> ut = time.mktime(dt.timetuple()) >> data['Time(PDT)'] = int(ut) >> >> # Save the object to Riak. >> json = simplejson.dumps(data) >> obj = bucket.new(data['Time(PDT)'], data=json) >> obj.store(return_body=False) >> >> This is giving me: >> >> TypeError: 1275598830 has type , but expected one of: >> (,) >> >> Am I misunderstanding something fundamental here? >> >> Thanks >> Mike >> >> ___ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Riak Search Unicode / String TypeError (Python client)
I wrote a simple Python client to insert data but when I try to
perform a query, I get:
Traceback (most recent call last):
File "./search.py", line 29, in
row = result.get()
File "build/bdist.linux-x86_64/egg/riak/mapreduce.py", line 302, in get
File "build/bdist.linux-x86_64/egg/riak/bucket.py", line 245, in get
File "build/bdist.linux-x86_64/egg/riak/riak_object.py", line 290, in reload
File "build/bdist.linux-x86_64/egg/riak/transports/pbc.py", line 152, in get
File "build/bdist.macosx-10.6-universal/egg/google/protobuf/reflection.py",
line 510, in setter
File
"build/bdist.macosx-10.6-universal/egg/google/protobuf/internal/type_checkers.py",
line 91, in CheckValue
TypeError: u'events' has type , but expected one of:
(,)
My query is:
search_query = client.search('events', 'name:Fred')
Is it complaining about the bucket name? I had to manually build protobuf.
Thanks
Mike
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak Search Unicode / String TypeError (Python client)
Using the following python script to load the basketball tweets, I can
do bucket.get(key), which works. client.search(), doesn't work for me,
get_data() returns None:
#!/usr/bin/python
import csv
import riak
import simplejson
import sys
import datetime
import time
import uuid
# Connect to Riak.
client = riak.RiakClient(port=8087, transport_class=riak.RiakPbcTransport)
def load_csv2(bucket_name, filename):
# Choose the bucket to store data in.
bucket = client.bucket(bucket_name)
c = csv.reader(open(filename))
old_fields = c.next()
fields = []
for f in old_fields:
if f == 'Time(PDT)':
f = 'timestamp'
nf = f.lower()
fields.append(nf)
print fields
for row in c:
data = {
}
for ii in range(0,len(fields)-1):
data[fields[ii]] = row[ii]
ts = data['timestamp']
dt = datetime.datetime.strptime(data['timestamp'], "%b %d, %Y
%H:%M:%S")
ut = time.mktime(dt.timetuple())
data['timestamp'] = int(ut)
id = str(uuid.uuid1())
print dt, id
obj = bucket.new(id, data=data)
obj.store(return_body=False)
#~ sys.exit(1)
load_csv2('twitter', 'game1.csv')
#~ load_csv2('games', 'game2.csv')
Anyone using Riak Search with Python?
Thanks
Mike
On Mon, Apr 4, 2011 at 11:17 AM, Mike Stoddart wrote:
> I wrote a simple Python client to insert data but when I try to
> perform a query, I get:
>
> Traceback (most recent call last):
> File "./search.py", line 29, in
> row = result.get()
> File "build/bdist.linux-x86_64/egg/riak/mapreduce.py", line 302, in get
> File "build/bdist.linux-x86_64/egg/riak/bucket.py", line 245, in get
> File "build/bdist.linux-x86_64/egg/riak/riak_object.py", line 290, in reload
> File "build/bdist.linux-x86_64/egg/riak/transports/pbc.py", line 152, in get
> File "build/bdist.macosx-10.6-universal/egg/google/protobuf/reflection.py",
> line 510, in setter
> File
> "build/bdist.macosx-10.6-universal/egg/google/protobuf/internal/type_checkers.py",
> line 91, in CheckValue
> TypeError: u'events' has type , but expected one of:
> (,)
>
> My query is:
>
> search_query = client.search('events', 'name:Fred')
>
> Is it complaining about the bucket name? I had to manually build protobuf.
>
> Thanks
> Mike
>
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Strange results from numerical range query in Riak Search
I'm installing the lakes/celtics twitter data linked through Basho's
site. I can get some basic text searches to work, but I'm seeing some
weird results from numerical range queries:
search_query = client.search('stuff', 'followers:[140 TO 150]')
Name:Kevin Shpunt Term:lakers Followers:146
Name:Carlos Ordonez Term:lakers Followers:15
Name:Kelvin Shortt Jr Term:lakers,celtics Followers:140
Name:Stanley T. Davis Term:lakers Followers:15
Name:John Robertson Term:lakers Followers:141
Name:Steven Wineglass Term:lakers Followers:1493
Name:Badral Jambal Term:lakers Followers:1432
Am I doing something wrong?
Thanks
Mike
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
error: unpacking of archive failed on file /usr/lib64/riak/erts-5.7.5/bin/beam; 4d53dbb2: cpio: Digest mismatch
I'm trying to install Riak on Fedora 12 using the RPM from Basho's download site. I'm getting a digest mismatch error - does this mean my Fedora is subtly different from the one used to build the original RPM? Thanks Mike yum install --nogpgcheck ~mike//download/downloads.basho.com/riak/CURRENT/riak-0.14.0-1.fc12.x86_64.rpm Loaded plugins: presto, refresh-packagekit Setting up Install Process Examining /home/mike//download/downloads.basho.com/riak/CURRENT/riak-0.14.0-1.fc12.x86_64.rpm: riak-0.14.0-1.fc12.x86_64 Marking /home/mike//download/downloads.basho.com/riak/CURRENT/riak-0.14.0-1.fc12.x86_64.rpm to be installed Resolving Dependencies --> Running transaction check ---> Package riak.x86_64 0:0.14.0-1.fc12 set to be installed --> Finished Dependency Resolution Dependencies Resolved = Package ArchVersion Repository Size = Installing: riak x86_64 0.14.0-1.fc12 /riak-0.14.0-1.fc12.x86_64 31 M Transaction Summary = Install 1 Package(s) Total size: 31 M Installed size: 31 M Is this ok [y/N]: y Downloading Packages: Running rpm_check_debug Running Transaction Test Transaction Test Succeeded Running Transaction Installing : riak-0.14.0-1.fc12.x86_64 1/1 Error unpacking rpm package riak-0.14.0-1.fc12.x86_64 error: unpacking of archive failed on file /usr/lib64/riak/erts-5.7.5/bin/beam;4d53dbb2: cpio: Digest mismatch Failed: riak.x86_64 0:0.14.0-1.fc12 Complete! [root@wv1 ~]# yum update Loaded plugins: presto, refresh-packagekit Setting up Update Process No Packages marked for Update [root@wv1 ~]# yum upgrade Loaded plugins: presto, refresh-packagekit Setting up Upgrade Process No Packages marked for Update ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: error: unpacking of archive failed on file /usr/lib64/riak/erts-5.7.5/bin/beam; 4d53dbb2: cpio: Digest mismatch
I noticed there's another version for Fedora 12: 0.14.0-2. I tried the RPM and it works. Thanks Mike On Thu, Feb 10, 2011 at 9:24 AM, David Smith wrote: > On Thu, Feb 10, 2011 at 5:38 AM, Mike Stoddart wrote: >> I'm trying to install Riak on Fedora 12 using the RPM from Basho's >> download site. I'm getting a digest mismatch error - does this mean my >> Fedora is subtly different from the one used to build the original >> RPM? > > Hi Mike... > > No, this is a red herring (on Redhat, haha). The mismatch is a result > of the Erlang build we currently have on the Fedora buildslave not > complying with some packaging requirements in Fedora. We are actively > working on addressing this, but in the interim I've uploaded a manual > build on Fedora 13: > > http://downloads.basho.com/riak/CURRENT/riak-0.14.0-1.fc13.x86_64.rpm > > I'm not sure if that works with Fedora 12 -- if it doesn't let me know > and I'll spin up a Fedora 12 system and try again. :) > > Thanks! > > D. > > -- > Dave Smith > Engineering Manager > Basho Technologies, Inc. > diz...@basho.com > ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
