Why not push the data (or references to it) to a queue (e.g. RabbitMQ) and then run single-threaded consumers that work well with PBC? That would also decouple the processes and allow you to scale them independently.
-jd 2012/7/26 Parnell Springmeyer <ixma...@gmail.com> > I'm using Riak in a 5 node cluster with LevelDB for the backend (we store > A LOT of archivable data) on FreeBSD. > > The data is mapped out as follows: I have a set of database objects that > are closely linked to user accounts - I needed to be able to compose > complex queries on these objects including joins with the user data; so it > made sense to keep those objects in MySQL. > > I have software that takes those database objects and produces DAILY stats > for each object (so we have months/years of data for each database object). > These stats are what we store in Riak. > > Now, that application also updates the MySQL database object with the key > under which the stat object is stored in Riak for quick and easy compiling > of the "latest" data (since it's just a GET operation and not a M/R job). > > Mashing up this data for small sets of MySQL database objects is quick and > painless. But once it starts approaching > 1000 objects I notice it slows > to a crawl and i notice Riak being pegged pretty hard (IOW it is Riak's > response). > > Now; here's the issue: with my web application I haven't figured out how > to use the RiakPBC connector - so we are going through the HTTP API. I have > a feeling this is where that bottle neck is occurring. > > Why do you ask? Because our Python web app is multi-threaded and the PBC > sockets don't play nice here. I'm not finding my experiments to solve this > very successful. So I wanted to ask the greater community if anyone HAS or > is willing to HELP me solve it! > > Thanks :) > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
