John, I don't really see a scenario where that would be a useful solution - that's a lot of overhead to add a message queuing system into the mix when my client should be able to handle the connection(s) just fine.
I think the primary issue comes from the Python client not pooling Riak connections. On Jul 26, 2012, at 5:39 PM, John D. Rowell wrote: > Why not push the data (or references to it) to a queue (e.g. RabbitMQ) and > then run single-threaded consumers that work well with PBC? That would also > decouple the processes and allow you to scale them independently. > > -jd > > 2012/7/26 Parnell Springmeyer <ixma...@gmail.com> > I'm using Riak in a 5 node cluster with LevelDB for the backend (we store A > LOT of archivable data) on FreeBSD. > > The data is mapped out as follows: I have a set of database objects that are > closely linked to user accounts - I needed to be able to compose complex > queries on these objects including joins with the user data; so it made sense > to keep those objects in MySQL. > > I have software that takes those database objects and produces DAILY stats > for each object (so we have months/years of data for each database object). > These stats are what we store in Riak. > > Now, that application also updates the MySQL database object with the key > under which the stat object is stored in Riak for quick and easy compiling of > the "latest" data (since it's just a GET operation and not a M/R job). > > Mashing up this data for small sets of MySQL database objects is quick and > painless. But once it starts approaching > 1000 objects I notice it slows to > a crawl and i notice Riak being pegged pretty hard (IOW it is Riak's > response). > > Now; here's the issue: with my web application I haven't figured out how to > use the RiakPBC connector - so we are going through the HTTP API. I have a > feeling this is where that bottle neck is occurring. > > Why do you ask? Because our Python web app is multi-threaded and the PBC > sockets don't play nice here. I'm not finding my experiments to solve this > very successful. So I wanted to ask the greater community if anyone HAS or is > willing to HELP me solve it! > > Thanks :) > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > >
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
