Oh, sorry I misunderstood. I thought celery workers were threads within the same Python interpreter. By all means have one client per Python process, but see if you can keep it around between tasks. If the throughput on your task queue is high enough, you should be able to make use of the benefit of persistent connections and load-balancing.
On Tue, Aug 6, 2013 at 8:10 PM, Matt Black <matt.bl...@jbadigital.com>wrote: > Hi Sean, > > I would indeed like to take advantage of the pooling features of the new > client. Shared objects across Celery workers isn't something I'd ever > really looked at before - but it seems the only real way to share across > workers (ie, processes) is to use memcache or similar. Which makes sense. > > Cheers > Matt > > > > On 6 August 2013 23:04, Sean Cribbs <s...@basho.com> wrote: > >> Hi Matt, >> >> You are correct about the Decaying class, it is a wrapper for an >> exponentially-decaying error rate. The idea is, you configure your client >> to connect to multiple Riak nodes, and if one goes down or is restarted, >> the possibility of its selection for new connections can be automatically >> reduced by detecting network errors. Of course, this is moot when you are >> connecting to the cluster via a load-balancer; at that point the >> error-tracking logic can't help you. >> >> If you are creating and throwing away RiakClient objects frequently, >> neither the error-tracking nor the connection pool will help you much. >> However, I urge you to consider keeping a single client around, in some >> globally accessible place (module constant? config object?). Your >> concurrent workers will get to take advantage of existing connections, and >> your socket/file-handle usage will be less. >> >> >> On Tue, Aug 6, 2013 at 1:07 AM, Matt Black <matt.bl...@jbadigital.com>wrote: >> >>> I've been looking through the changes to the RiakClient class - and have >>> a question about the nodes list and RiakNode/Decaying class. >>> >>> It seems that the Decaying class is used internally to track error_rates >>> (request failure_rate?) of nodes in the available pool. >>> >>> In the application I maintain, we're accessing Riak in many concurrent >>> celery workers - so my assumption would be that this internal error_rate >>> tracking wouldn't do much good. Have I got this right? Is the client's >>> internal "nodes" list only really useful in a long running application? >>> (ie, when I'm not instantiating RiakClient fresh for each query). >>> >>> We currently have a "pool" of Riak nodes in some configuration, from >>> which a random IP is chosen for each request. >>> >>> Thanks >>> >>> >>> >>> >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >> >> >> -- >> Sean Cribbs <s...@basho.com> >> Software Engineer >> Basho Technologies, Inc. >> http://basho.com/ >> > > -- Sean Cribbs <s...@basho.com> Software Engineer Basho Technologies, Inc. http://basho.com/
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
