Geez .... what a HUGE duh moment. This will ultimately run on multiple remote servers, but, while doing initial testing, I have been running it on just my dev box. And, of course, I keep forgetting that twisted (well, python) doesn't make use of multiple cores. I wasted all yesterday trying to figure out why this was running the way it was..... A million thanks.
Sean On Fri, Jul 3, 2009 at 7:52 AM, Johann Borck <johann.bo...@densedata.com>wrote: > Sean Hollingsworth wrote: > > Let me start off by warning everyone that I am extremely new to > > twisted (I've been using it maybe a week) and it's highly likely some > > things don't work the way I think they should. Or, I'm just using them > > incorrectly. > > > > [..] > > > > The problems: > > > > Right now, for development and testing, I have three servers running > > on localhost on three ports. > Are you running them in a single process? > > When I run my test code, it seems that each Perspective Broker runs > > sequentially, rather than concurrently. For example, if i watch the > > debugging output from the server, I see the following: > > > > server 0 processing dataval1 > > server 0 processing dataval2 > > server 0 processing dataval3 > > server 0 processing dataval4 > > server 1 processing dataval*5* > > server 1 processing dataval6 > > server 1 processing dataval7 > > server 2 processing dataval8 > > server 2 processing dataval9 > > server 2 processing dataval10 > > > > My understanding is that the perspective brokers would work > > concurrently. Is that incorrect? My guess is that they should work > > concurrently and I am just doing something wrong in my code, due to my > > very, very limited understanding of how they work. Well, how much of > > twisted works, really. > > > > > Below is my the relevant code for the client (I've taken out code that > > just deals with prepping data or debugging). Please keep in mind that > > this is mostly testing code, while I get a better understanding of how > > perspective brokers work and was cobbled together from examples and > > docs found online. > > > > [...] > > With the client code, nothing seems wrong, since it doesn't enforce one > job to be finished before starting the next one - the problem has to be > on the server side. Actually I suspect you're running the same service > on 3 ports but within one process, right?. If this is the case it's no > wonder the order of jobs completed matches exactly the order they're > arriving in because it all happens in a single thread. If you want > parallel processing you will have to use multiple processes. As a side > note (which doesn't seem to apply here): if the work they're doing is > not CPU-bound, but IO-bound (meaning the work is nothing but waiting for > something to complete, which for some reason can't be done in an > asynchronous manner, the most prominent examples being blocking database > interfaces), you also have the alternative of using threads. > > I think the basic problem is you're assuming twisted to somehow schedule > the work to different cores/cpus, which it doesn't do by default. You > might want to look at ampoule https://launchpad.net/ampoule, which is a > process pool, there was another similar thing called AsynQueue, but the > site is down atm: http://foss.eepatents.com/AsynQueue. > > hth, > Johann > > > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python >
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
