Re: [Twisted-Python] untwisting twistd
On 2 Jul 2009, at 23:50, Mikhail wrote:
In my example I should have written
run(application)
instead of
run(...)
Then the difference would be more apparent. Namely, in my 'wrapper'
it is explicit what application will be run and where all the services
came from, your 'wrapper' is just a customized version of twistd and
what will be run depends on the command line and what will be found
in the file system. In some cases I'd like to _explicitly_ code into
main script what functionality my application provides and I do not
want twistd search file system for plugins at all.
I'm not using plugins, but I had the same requirement as you to
explicitly create the app and then run it - because I need the freeze
scripts to make a single application.
My solution looked something like:
# myapp.py
from twisted.scripts._twistd_unix import UnixApplicationRunner
# Of course, that's just because the app is unix only
class MyRunner(UnixApplicationRunner):
def createOrGetApplication(self):
# The stuff that's usually in a tac file
application = # ...
return application
def main():
setup_logging()
config = { # stuff pinched by dumping 'config' during a twistd run
}
# Also do the ServerOptions thing here if necessary
MyRunner(config).run()
if __name__ == '__main__'
main()
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Perspective broker remote calls to multiple servers seem to run sequentially
Sean Hollingsworth wrote: > Let me start off by warning everyone that I am extremely new to > twisted (I've been using it maybe a week) and it's highly likely some > things don't work the way I think they should. Or, I'm just using them > incorrectly. > > [..] > > The problems: > > Right now, for development and testing, I have three servers running > on localhost on three ports. Are you running them in a single process? > When I run my test code, it seems that each Perspective Broker runs > sequentially, rather than concurrently. For example, if i watch the > debugging output from the server, I see the following: > > server 0 processing dataval1 > server 0 processing dataval2 > server 0 processing dataval3 > server 0 processing dataval4 > server 1 processing dataval*5* > server 1 processing dataval6 > server 1 processing dataval7 > server 2 processing dataval8 > server 2 processing dataval9 > server 2 processing dataval10 > > My understanding is that the perspective brokers would work > concurrently. Is that incorrect? My guess is that they should work > concurrently and I am just doing something wrong in my code, due to my > very, very limited understanding of how they work. Well, how much of > twisted works, really. > > Below is my the relevant code for the client (I've taken out code that > just deals with prepping data or debugging). Please keep in mind that > this is mostly testing code, while I get a better understanding of how > perspective brokers work and was cobbled together from examples and > docs found online. > [...] With the client code, nothing seems wrong, since it doesn't enforce one job to be finished before starting the next one - the problem has to be on the server side. Actually I suspect you're running the same service on 3 ports but within one process, right?. If this is the case it's no wonder the order of jobs completed matches exactly the order they're arriving in because it all happens in a single thread. If you want parallel processing you will have to use multiple processes. As a side note (which doesn't seem to apply here): if the work they're doing is not CPU-bound, but IO-bound (meaning the work is nothing but waiting for something to complete, which for some reason can't be done in an asynchronous manner, the most prominent examples being blocking database interfaces), you also have the alternative of using threads. I think the basic problem is you're assuming twisted to somehow schedule the work to different cores/cpus, which it doesn't do by default. You might want to look at ampoule https://launchpad.net/ampoule, which is a process pool, there was another similar thing called AsynQueue, but the site is down atm: http://foss.eepatents.com/AsynQueue. hth, Johann ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Perspective broker remote calls to multiple servers seem to run sequentially
Geez what a HUGE duh moment. This will ultimately run on multiple remote servers, but, while doing initial testing, I have been running it on just my dev box. And, of course, I keep forgetting that twisted (well, python) doesn't make use of multiple cores. I wasted all yesterday trying to figure out why this was running the way it was. A million thanks. Sean On Fri, Jul 3, 2009 at 7:52 AM, Johann Borck wrote: > Sean Hollingsworth wrote: > > Let me start off by warning everyone that I am extremely new to > > twisted (I've been using it maybe a week) and it's highly likely some > > things don't work the way I think they should. Or, I'm just using them > > incorrectly. > > > > [..] > > > > The problems: > > > > Right now, for development and testing, I have three servers running > > on localhost on three ports. > Are you running them in a single process? > > When I run my test code, it seems that each Perspective Broker runs > > sequentially, rather than concurrently. For example, if i watch the > > debugging output from the server, I see the following: > > > > server 0 processing dataval1 > > server 0 processing dataval2 > > server 0 processing dataval3 > > server 0 processing dataval4 > > server 1 processing dataval*5* > > server 1 processing dataval6 > > server 1 processing dataval7 > > server 2 processing dataval8 > > server 2 processing dataval9 > > server 2 processing dataval10 > > > > My understanding is that the perspective brokers would work > > concurrently. Is that incorrect? My guess is that they should work > > concurrently and I am just doing something wrong in my code, due to my > > very, very limited understanding of how they work. Well, how much of > > twisted works, really. > > > > > Below is my the relevant code for the client (I've taken out code that > > just deals with prepping data or debugging). Please keep in mind that > > this is mostly testing code, while I get a better understanding of how > > perspective brokers work and was cobbled together from examples and > > docs found online. > > > > [...] > > With the client code, nothing seems wrong, since it doesn't enforce one > job to be finished before starting the next one - the problem has to be > on the server side. Actually I suspect you're running the same service > on 3 ports but within one process, right?. If this is the case it's no > wonder the order of jobs completed matches exactly the order they're > arriving in because it all happens in a single thread. If you want > parallel processing you will have to use multiple processes. As a side > note (which doesn't seem to apply here): if the work they're doing is > not CPU-bound, but IO-bound (meaning the work is nothing but waiting for > something to complete, which for some reason can't be done in an > asynchronous manner, the most prominent examples being blocking database > interfaces), you also have the alternative of using threads. > > I think the basic problem is you're assuming twisted to somehow schedule > the work to different cores/cpus, which it doesn't do by default. You > might want to look at ampoule https://launchpad.net/ampoule, which is a > process pool, there was another similar thing called AsynQueue, but the > site is down atm: http://foss.eepatents.com/AsynQueue. > > hth, > Johann > > > > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Twisted Trial changes working directory
I have a (Twisted) program that saves & uses a file in the current working directory. When running tests for the program, Trial changes the working directory, causing this file to be written into the '_trial_temp' directory instead. ...so the program works differently when run via Trial vs. in normal use. There are a number of solutions to this, but I'd just like to understand the reasoning behind this surprising behaviour. The docs say that this: "...allows them to write whatever data they like to disk, and not worry about polluting the current working directory" ...couldn't they just do this without changing the working directory? One of the things I love about python is that things just seem to work like you expect - I'm rarely surprised by it's behaviour. Twisted, OTOH, continually surprises me...and not like getting-a-pony-on-your-10th-birthday type of surprise. Don't get me wrong, I love Twisted...it makes writing networked/asynchronous programs a breeze. ...everything I write now (in Python), just seems to "need" Twisted. I just find Twisted's idiosyncrasies frustrating sometimes. Sorry for the rant, Gerrat ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] (Mis)indented Posts
...ok, I've created a new post twice on this list twice, and both times my post has been indented and placed under a completely different/irrelevant post Here: http://twistedmatrix.com/pipermail/twisted-python/2009-June/019859.html And Here: http://twistedmatrix.com/pipermail/twisted-python/2009-July/019902.html ...if this post hijacks a thread as well, then it will be 3 times. ...this seems to be something specific to the twisted python mailing list: http://www.mail-archive.com/mailman-users%40python.org/msg53996.html Gerrat ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted Trial changes working directory
On Fri, 3 Jul 2009 10:17:12 -0400, Gerrat Rickert wrote: >I have a (Twisted) program that saves & uses a file in the current >working directory. >When running tests for the program, Trial changes the working directory, >causing this file to be written into the '_trial_temp' directory >instead. >...so the program works differently when run via Trial vs. in normal >use. The only difference I can infer from your description is whether the file in question is created in the directory you invoke the program from (in the non-trial case) or in the _trial_temp subdirectory of that directory (in the trial case). It's not obvious to me why this might be a problem. > >There are a number of solutions to this, but I'd just like to understand >the reasoning behind this surprising behaviour. >The docs say that this: "...allows them to write whatever data they like >to disk, and not worry about polluting the current working directory" >...couldn't they just do this without changing the working directory? This is an ancient feature of trial, from the days when Twisted's own test suite (the primary driver of trial development for many years) was even more crufty than it is today. In those days, many tests just created files in the working directory. The _trial_temp feature provided a single fix which addressed the mess created by all of the tests in one fell swoop. The behavior is still convenient for tests which aren't written carefully in this regard. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] (Mis)indented Posts (list administrativia)
On Fri, 3 Jul 2009 10:32:07 -0400, Gerrat Rickert wrote: >...ok, I've created a new post twice on this list twice, and both times >my post has been indented and placed under a completely >different/irrelevant post > >Here: >http://twistedmatrix.com/pipermail/twisted-python/2009-June/019859.html >And Here: >http://twistedmatrix.com/pipermail/twisted-python/2009-July/019902.html > >...if this post hijacks a thread as well, then it will be 3 times. > >...this seems to be something specific to the twisted python mailing >list: >http://www.mail-archive.com/mailman-users%40python.org/msg53996.html > I don't really know much about how mailman generates those threaded views. twisted-python is running on an unmodified mailman 1:2.1.9-9ubuntu1 install. Perhaps this could be caused by exim4 configuration somehow (we do have various exim4 configuration customizations), but that's the only thing I can think of. Any pointers on what information is used to determine the threading structure of messages? Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] Twisted Trial changes working directory
>-Original Message- >From: twisted-python-boun...@twistedmatrix.com [mailto:twisted-python- >boun...@twistedmatrix.com] On Behalf Of Jean-Paul Calderone >Sent: Friday, July 03, 2009 11:27 AM >To: Twisted general discussion >Subject: Re: [Twisted-Python] Twisted Trial changes working directory > >On Fri, 3 Jul 2009 10:17:12 -0400, Gerrat Rickert > wrote: >>I have a (Twisted) program that saves & uses a file in the current >>working directory. >>When running tests for the program, Trial changes the working >directory, >>causing this file to be written into the '_trial_temp' directory >>instead. >>...so the program works differently when run via Trial vs. in normal >>use. > >The only difference I can infer from your description is whether the >file >in question is created in the directory you invoke the program from (in >the non-trial case) or in the _trial_temp subdirectory of that directory >(in the trial case). It's not obvious to me why this might be a >problem. ...and it turns out not to really be a Trial-specific problem - more a problem with my expectations. The file takes a while to generate, and I just expected it to be there after running the program the first time. - it will get regenerated if it doesn't exist, and I can easily copy it to the _trial_temp directory for testing > >> >>There are a number of solutions to this, but I'd just like to >understand >>the reasoning behind this surprising behaviour. >>The docs say that this: "...allows them to write whatever data they >like >>to disk, and not worry about polluting the current working directory" >>...couldn't they just do this without changing the working directory? > >This is an ancient feature of trial, from the days when Twisted's own >test >suite (the primary driver of trial development for many years) was even >more crufty than it is today. In those days, many tests just created >files >in the working directory. The _trial_temp feature provided a single fix >which addressed the mess created by all of the tests in one fell swoop. >The >behavior is still convenient for tests which aren't written carefully in >this regard. > >Jean-Paul Thanks for the explanation. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] (Mis)indented Posts (list administrativia)
>-Original Message- >From: Jean-Paul Calderone [mailto:exar...@divmod.com] >Sent: Friday, July 03, 2009 11:49 AM >To: Twisted general discussion >Subject: Re: [Twisted-Python] (Mis)indented Posts (list administrativia) > >On Fri, 3 Jul 2009 10:32:07 -0400, Gerrat Rickert > wrote: >>...ok, I've created a new post twice on this list twice, and both times >>my post has been indented and placed under a completely >>different/irrelevant post >> >>Here: >>http://twistedmatrix.com/pipermail/twisted-python/2009-June/019859.htm l >>And Here: >>http://twistedmatrix.com/pipermail/twisted-python/2009-July/019902.htm l >> >>...if this post hijacks a thread as well, then it will be 3 times. >> >>...this seems to be something specific to the twisted python mailing >>list: >>http://www.mail-archive.com/mailman-users%40python.org/msg53996.html >> > >I don't really know much about how mailman generates those threaded >views. >twisted-python is running on an unmodified mailman 1:2.1.9-9ubuntu1 >install. >Perhaps this could be caused by exim4 configuration somehow (we do have >various exim4 configuration customizations), but that's the only thing I >can think of. Any pointers on what information is used to determine the >threading structure of messages? > >Jean-Paul > Well, according to Mark Sapiro on the mailman list, in general, "It is based on In-Reply-To: and/or References: headers" ...but I assume you're looking for something a little more specific here. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] (Mis)indented Posts (list administrativia)
On Fri, 3 Jul 2009 16:11:38 -0400, Gerrat Rickert wrote: > >Well, according to Mark Sapiro on the mailman list, in general, >"It is based on In-Reply-To: and/or References: headers" > >...but I assume you're looking for something a little more specific >here. > A bit. Looking at the messages in question, there are no In-Reply-To or References headers which would suggest the threading structure mailman is deciding on. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
