Re: [Twisted-Python] How to detect when there is nothing waiting in the event queue
On 04:11 pm, ma...@lumierevfx.com wrote: >Hi, we've been using Twisted in an internal project for the past 4 >months. It's a system that polls for events from a web source (looping >call) and also accepts connections via perspective broker. Each of >these >events is turned into a series of Deferreds that run various tasks. >Some >of these tasks are long-running and can take several hours to complete. > >Because of this, it's very difficult to restart the server when we roll >out a new release. > >I've added a signal handler to catch SIGHUP and call stop() on the >looping call and stopListening() on the perspective broker Root. How >can >I detect when all remaining Deferreds have fired so I can stop the >reactor safely, without stranding any running processes? There's no way to know when "all remaining Deferreds have fired". Since Deferreds are general purpose and any part of Twisted may internally use them for anything they're suited for, this isn't really what you want anyway. You want to know when all of *your* tasks have completed. So, you should look at all the APIs you have which return Deferreds, decide which APIs need to be allowed to complete (perhaps it will be all of them) before shutdown can happen, and then instrument those APIs to keep track of their own status. This may be as simple as internally adding a callback to the Deferreds being returned (before returning them), letting you know when each task has completed. This should let you decide when things are quiet enough to shut down. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] simple global counter
Tim Allen пишет: > On Mon, Sep 14, 2009 at 10:09:38PM +0700, Artem Bokhan wrote: > >> May somebody give a sample code of simple global counter which could be >> used with twisted (non-web) enviroment? >> > > I'm not sure what you mean. I want find out if it's possible to count things in context of the whole twisted server, not just one tcp session. > If you want to keep a count of something, you can just > store it in a variable like anything else. > > event_counter = 0 > > def handle_event(event): > global event_counter > > event.do_something() > event_counter += 1 > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] simple global counter
On Tue, Sep 15, 2009 at 04:54:09PM +0700, Artem Bokhan wrote: > Tim Allen пишет: > > On Mon, Sep 14, 2009 at 10:09:38PM +0700, Artem Bokhan wrote: > > > >> May somebody give a sample code of simple global counter which could be > >> used with twisted (non-web) enviroment? > > > > I'm not sure what you mean. > I want find out if it's possible to count things in context of the > whole twisted server, not just one tcp session. Well, of course - you can create global variables in Python, and nothing in Twisted affects that. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] run queries in deffered, but not in transaction
Hi, I'd like to run several queries in background, some of them may fail. Currently, I do runInteraction for each query. Is there a way to run them all in one deferred, so that if one of them fails, other are not aborted? Thanks in advance! Pet ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] network server on multiprocessor hardware
Is there any know implementation of the method author of this thread http://twistedmatrix.com/pipermail/twisted-python/2009-April/019478.html talks about? Bokhan Artem пишет: > Hello. > > Is there right (standard) way to write twisted network server > (tcp/web/mail/etc) which could use several cpu cores? For example, > master+workers model, where every worker doing the same job. > If so, may somebody help to wrap the sample code below? > May be some well-known twisted based software can do that? > > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] memory leak on 8.2.0/2.6 versus 2.5/2.4
On 14 Sep, 08:29 pm, matu...@yahoo.com wrote: >I am running identical twisted servers under high load, one on 2.5.0 >(patched for epoll bug) on python 2.4.4 and second one on 8.2.0 on >python >2.6.2. >The 8.2/2.6 one leaks memory compared to 2.5/2.4.4 to the point that I >am >about to roll back to older Twisted and then to older Python. > >Any insight on this? Hi Alec, Nothing springs to mind, unfortunately. What parts of Twisted does the app use? Maybe narrowing the field a bit will help jog someone's memory... Otherwise, good luck in tracking it down, and I hope you'll let us know if you come to any conclusions. Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] run queries in deffered, but not in transaction
On 10:37 am, petshm...@googlemail.com wrote: >Hi, > >I'd like to run several queries in background, some of them may fail. >Currently, I do runInteraction for each query. Is there a way to run >them all in one deferred, so that if one of them fails, other are not >aborted? If you have a function along the lines of this one: def someInteractions(db): interactions = [ db.runInteraction(one), db.runInteraction(two), db.runInteraction(three)] Then a failure in one shouldn't affect two or three; likewise for any other failure or combination of failures. They are naturally (ugh, not a good word, but I can't think of a better one) independent. You have to go out of your way to associate them somehow. Since it sounds like this isn't the behavior you're seeing, I think I'm probably misunderstanding some aspect of your question. If this doesn't help, perhaps you can clarify? Jean-Paul ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] run queries in deffered, but not in transaction
exar...@twistedmatrix.com wrote: > On 10:37 am, petshm...@googlemail.com wrote: >> >> I'd like to run several queries in background, some of them may fail. >> > > If you have a function along the lines of this one: > > def someInteractions(db): > interactions = [ > db.runInteraction(one), > db.runInteraction(two), > db.runInteraction(three)] > > Then a failure in one shouldn't affect two or three; likewise for any > other failure or combination of failures. They are naturally (ugh, not > a good word, but I can't think of a better one) independent. You have > to go out of your way to associate them somehow. > I think he might mean he wants them to run sequentially, even if one fails. You can do that explicitly via @inlineCallbacks like this: @inlineCallbacks def someInteractions(db): try: yield db.runInteraction(one) except: pass try: yield db.runInteraction(two) except: pass try: yield db.runInteraction(three) except: pass Or with callback/errbacks, like this: def someInteractions(db) d = db.runInteraction(one).addBoth(db.runInteraction, two).addBoth(db.runInteraction, three) addBoth is a convenience method that adds the same function as a callback and an errback: http://twistedmatrix.com/documents/8.2.0/api/twisted.internet.defer.Deferred.html#addBoth -- Mark Visser, Software Director Lumière VFX Email: ma...@lumierevfx.com Phone: +1-514-316-1080 x3030 ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] run queries in deffered, but not in transaction
On Tue, Sep 15, 2009 at 5:19 PM, Mark Visser wrote: > exar...@twistedmatrix.com wrote: >> On 10:37 am, petshm...@googlemail.com wrote: >>> >>> I'd like to run several queries in background, some of them may fail. >>> >> >> If you have a function along the lines of this one: >> >> def someInteractions(db): >> interactions = [ >> db.runInteraction(one), >> db.runInteraction(two), >> db.runInteraction(three)] >> >> Then a failure in one shouldn't affect two or three; likewise for any >> other failure or combination of failures. They are naturally (ugh, not >> a good word, but I can't think of a better one) independent. You have >> to go out of your way to associate them somehow. >> > I think he might mean he wants them to run sequentially, even if one fails. > > You can do that explicitly via @inlineCallbacks like this: > > @inlineCallbacks > def someInteractions(db): > try: > yield db.runInteraction(one) > except: > pass > > try: > yield db.runInteraction(two) > except: > pass > > try: > yield db.runInteraction(three) > except: > pass > > Or with callback/errbacks, like this: > > def someInteractions(db) > d = db.runInteraction(one).addBoth(db.runInteraction, > two).addBoth(db.runInteraction, three) Hi Mark! Yes, I'd like run them sequentially, it was not clear for me, how to do it in one deferred. I will try your suggestions out. Thanks for help! Pet > > addBoth is a convenience method that adds the same function as a > callback and an errback: > > http://twistedmatrix.com/documents/8.2.0/api/twisted.internet.defer.Deferred.html#addBoth > > -- > Mark Visser, Software Director > Lumière VFX > Email: ma...@lumierevfx.com > Phone: +1-514-316-1080 x3030 > > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] hi !!!
I wrote my problem in the last message an I wasn't clear with that... Ok... I'm going again... !!! I have a web server... this server was implemented with twisted using XMLRPC... when the server start to run, in this moment I did "reactor.run()"... well, my server is listening by 8080 port, and when one command had become, I catch the command, interpret it, and after I need to run another process tu execute the command. To execute another process I have to call "reactor.spawnProcess(...)" with its parameters, and after I have to say again "reactor.run()", but the reactor is running because the server isn't stop. I need to run another process with twisted without reactor how can I do that ?? or, I can do the same thing with another variant ??? please help me !!! --- [Tank] "Amar desenfrenadamente es abrazar la utopía de que tenemos lo que queremos." --- Freddy Martinez Garcia Lic. en Ciencia de la Computación. Edad: 27 Años Santiago de Cuba e-mails: * fre...@cbm.uo.edu.cu * freddy311...@gmail.com * freddymg311...@hotmail.com -- Este mensaje ha sido analizado por MailScanner en busca de virus y otros contenidos peligrosos, y se considera que está limpio. For all your IT requirements visit: http://www.transtec.co.uk ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] hi !!!
On Tue, Sep 15, 2009 at 6:47 PM, Freddy Martinez Garcia < fre...@cbm.uo.edu.cu> wrote: > I wrote my problem in the last message an I wasn't clear with that... > > > Ok... I'm going again... !!! > > > I have a web server... this server was implemented with twisted using > XMLRPC... when the server start to run, in this moment I did > "reactor.run()"... > > > well, my server is listening by 8080 port, and when one command had become, > I catch the command, interpret it, and after I need to run another process > tu execute the command. To execute another process I have to call > "reactor.spawnProcess(...)" with its parameters, and after I have to say > again "reactor.run() > No, as was explained you do not have to call reactor.run() again. Twisted use a global event loop (so to speak), you have one and you start it once, and when it is running all defereds will execute. ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
[Twisted-Python] Perspective Broker for plugins
Hello, I'm just now undergoing my first real twisted application. There is a main server which users authenticate with, and the main server also has different plugins for chat/whatever features. The client side also has a handler for each of these plugins and then a GTK gui associate with them. I've been reading the documentation, tutorials, etc, and I'm a little confused about what the best approach would be. Would you all suggest using Perspective Broker? Each plugin has it's own set of communication that needs to happen with the plugin handler server side as well. Would this be easier some other way? Another question has to do with a GTK handler. I saw that twisted has a reactor built in for that. I do have events that need to be triggered from the server. Should I use twisted for that as well, or just send requests to a callback within the GTK application? Really there is just so much information out there about twisted and everyone does it differently that it's confusing to get started. Any help would be appreciated. Thanks. Tim _ Hotmail: Free, trusted and rich email service. http://clk.atdmt.com/GBL/go/171222984/direct/01/___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Re: [Twisted-Python] memory leak on 8.2.0/2.6 versus 2.5/2.4
> Nothing springs to mind, unfortunately. What parts of Twisted does the > app use? Maybe narrowing the field a bit will help jog someone's > memory... Nothing special: Epoll reactor and twisted.enterprise.adbapi Will keep you guys posted. > -Original Message- > From: twisted-python-boun...@twistedmatrix.com [mailto:twisted-python- > boun...@twistedmatrix.com] On Behalf Of exar...@twistedmatrix.com > Sent: Tuesday, September 15, 2009 7:52 AM > To: Twisted general discussion > Subject: Re: [Twisted-Python] memory leak on 8.2.0/2.6 versus 2.5/2.4 > > On 14 Sep, 08:29 pm, matu...@yahoo.com wrote: > >I am running identical twisted servers under high load, one on 2.5.0 > >(patched for epoll bug) on python 2.4.4 and second one on 8.2.0 on > >python > >2.6.2. > >The 8.2/2.6 one leaks memory compared to 2.5/2.4.4 to the point that I > >am > >about to roll back to older Twisted and then to older Python. > > > >Any insight on this? > > Hi Alec, > > Nothing springs to mind, unfortunately. What parts of Twisted does the > app use? Maybe narrowing the field a bit will help jog someone's > memory... Otherwise, good luck in tracking it down, and I hope you'll > let us know if you come to any conclusions. > > Jean-Paul > > ___ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python ___ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
