Re: [Twisted-Python] reactor.stop() and thread and wxPython problem

[Reza Lotun]

Hi Gabriel,

I don't quite understand your question. Do you mean to say that you
have written some code with wx and Twisted in an integrated event
loop, and are doing something like reactor.callInThread(someCall), and
when you call reactor.stop the app doesn't exit? AFAIK, any thread
spawning goes through Twisted's internal thread pool and thread
joining should be handled by reactor.stop - I suppose it depends on
what those threads are doing.

Based on my experience with wx and Twisted, I found the following
works quite well:

* Have the reactor run in a seperate thread from wx (making sure to
use reactor.run(installSignalHandlers=False))
* Spawn threads from wx->Twisted in the proper way: use
reactor.callFromThread(reactor.callInThread(someCall))

Hope this helped,
Reza

On Fri, Jan 16, 2009 at 7:37 AM, Gabriel Rossetti
 wrote:
> Hello everyone,
>
> I have written some code using Twisted that spawns threads and another using
> wxPython integration. For some reason when reactor.stop() is called they
> don't always exit. I suspect that the reactor is waiting on the threads to
> finish, which they apparently don't do, and as for the wx integration, I
> don't know, maybe wx's main loop is in a separate thread and it would be the
> same problem. Does anyone know/have an idea or has anyone else experienced
> this?
>
> Thank you,
> Gabriel
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>



-- 
Reza Lotun
Senior Software Engineer
Peer Technologies Limited
r...@getpeer.com

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] Write into a persistent connection before stopping

[Boeuf, Jean-Francois]

 
Hello,

I have an application using twisted in which the web browser opens a
persistent connection when a user logs in (COMET model). I want to write
data into active persistent connections at server stop to notify client to
logout properly.

To do that, I add a system event trigger to the reactor calling the method
that closes session before shutdown when I start the webService. This method
is called when twisted receives a SIGTERM and I can see in twistd.log traces
of writing to the persistent connexion before services stops and connexions
are closed. But the web browser never receives this content and listening
the traffic between server and client with wireshark, I can't see anything
else than the [FIN, ACK] packets that are sent when server ends all
connections (delayed for maximum clarity in the logs).

The same method is called to logout client properly when session expires and
does work fully.

You can see below the twistd.log file when the servers stops.

Thank you for your help.

Regards

Jean-François


twistd.log
--
2009-01-16 13:26:16+0100 [-] Received SIGTERM, shutting down.
2009-01-16 13:26:16+0100 [-] 10.37.68.212 - - [16/Jan/2009:12:26:16 +]
"GET /sim/listen?sourceScreen=216 HTTP/1.1" 200 166
"http://peyo:8080/sim?id=216"; "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr;
rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16"
2009-01-16 13:26:16+0100 [-] 10.37.68.212 - - [16/Jan/2009:12:26:16 +]
"GET /sim/listen?sourceScreen=217 HTTP/1.1" 200 166
"http://peyo:8080/sim?id=217"; "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr;
rv:1.8.1.16) Gecko/20080702 Firefox/2.0.0.16"
2009-01-16 13:26:16+0100 [-] 10.37.68.212 - - [16/Jan/2009:12:26:16 +]
"GET /sim/listen?sourceScreen=218 HTTP/1.1" 200 166
"http://peyo:8080/sim?id=218&screen0=216&screen1=217&nbScreen=2";
"Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1.16) Gecko/20080702
Firefox/2.0.0.16"
2009-01-16 13:26:25+0100 [-] (Port 8080 Closed)
2009-01-16 13:26:25+0100 [-] Stopping factory

2009-01-16 13:26:25+0100 [-] (Port 8082 Closed)
2009-01-16 13:26:25+0100 [-] Stopping factory

2009-01-16 13:26:25+0100 [-] Main loop terminated.
2009-01-16 13:26:25+0100 [-] Server Shut Down

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] reactor.stop() and thread and wxPython problem

[Gabriel Rossetti]

Reza Lotun wrote:

Hi Gabriel,

I don't quite understand your question. Do you mean to say that you
have written some code with wx and Twisted in an integrated event
loop, and are doing something like reactor.callInThread(someCall), and
when you call reactor.stop the app doesn't exit? AFAIK, any thread
spawning goes through Twisted's internal thread pool and thread
joining should be handled by reactor.stop - I suppose it depends on
what those threads are doing.

  


Almost :-) I have 2 programs, one is Twisted + wx using the integrated 
event loop, as described in the Twisted documentation. This program has 
no threads, but I use reactor.callFromThread sometimes from the wx part.


The other program is Twisted with no wx but it uses threads 
(threads.deferToThread).


They both have trouble sometimes closing when reactor.stop() is called, 
not always though.




Based on my experience with wx and Twisted, I found the following
works quite well:

* Have the reactor run in a seperate thread from wx (making sure to
use reactor.run(installSignalHandlers=False))
* Spawn threads from wx->Twisted in the proper way: use
reactor.callFromThread(reactor.callInThread(someCall))

  


Hmmm, I'll give that a try later on, does it work better than the 
official way of doing it? Has anyone else tried this?



Hope this helped,
Reza

  

Thanks Reza

On Fri, Jan 16, 2009 at 7:37 AM, Gabriel Rossetti
 wrote:
  

Hello everyone,

I have written some code using Twisted that spawns threads and another using
wxPython integration. For some reason when reactor.stop() is called they
don't always exit. I suspect that the reactor is waiting on the threads to
finish, which they apparently don't do, and as for the wx integration, I
don't know, maybe wx's main loop is in a separate thread and it would be the
same problem. Does anyone know/have an idea or has anyone else experienced
this?

Thank you,
Gabriel

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python






  


___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] reactor.stop() and thread and wxPython problem

[Stephen Waterbury]

Reza Lotun wrote:

* Have the reactor run in a seperate thread from wx (making sure to
use reactor.run(installSignalHandlers=False))
* Spawn threads from wx->Twisted in the proper way: use
reactor.callFromThread(reactor.callInThread(someCall))


I'd second that -- I've used the pattern described here:
http://netofpeers.com/wiki2/index.php/WxPython_And_Twisted_Example

... to implement a wxpython client for David Ripton's ampchat example
(his uses a pygtk client):
http://ripton.net/hg/ampchat

... you can access my clone of ampchat (with wxpython client) here:
https://pangalactic.us/repo/ampchat/

Steve

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] reactor.stop() and thread and wxPython problem

[Reza Lotun]

>> * Have the reactor run in a seperate thread from wx (making sure to
>> use reactor.run(installSignalHandlers=False))
>> * Spawn threads from wx->Twisted in the proper way: use
>> reactor.callFromThread(reactor.callInThread(someCall))

A small point, but my code snippet example should have been:

reactor.callFromThread(reactor.callInThread, someCall)

Cheers,
Reza

-- 
Reza Lotun
Senior Software Engineer
Peer Technologies Limited
r...@getpeer.com

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] django with twisted 8.2.x: 2009

[Alex Clemesha]

Vlad,

I appreciate you detailed response, thanks.
>
> If yo have a really long running tasks, I suggest you donot
> 'bind' gui into those -- and instead do what I am doing -- just
> submit requests to the database, have background servers process them
> and then change the status of the request to 'Done' so that the GUI
> can either poll it or the user will see the task is completed.
> Because during those tasks you can have user logoff, DB shutdown, etc.
Absolutely - I agree that the pattern of having background server
process handle the
work of long running tasks is the way to go.

Good luck on implementing your app.
-Alex


-- 
Alex Clemesha
clemesha.org

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Tracebacks being dropped from exceptions when using inlineCallbacks

[Phil Christensen]

On Jan 12, 2009, at 8:36 PM, Terry Jones wrote:
I think I've finally gotten to the bottom of why exceptions  
sometimes lose

their tracebacks when using inlineCallbacks.

[snip]

We make a failure object, it has a
traceback, but after passing it to the errback method on a deferred  
the

traceback is gone.

This happens because _runCallbacks in defer.py finds no call/errback
functions to call on the deferred, drops into this code:

   if isinstance(self.result, failure.Failure):
   self.result.cleanFailure()

which sets the __dict__ on the failure object to the result of the
failure's __getstate__ method, which sets the traceback to None:

   # added 2003-06-23. See comment above in __init__
   c['tb'] = None

But the comment in __init__ seems to have been deleted.


The comment was this:

# added 2003-06-23 by Chris Armstrong. Yes, I actually have a
# use case where I need this traceback object, and I've made
# sure that it'll be cleaned up.
self.tb = tb

presumably referring to the problem with keeping references to  
tracebacks that was a potential pitfall in the Python version of that  
time. That's a non-issue since Python 2.2, though, which is probably  
why the comment got deleted.


Originally all __getstate__ did was stringify the object's state,  
which is probably why cleanFailure calls it directly. It looks to me  
like most of the code from __getstate__ should really be moved into a  
separate method, which would be called from cleanFailure as well as  
__getstate__.


This assumes that __getstate__ would still set c['tb'] to None, but  
that cleanFailure would not. I'm not sure what the repercussions of  
this would be, though. It seems the old problem with keeping  
references to tracebacks is less of an issue now, based on this from  
the current Python docs:


Note Beginning with Python 2.2, such cycles are automatically  
reclaimed when garbage collection is enabled and they become  
unreachable, but it remains more efficient to avoid creating cycles.


I'd appreciate any feedback.

-phil

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Tracebacks being dropped from exceptions when using inlineCallbacks

[Terry Jones]

Hi Phil

> "Phil" == Phil Christensen  writes:
Phil> On Jan 12, 2009, at 8:36 PM, Terry Jones wrote:
>> # added 2003-06-23. See comment above in __init__
>> c['tb'] = None
>> 
>> But the comment in __init__ seems to have been deleted.

Phil> The comment was this:

Phil> # added 2003-06-23 by Chris Armstrong. Yes, I actually have a
Phil> # use case where I need this traceback object, and I've made
Phil> # sure that it'll be cleaned up.
Phil> self.tb = tb

I'd seen that comment, and even the same date on it, but assumed there was
more detail in __init__.  I don't know what Chris' use case was, but we
definitely have another one now.

I don't really understand the comment in any case. If someone creates a
Failure instance by explicitly passing in all of exc_value, exc_type, and
exc_tb - with the probable intention that this failure instance is passed
along an errback chain - then I think it's reasonable for them to expect
the Failure will keep the tb and not toss it.

I may well be misunderstanding things, but I don't see how there can be any
question that the self.tb = tb line is desirable.

Phil> Originally all __getstate__ did was stringify the object's state,
Phil> which is probably why cleanFailure calls it directly. It looks to me
Phil> like most of the code from __getstate__ should really be moved into a
Phil> separate method, which would be called from cleanFailure as well as
Phil> __getstate__.

That sounds fine to me.

If I understand you and the code correctly, the only reason for calling
cleanFailure is to reduce the number of references to other objects, thus
clearing the way for garbage collection that would not otherwise happen. Is
that right?

Phil> I'd appreciate any feedback.

And thanks a lot for yours. It would be very useful to get this fixed. I'm
happy to submit a patch, though your suggested change looks like a couple
of minutes' work (just swapping the names of the cleanFailure and
__getstate__ methods gets you almost all the way there).

Terry

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] getProcessOutput on windows

[Shawn Church]

I am developing an application on Linux for deployment on Windows.  I want
to use the getProcessOutput function but seem to remember problems with the
windows reactor and processes last time I tried this (a couple years ago).
I've gone through the active bugs and did not see anything critical to my
usage except maybe #2789,  which is a year old and apparently not going to
be addressed due to Windows limitations.

Is getProcessOutput usable on Windows?  If not any suggestions for an
alternative?  Note that for this application I do not care about output
while the process is running,  I just want to reap the output (or error)
when the process is done.

Thanks for your help,


Shawn
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Tracebacks being dropped from exceptions when using inlineCallbacks

[Christopher Armstrong]

On Fri, Jan 16, 2009 at 7:28 PM, Phil Christensen  wrote:
> The comment was this:
>
># added 2003-06-23 by Chris Armstrong. Yes, I actually have a
># use case where I need this traceback object, and I've made
># sure that it'll be cleaned up.
>self.tb = tb
>
> presumably referring to the problem with keeping references to tracebacks
> that was a potential pitfall in the Python version of that time. That's a
> non-issue since Python 2.2, though, which is probably why the comment got
> deleted.

Keeping references to tracebacks still has many potential pitfalls.
It's a fundamental problem: tracebacks refer to all their frames,
which refer to all their locals; this makes it *really* easy to create
uncollectable cycles if you have any __del__ methods *anywhere*. It's
just not safe to keep the traceback reference in the general case.
Failure needs to work with arbitrary code.

> Originally all __getstate__ did was stringify the object's state, which is
> probably why cleanFailure calls it directly. It looks to me like most of the
> code from __getstate__ should really be moved into a separate method, which
> would be called from cleanFailure as well as __getstate__.

Sure, that would be a slightly better factoring.

> This assumes that __getstate__ would still set c['tb'] to None, but that
> cleanFailure would not. I'm not sure what the repercussions of this would
> be, though. It seems the old problem with keeping references to tracebacks
> is less of an issue now, based on this from the current Python docs:
>
>> Note Beginning with Python 2.2, such cycles are automatically reclaimed
>> when garbage collection is enabled and they become unreachable, but it
>> remains more efficient to avoid creating cycles.
>
> I'd appreciate any feedback.
>
> -phil

-- 
Christopher Armstrong
http://radix.twistedmatrix.com/
http://planet-if.com/
http://canonical.com/

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Tracebacks being dropped from exceptions when using inlineCallbacks

[glyph]

On 02:31 am, ra...@twistedmatrix.com wrote:
On Fri, Jan 16, 2009 at 7:28 PM, Phil Christensen 
 wrote:



Keeping references to tracebacks still has many potential pitfalls.
It's a fundamental problem: tracebacks refer to all their frames,
which refer to all their locals; this makes it *really* easy to create
uncollectable cycles if you have any __del__ methods *anywhere*. It's
just not safe to keep the traceback reference in the general case.
Failure needs to work with arbitrary code.


This used to be pretty much 100% of the time, since Deferred itself used 
to have a __del__, so if you had a Deferred anywhere on the stack you 
were dead.  Nowadays it's a little better than that, but a surprising 
number of objects from elsewhere are uncollectable.
Originally all __getstate__ did was stringify the object's state, 
which is
probably why cleanFailure calls it directly. It looks to me like most 
of the
code from __getstate__ should really be moved into a separate method, 
which

would be called from cleanFailure as well as __getstate__.


Nevertheless cleanFailure will still need to remove the traceback, for 
the reasons that Chris mentioned.

Sure, that would be a slightly better factoring.


Would that help, though?

I don't fully understand the issue, so this is probably wrong in some 
obvious way, but: do we really need to hang on to a real traceback 
object to resolve the issue in question?  It sounds like the real issue 
here is that we don't have a channel to communicate a "cleaned 
traceback" (i.e., failure with stringified frames, but no traceback) 
from one bit of code to another?
Note Beginning with Python 2.2, such cycles are automatically 
reclaimed
when garbage collection is enabled and they become unreachable, but 
it

remains more efficient to avoid creating cycles.


I'm not sure what Python documentation you're quoting here, but that's 
not accurate in the general case.  For example, see 
:


   "Objects that have __del__() methods and are part of a reference 
cycle cause the entire reference cycle to be uncollectable".

I'd appreciate any feedback.


Two bits of feedback:

1. More of this discussion should be on a ticket.  The mailing list is 
good for getting discussions started and figuring out what needs to be 
done, but now we're talking about the technical specifics of resolving a 
specific problem.  Someone (terry?) should take the specifics in the 
email that started this thread and file an appropriate ticket.


2. When filing that ticket, I'd really like a full working example of 
inlineCallbacks not showing a traceback to play with, not just examples 
of portions of the problem.  The obvious example I tried,


   from twisted.internet.defer import inlineCallbacks

   def buggy2():
   raise RuntimeError("Whoops!")

   @inlineCallbacks
   def buggy():
   yield buggy2()

   buggy()

actually prints a nice descriptive traceback.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python