Re: [Twisted-Python] Difficulty restarting/shutting down a twisted server - ports not closing

[Glyph]

On Jun 11, 2013, at 6:20 PM, Paul Sajna  wrote:

> I continue to use threads because my application uses a GUI that becomes 
> blocked when the server starts. Also, I am now having a problem where port is 
> undefined until the server is stopped and started again.

Paul,

Until you stop using threads in this manner, you are using threads in a totally 
unsupported way and Twisted will be broken.

Twisted supports most GUIs directly though; there is generally no need to use 
threads.  What GUI library are you using?

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] reload / restart best practices ?

[Jonathan Vanasco]

I have a twisted application that seems to occasionally jam up.  It dispatches 
a lot of work via DeferToThread, and in some instances it appears that the 
specific thread pool locks up ( haven't found out why yet ).  I'm fairly 
certain that the problem is stemming from getting HTTP documents ( there's a 
web indexing component and 3 external api tools ).

An acceptable fix for now would be periodic restarts ( daily or even weekly 
would more than suffice at this point ). 

Does anyone have a suggestion on how to best approach this ?  I saw some old 
discussions about ideas for implementing graceful restarts via autoload, and 
other stuff.   Because my application might be in mid-conversation with 
external APIs , I don't want to kill the process and respawn.  I'd like them to 
finish , and shutdown/restart N seconds later ( assuming that I wouldn't be 
able to fully shutdown because of an issue with a locked thread ).  
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

[Twisted-Python] reload / restart best practices ?

[Jonathan Vanasco]

I have a twisted application that seems to occasionally jam up.  It dispatches 
a lot of work via DeferToThread, and in some instances it appears that the 
specific thread pool locks up ( haven't found out why yet ).  I'm fairly 
certain that the problem is stemming from getting HTTP documents ( there's a 
web indexing component and 3 external api tools ).

An acceptable fix for now would be periodic restarts ( daily or even weekly 
would more than suffice at this point ). 

Does anyone have a suggestion on how to best approach this ?  I saw some old 
discussions about ideas for implementing graceful restarts via autoload, and 
other stuff.   Because my application might be in mid-conversation with 
external APIs , I don't want to kill the process and respawn.  I'd like them to 
finish , and shutdown/restart N seconds later ( assuming that I wouldn't be 
able to fully shutdown because of an issue with a locked thread ).  
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Difficulty restarting/shutting down a twisted server - ports not closing

[Paul Sajna]

I'm using Tkinter. When I do the reactor.listenTCP(), the server takes over
the main thread and the GUI freezes. Could you help me craft a GUI-safe
solution?


On 12 June 2013 00:06, Glyph  wrote:

>
> On Jun 11, 2013, at 6:20 PM, Paul Sajna  wrote:
>
> I continue to use threads because my application uses a GUI that becomes
> blocked when the server starts. Also, I am now having a problem where port
> is undefined until the server is stopped and started again.
>
>
> Paul,
>
> Until you stop using threads in this manner, you are using threads in a
> totally unsupported way and Twisted will be broken.
>
> Twisted supports most GUIs directly though; there is generally no need to
> use threads.  What GUI library are you using?
>
> -glyph
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Difficulty restarting/shutting down a twisted server - ports not closing

[Laurens Van Houtven]

Hi Paul

The documentation covers how to use Twisted with Tk (and many other event
loops). Here's the link you want:
https://twistedmatrix.com/documents/current/core/howto/choosing-reactor.html#auto16

Hope that helps :)
lvh
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] reload / restart best practices ?

[Glyph]

On Jun 12, 2013, at 11:40 AM, Jonathan Vanasco  wrote:

> Does anyone have a suggestion on how to best approach this ?  I saw some old 
> discussions about ideas for implementing graceful restarts via autoload, and 
> other stuff.

I'm not sure what "autoload" is.  I am assuming it's some kind of 
auto-re-loading technology though.

> Because my application might be in mid-conversation with external APIs , I 
> don't want to kill the process and respawn.

You do want to re-start your process.  What you're diagnosing is some weird, 
unknown thread/process interaction and unless you get a clean slate, you don't 
know what is happening.

You may not want to do so immediately, though, since, as you say:

> I'd like them to finish , and shutdown/restart N seconds later ( assuming 
> that I wouldn't be able to fully shutdown because of an issue with a locked 
> thread ).  

The issue is with letting them finish.  In order to accomplish this, you should 
write IService implementations that have stopService methods that, when called, 
will wait for any operations in flight associated with that service.

Once everything is quiesced, you can re-exec ([sys.executable] + sys.argv) and 
that ought to result in the same process starting up again.  If you've done the 
appropriate thing and made all your stopService methods wait for their 
operations to complete, then waiting for them all to complete is just putting 
them all into a MultiService, calling stopService() on that MultiService, and 
adding a callback/errback to the Deferred it returns which calls os.exec.

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] reload / restart best practices ?

[Jonathan Vanasco]

On Jun 12, 2013, at 4:31 PM, Glyph wrote:

> I'm not sure what "autoload" is.  I am assuming it's some kind of 
> auto-re-loading technology though.

The earlier discussions talked about Django's autoload, which reloads the 
entire process on a config file change.

> You do want to re-start your process.  What you're diagnosing is some weird, 
> unknown thread/process interaction and unless you get a clean slate, you 
> don't know what is happening.
> 
> You may not want to do so immediately, though, since, as you say:

Correct on both accounts.  I don't want to kill the current process while it's 
talking to an API.  I just want it to not talk to any additional APIs, then 
kill the process a bit later.

>> I'd like them to finish , and shutdown/restart N seconds later ( assuming 
>> that I wouldn't be able to fully shutdown because of an issue with a locked 
>> thread ).  
> 
> The issue is with letting them finish.  In order to accomplish this, you 
> should write IService implementations that have stopService methods that, 
> when called, will wait for any operations in flight associated with that 
> service.
> 
> Once everything is quiesced, you can re-exec ([sys.executable] + sys.argv) 
> and that ought to result in the same process starting up again.  If you've 
> done the appropriate thing and made all your stopService methods wait for 
> their operations to complete, then waiting for them all to complete is just 
> putting them all into a MultiService, calling stopService() on that 
> MultiService, and adding a callback/errback to the Deferred it returns which 
> calls os.exec.

ok great !  I'll look into StopService and IService tomorrow !

thanks a ton.  that's pretty much what i want/need.

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Difficulty restarting/shutting down a twisted server - ports not closing

[Naveen Michaud-Agraw]

Hi Paul,

Have you tried the solution listed here:

http://twistedmatrix.com/documents/13.0.0/core/howto/choosing-reactor.html#auto16

With this approach, twisted would manage tkinter and you won't block on any 
twisted API calls.

Sent from my iPhone

On Jun 12, 2013, at 3:58 PM, Paul Sajna  wrote:

> I'm using Tkinter. When I do the reactor.listenTCP(), the server takes over 
> the main thread and the GUI freezes. Could you help me craft a GUI-safe 
> solution?
> 
> 
> On 12 June 2013 00:06, Glyph  wrote:
>> 
>> On Jun 11, 2013, at 6:20 PM, Paul Sajna  wrote:
>> 
>>> I continue to use threads because my application uses a GUI that becomes 
>>> blocked when the server starts. Also, I am now having a problem where port 
>>> is undefined until the server is stopped and started again.
>> 
>> Paul,
>> 
>> Until you stop using threads in this manner, you are using threads in a 
>> totally unsupported way and Twisted will be broken.
>> 
>> Twisted supports most GUIs directly though; there is generally no need to 
>> use threads.  What GUI library are you using?
>> 
>> -glyph
>> 
>> 
>> ___
>> Twisted-Python mailing list
>> Twisted-Python@twistedmatrix.com
>> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
> 
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] reload / restart best practices ?

[Terry Jones]

Hi Jonathan

You might also find something like this useful:
http://twistedmatrix.com/pipermail/twisted-python/2009-December/021107.html

Optionally, it would be easy to add a timeout value to the class that could
cancel outstanding deferreds after a certain amount of time and then fire
all the waiters.

Terry



On Wed, Jun 12, 2013 at 9:40 PM, Jonathan Vanasco
wrote:

>
> On Jun 12, 2013, at 4:31 PM, Glyph wrote:
>
> I'm not sure what "autoload" is.  I am assuming it's some kind of
> auto-re-loading technology though.
>
>
> The earlier discussions talked about Django's autoload, which reloads the
> entire process on a config file change.
>
> You *do* want to re-start your process.  What you're diagnosing is some
> weird, unknown thread/process interaction and unless you get a clean slate,
> you don't know what is happening.
>
> You may not want to do so *immediately*, though, since, as you say:
>
>
> Correct on both accounts.  I don't want to kill the current process while
> it's talking to an API.  I just want it to not talk to any additional APIs,
> then kill the process a bit later.
>
> I'd like them to finish , and shutdown/restart N seconds later ( assuming
> that I wouldn't be able to fully shutdown because of an issue with a locked
> thread ).
>
>
> The issue is with letting them finish.  In order to accomplish this, you
> should write IService implementations that have stopService methods that,
> when called, will wait for any operations in flight associated with that
> service.
>
> Once everything is quiesced, you can re-exec ([sys.executable] + sys.argv)
> and that ought to result in the same process starting up again.  If you've
> done the appropriate thing and made all your stopService methods wait for
> their operations to complete, then waiting for them all to complete is just
> putting them all into a MultiService, calling stopService() on that
> MultiService, and adding a callback/errback to the Deferred it returns
> which calls os.exec.
>
>
> ok great !  I'll look into StopService and IService tomorrow !
>
> thanks a ton.  that's pretty much what i want/need.
>
>
> ___
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Testing AMP-based code

[exarkun]

On 5 Jun, 07:13 pm, f...@64studio.com wrote:

Hi,

following up from ticket #6502, I'm looking for recommendations/best
practices for writing unit-tests for AMP-based code.

As described in the ticket, the issue I'm currently facing is that the
AMP implementation is subtly not re-entrant safe and doesn't work with 
a

synchronous transport, for example this code raises an exception:

http://twistedmatrix.com/trac/attachment/ticket/6502/example.py

I'm starting to think that the most appropriate testing strategy would
be to mock/stub AMP.callRemote (or the protocol class altogether)
instead of trying to use a fake transport.

Thoughts?


I think this is thinking in the right direction.  Twisted generally 
tries to be responsible for testing its own code, and the serialization 
from commands to bytes (and the reverse) that AMP does is part of 
Twisted, so you should really be free from the burden of testing that 
that stuff works.


One thing it's worth noticing is that the AMP class itself contains very 
little code.  Instead, it inherits most functionality from a few base 
classes.  It's worth learning about the division of responsibility 
between these classes because they can be helpful in writing cleaner AMP 
code and - relevant to this topic - writing AMP unit tests.


One of the base classes, BinaryBoxProtocol, is almost entirely concerned 
with serialization logic.  You can probably ignore this one entirely to 
begin with (although consider the consequences of 
serialization/deserialization living in this one class, independent of 
the rest of the protocol logic: perhaps you want to exchange AMP 
commands with a web browser and would find JSON an easier format to work 
with than AMP's int16 string scheme... etc).


Next, BoxDispatcher.  This one is what actually implements `callRemote` 
and the reverse - ampBoxReceived, turning an AMP box (already parsed), 
into an incoming method call or the result of a previous outgoing method 
call.  It operates on locator (to look up how to handle incoming method 
calls) and a box sender (to send out boxes representing method calls or 
responses).  It doesn't know about the network, so your box sender can 
be a purely in-memory thing, implementing some box handling logic purely 
as Python code and no I/O.


As far as the locator goes, if you want the standard 
`@SomeCommand.responder` functionality, then you can easily get this by 
re-using the next base class of AMP, CommandLocator.  This one's pretty 
straight-forward: subclass it and those decorators will work for you.


Ignore the last one, SimpleStringLocator, it's for extremely old-style 
AMP code that no one should be writing anymore.


So this all means that your application logic can all live on a 
CommandLocator subclass.  When you really want to put this on an AMP 
server, you can hook an AMP instance up to your CommandLocator subclass 
(AMP takes a locator as an __init__ argument).  When you want to test 
your command implementations, you can hook the CommandLocator up to a 
BoxDispatcher and a box sender and throw boxes straight at it with no 
network interation.


Some pieces are probably still missing from the public API - for 
example, you do want to test that your objects all get properly 
serialized and deserialized through AMP, particularly if you're 
implementing custom Argument types.  There are some private APIs, 
_objectsToStrings and _stringsToObjects mostly, that really help with 
testing this, and we should think about how to expose this functionality 
publically.  Also, we should document this whole pile of stuff.  Maybe 
you'd be interested in writing something up after you've had a chance to 
play with these ideas?


Jean-Paul

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python

Re: [Twisted-Python] Testing AMP-based code

[Glyph]

On Jun 12, 2013, at 4:16 PM, exar...@twistedmatrix.com wrote:

[snip]

> So this all means that your application logic can all live on a 
> CommandLocator subclass.  When you really want to put this on an AMP server, 
> you can hook an AMP instance up to your CommandLocator subclass (AMP takes a 
> locator as an __init__ argument).  When you want to test your command 
> implementations, you can hook the CommandLocator up to a BoxDispatcher and a 
> box sender and throw boxes straight at it with no network interation.

Thanks a lot for this description.  Someone (maybe you if you're feeling 
generous, or someone else if they'd like to volunteer) should take most of the 
prose from this post, edit it a little bit to fit into the flow, and use it to 
expand the somewhat bare explanation of these concepts in 
.

> Some pieces are probably still missing from the public API - for example, you 
> do want to test that your objects all get properly serialized and 
> deserialized through AMP, particularly if you're implementing custom Argument 
> types.  There are some private APIs, _objectsToStrings and _stringsToObjects 
> mostly, that really help with testing this, and we should think about how to 
> expose this functionality publically.  Also, we should document this whole 
> pile of stuff.  Maybe you'd be interested in writing something up after 
> you've had a chance to play with these ideas?

It would also be nice if someone could find a link to the tickets for this 
functionality - or, file one, if none exist.

Again, thanks a lot for the great explanation, JP.

-glyph

___
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python