On 05:48 pm, mcfle...@vrplumber.com wrote:
exar...@twistedmatrix.com wrote:
On 05:55 am, jacopo.pe...@gmail.com wrote:
...
results to be ready, I collect and process them. From now on I don
19t
need a the system to be event drive any more, the processing should
occur only on the master machine, following a deterministic flow.
As soon as finished I am ready to start again to resubmit the models
for recalculation and so on. This should go on forever.
Jean-Paul is obviously far more authoritative on the "twisted way" than
I am, so if he says you can just run your synchronous operation in-
situ,
that's probably the way to go, but IIRC there's a
reactor.deferToThread() function which can run your synchronous code
"off to the side", while allowing the twisted code to continue to
process incoming operations. Thus you'd do something like:
def process( dataset ):
dl = [ remote_call( x ) for x in dataset]
dl = defer.DeferredList( dl )
def on_all_results( results ):
reactor.deferToThread( sync_process, (results,)).addCallback(
process )
return dl.addCallback( on_all_results )
(I'm typing all of that from the distance of a few years of memory
decay, so take it as "loosely this, with the proper function names and
the like"). Threads aren't really part of the "twisted way" in my
understanding, but they can be used if necessary AFAIK, and they will
let your application remain responsive to network events during the
processing.
Yep, you're correct here Mike (except it's
`twisted.internet.defer.deferToThread` rather than
`twisted.internet.reactor.deferToThread`). If it is safe to call
`sync_process` in a thread, then this may be a good approach as well,
and it will free up the reactor to continue to respond to events
(assuming `sync_process` plays nicely - ie, is written in Python or is
an extension that releases the GIL, of course).
In my post, I was trying to highlight the idea that there's not really
anything special going on in a Twisted program. You can choose to block
if you wish, if the consequences (events go unserviced for a while) are
acceptable to you.
Jean-Paul
--
http://mail.python.org/mailman/listinfo/python-list
