On 02:49 am, jjo...@nextdigital.com wrote:
>I'm trying to make a simple deferred that will continue get a message
>of
>a message queue, process it and then wait for another message.
>
>While I can get the first message easily I am unable to work out how I
>can get a second message without creating a new deferred in the
>printMsg
>function, which eventually gives me a maximum recursion depth exceeded.
>
>Basic code is a follows:
>
>from twisted.internet import reactor,defer
>
>def printMsg(msg):
> print "Message is:"
> print msg
>
> deferred=getMsg()
> deferred.addCallback(printMsg)
>
>def getMsg() :
> d=defer.Deferred()
> #replaced with code that actually goes to a queue to get the message
> msg="This is a message"
> d.callback(msg)
> return d
>
>deferred=getMsg()
>deferred.addCallback(printMsg)
>reactor.run()
>
>Can any one point me on the right path to solve this?
>
>Thanks
>
>John
The above code is basically another way of writing this:
def getMsg():
return "This is a message"
def printMsg(msg):
print "Message is:"
print msg
printMsg(getMsg())
printMsg(getMsg())
Hopefully it's obvious why this version of the code would fail
eventually with a recursion error. These are equivalent because
d.addCallback(f) will call f immediately (ie, before addCallback
returns) if d.callback(result) has already been called.
This only changes if getMsg (the original Deferred-returning one) ever
returns a Deferred which has *not* already fired. In such a case, the
call stack will unwind back to the main loop and the code won't run any
further until the Deferred returned by getMsg is called back.
I would expect the actual queue-retrieval version of the code to
sometimes (or always) return a Deferred which has not yet been called
back. However, if you're seeing recursion errors, then this must not be
the case. Are you expecting the queue-retrieval version to return
Deferreds that have already fired?
If you aren't, then I don't think I understand why you're encountering
this error (or you're mistaken in your expectations). An example closer
to your actual program might help clear things up.
If you are, then you have two options.
First, you can stop using Deferreds and change the code from recursive
to iterative. eg,
def getMsg():
return "This is a message"
def printMsg(msg):
print "Message is:"
print msg
while True:
printMsg(getMsg())
Of course, that's not very event loop friendly, so you might try making
it cooperative by hooking it up to a scheduler:
def messageIteration():
printMsg(getMsg())
reactor.callLater(0, messageIteration)
Or, factoring the specific scheduler policy out:
def messagePrinter():
while True:
yield printMsg(getMsg())
from twisted.internet.task import coiterate
coiterate(messagePrinter())
Second, if there's a good reason to use Deferreds in this code (for
example, there are multiple implementations of getMsg, some of which are
synchronous, some of which are asynchronous, and preserving the
interface between all of them is desirable), then you may want to use
one of the above scheduler techniques as a trampoline so that you can
avoid having the stack build up when a synchronous implementation is
used:
def messagePrinter():
d = getMsg()
d.addCallback(printMsg)
d.addCallback(lambda ignored: reactor.callLater(0,
messagePrinter))
Or you might use inlineCallbacks to define the trampoline:
@inlineCallbacks
def messagePrinter():
while True:
pringMsg(yield getMsg())
The idea here being that you never let recursion continue unbounded.
You process one iteration and then let its call frames finish, return,
and pop off the call stack so that they're out of the way for the next
iteration.
Incidentally, there are a couple of other ways that Deferreds can trick
you into hitting the call stack limit. They can be addressed in ways
similar to the ones I've described here. However, they're also due to
implementation limitations which hopefully will be removed someday. The
problem that you've run into here, though, doesn't seem to be due to an
implementation limitation (at least not one that I can imagine any
resolution to now) - it's just a pattern you need to avoid.
Hope this helps,
Jean-Paul
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
