On Fri, 27 Feb 2009 15:26:43 +0100, Marcel Keller <mkel...@cs.au.dk> wrote:
Hi,
I am working on the VIFF project (viff.dk) which uses Twisted. I found out
that our code is sometimes inefficient because we are generating many
deferreds (maybe about 10000) in a callback. While doing that, no network
communication is performed. Therefore, I investigated the possibility of
adding a function to the reactor which is called every iteration and from
which the iteration could be called safely. Then, we could generate all
deferreds in that function and activate the reactor from to time. See the
attached patch for details.
So you're doing a ton of work all at once now and you want to split up that
ton of work into smaller pieces and do it a little at a time?
If that's the case, then you don't need to modify the reactor, you just
need to split up the work your code is going. There are a lot of techniques
for doing this. coiterate and inlineCallbacks are two solutions which are
closest to "cookie cutter" (ie, you have the least flexibility in deciding
how to use them).
Some of our code runs twice as fast when using that hack. Are there any
chances something similar might be included in Twisted? Or does anyone have
a better solution for the described problem?
Give coiterate or inlineCallbacks a try. If you need help applying either
of these to your problem, please ask. :) I can't make any specific
suggestions now because I can only guess at how you're using the reactor
modification you attached.
You have a very long, steep, uphill battle to convince me that adding support
for re-entrant iteration is a good idea.
Furthermore, I have a question, which is probably related. The documentation
of IReactorCore says about the iterate() function:
"The reactor must have been started (via the run() method) prior to any
invocations of this method. It must also be stopped manually after the last
call to this method (via the stop() method). This method is not re-entrant:
you must not call it recursively; in particular, you must not call it while
the reactor is running."
This looks to me as if the reactor needs to be running and not running at
the same time so that iterate() can be called. Is there an error in my
reasoning?
It's very subtle and intentionally written to sound impossible to use (maybe
that is a stupid idea, but that was my intent when I wrote it ;). What
you're missing is that there is a way to make reactor.run() return without
stopping the reactor.
Jean-Paul
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
