On Oct 6, 2009, at 10:57 PM, Glyph Lefkowitz wrote:
However, you can experiment with it pretty easily using
DeferredSemaphore: http://twistedmatrix.com/documents/8.2.0/api/twisted.internet.defer.DeferredSemaphore.html
Cool, I didn't know about that, I'll give it a look. Thanks!
If your app is doing something strange that you don't understand,
you should instrument it until you understand it.
It's not that I don't understand what's supposed to be happening, or
that it's doing something strange, it just seems that sites up to
about 2000 pages work fine, then things get dicey.
I was just looking for some guidance on the "max-deferred" that has
been found in practical experience more than anything else.
Regardless of any practical advice you may receive as a temporary
stopgap, there's always a chance that something else is going wrong,
and by reducing the number of concurrent requests you're just
decreasing its likelihood rather than properly fixing it.
I understand, and agree. I'm not looking for a stopgap, just maybe a
ballpark of "don't set more than 2000 in-flight deferreds at one time"
type of guideline.
I understand that every situation is different, I'm working to limit
my in-filght requests to a manageable number,
It's highly unlikely that it's actually the number of Deferreds. A
Deferred is just a Python object, so if you've got the RAM to store
them and their associated callbacks, you should be fine.
Yes, I understand that, thank you for clarifying.
It's more likely that it has something to do with long callback
chains, or hitting some kind of file-descriptor limit
The callback chains are short, and I'm not getting a file-descriptor
limit exception, or any exception that's getting percolated up.
(what version of Twisted are you using?)
Sorry for not including this earlier...
# python -V
Python 2.6.1
>>> import twisted
>>> twisted.__version__
'8.2.0'
I'm running right out of the release versions for these tests since
that's what my users will have installed.
or perhaps that 5000 pages is just a lot of pages to request and you
might need to wait a while.
Yes, it is a lot of stuff...
What I'm working on determining is whether limiting the number of "in-
flight" URL getters would beneficial.
Thanks,
S
Good luck,
-Glyph
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
