Thanks for the offer to help.
I was hoping someone would see an apparent bug in the outline, so i wouldn't
have to build a a SSCCE
unfortunately, that wasn't going to fly, so I built out a self-contained
version of the issue
Before sharing it, I added in some docs references to the example...
and then I noticed something peculiar, and seemed to have solved the problem !
the issue was this:
1. I used twisted.internet.defer.DeferredSemaphore to set up a semaphore service
2. I queued tasks with `semaphoreService.run( to_thread_function )`
3. `to_thread_function` ran a configurable method through
`threads.deferToThread`
4. the configurable method ran something in
twisted.enterprise.adbapi.ConnectionPool's `runInteraction`
when copying docs, i realized that I was running `toThread` and then
`runInteraction` , which uses it's own thread. so i had threads spawning
threads.
the base 'scaffold' for this daemon has been modified and patched since 2005 ,
so at some point i made an improvement and left some semi-functional legacy
cruft in there.
i'm not sure of the specifics on how / why this manifests, but if I just use
runInteraction and bypass using `deferToThread`, everything works out perfect.
if you're curious , i tossed the example online here
https://github.com/jvanasco/twisted_gist_001
`constants.py` has some toggles for playing with the return values of
the various functions ( controls the base class and subclass )
it also lets you toggle to use the broken functionality ( thread within
a thread ) or what seems to work fine now
happy i seemed to have solved this myself. still confused why the issue
happened , but this fix ( only 1 thread ) seems to be fine and the more ideal
approach
On Jan 18, 2014, at 7:17 PM, GMail wrote:
> Hi Jonathan,
>
> On Jan 17, 2014, at 6:22 PM, Jonathan Vanasco <twisted-pyt...@2xlp.com> wrote:
>
>> the following is a rough composite of what is going on. if anyone sees an
>> obvious fix, i'd be greatly appreciative.
>
> I'd love to help, but a "rough composite" is hard to make guesses about,
> especially since you're talking about hard-to-predict memory-consumption
> behavior.
>
> Can you attach a <http://sscce.org/> which is actually runnable, for example,
> with a canned list of input URLs (or better yet with an included web server
> so the URLs can be localhost and more predictable), so we can debug and
> diagnose a running program instead of ideas about the outline of one?
>
> -glyph
>
> _______________________________________________
> Twisted-Python mailing list
> Twisted-Python@twistedmatrix.com
> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________
Twisted-Python mailing list
Twisted-Python@twistedmatrix.com
http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
