> On Oct 18, 2016, at 4:05 PM, Kevin Conway <kevinjacobcon...@gmail.com> wrote: > > > This is why twisted.python.context came to exist in the first place; I > > always wanted to attach it to Deferred somehow > > Well, it's not something we've announced yet through any official channel, > but we had to solve the context propagation problem at Atlassian to > instrument our services with traceable logging. We open sourced our solution > at https://bitbucket.org/hipchat/txlocal > <https://bitbucket.org/hipchat/txlocal>. The answer for us was an extension > for the reactor, thread pool, and inline callbacks that maintain the needed > state. There's a readme with some insight into how we tool our services. > This is (A) very cool, and (B) making such aggressive use of private APIs that it could win a contest about how to ensure that you break on every new release of Twisted :). I'm super impressed that you tracked the introduction of twisted._threads and support both old- and new-style thread pools! > We've had it on our backlog to address the mailing list and, possibly even, > discuss what it would take to put this into Twisted. I guess now is as good > if a time as any. Feel free to spin off a another thread or reach out to me > off list with any questions or feedback > This seems like as good a time to talk about it as any! Integrating this into the core in some fashion would be good, but I imagine that it's a non-trivial impact to performance, so it would be worthwhile to track that.
Speaking of performance - I found the long digression on "Don't Switch In The Core" interesting, since we actually _do_ tracking this kind of context already, for logging. I was a little surprised you didn't integrate with this at all. To be fair, this is something that our friends over at PyPy have been bugging us about since forever; when you're benchmarking raw wire speed it does tend to show up in profiling. I also take it from the performance notes that you're not using PyPy? __slots__ shouldn't make much of a difference there. (In fact I'm given to believe it's a slight _decrease_ in performance on pypy...) -glyph > On Tue, Oct 18, 2016, 14:47 Glyph Lefkowitz <gl...@twistedmatrix.com > <mailto:gl...@twistedmatrix.com>> wrote: >> On Oct 18, 2016, at 5:50 AM, Itamar Turner-Trauring <ita...@itamarst.org >> <mailto:ita...@itamarst.org>> wrote: >> >> Not been doing much Twisted lately, but have been doing async stuff >> elsewhere, and I've learned some useful things. > > Thanks for writing these up, Itamar! This sort of reflection is rare and > it's always helpful :). > >> 1. Callbacks should be sync or async, but never >> sometimes-one-sometimes-the-other. For details go read >> http://blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/ >> <http://blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/>. >> For example, Deferred.addCallback(f) really should never run f() >> immediately. > > This has come up a lot in a compare-and-contrast of Twisted vs. asyncio. > > I agree that the problems with synchronous callbacks are not insignificant > (reentrancy is a degenerate form of preemption, and as we all know preemption > is the corrupt wellspring of all bugs). However, the benefit, i.e. > consistency of behavior with respect to reentrancy, comes with a cost: tight > coupling to an event loop. In asyncio, Future's tight coupling to call_soon > is a source of problems; it makes it hard to write a test without setting up > an elaborate scheduling trampoline, whereas successResultOf/failureResultOf > are quite simple to work with. > > I think Deferred as it is today is a pretty good compromise between the two > positions. On the one hand it is decoupled from the event loop. On the > other - and this is important - no Deferred-returning API will ever call your > callbacks synchronously. Deferred.addCallback will, of course, but savvy > Twisted programmers can (and should) do this, if they have dependent state > changes: > > self.manipulateSomeStateForSetup() > d = doSomethingPotentiallySynchronous() > self.manipulateSomeStateForProcessing() > d.addCallback(completeOperation) > > As a caller, you can always decide whether you can safely be re-entered or > not. In most cases, simply moving the 'addCallback' to the end of the > function (a-la Go's "defer", oddly enough) is fine. In more complex cases > where you really need to unwind reentrancy completely, you can do your own > callLater(0) or callFromThread() from an object with a reference to a reactor. > >> 3. > > What happened to '2'? :) > >> By instrumenting all callbacks it manages, which may or may not >> require item #1, Twisted can have a context that automatically follows >> callbacks. Node has this and it is extremely useful. >> http://fredkschott.com/post/2014/02/conquering-asynchronous-context-with-cls/ >> >> <http://fredkschott.com/post/2014/02/conquering-asynchronous-context-with-cls/> >> is best summary I've found with a bit of searching. > > This was _always_ supposed to be the way that Twisted worked, but frankly I > just wasn't smart enough to figure it out. This is why > twisted.python.context came to exist in the first place; I always wanted to > attach it to Deferred somehow. I will watch this talk intently; if #1 really > is required to address this, my opinion might change. A PR would be > intensely appreciated. > > -glyph > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com> > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > <http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python> > _______________________________________________ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
