Specifically: don't pass nil Contexts. They are not valid, and most code that uses Contexts don't check for nil and will panic. On Sun, May 14, 2017 at 11:43 AM Sameer Ajmani <sam...@golang.org> wrote:
> Generally I'd suggest passing context.Background() when calling functions > that need a context from main or tests. > On Sat, May 13, 2017 at 8:27 AM Peter Weinberger (温博格) <p...@google.com> > wrote: > >> Hi. I was one of the people who failed in an attempt to auto-insert >> contexts in all of google3. I no longer remember all the obstacles I failed >> to overcome, but would encourage others to take on the project. >> >> One issue was libraries that were used both in paths from http requests >> (so they needed the context propagated through them) but were also used in >> places where their callers (starting at main) didn't have any contexts at >> all. Sitting by my coffee this morning this no longer seems like an >> insuperable obstacle (e.g., pass nils and change them to >> context.Background() inside, or, like appengine's log, split packages into >> one that requires contexts and one that doesn't). But I had lots of what I >> thought were plausible ideas that broke on the hard rock of google3, and I >> suspect these would too. >> >> (The project wasn't a total failure. It did plumb contexts through a >> bunch of packages. But the general case was too hard for me.) >> >> On Sat, May 13, 2017 at 3:50 AM, <mhhc...@gmail.com> wrote: >> >>> A reference i found about "reactive programming" >>> https://github.com/dsyer/reactive-notes/blob/master/intro.adoc >>> >>> Hope this introduced the concept correctly, thanks for pointing that. >>> >>> Following are only some thoughts, >>> >>> Two things surprising here, >>> 1/ a programmer prefers not to solve a problem >>> 2/ one failed attempt mean the end of all attempts** >>> >>> That being said, my little understanding so far is that current context >>> is used for two purposes, >>> - cancellation >>> - data (transmission? not sure) (+/- ts, have not checked i expect so in >>> regards to previous referenceS cited) >>> >>> While cancellation is a feature i really want to take advantage of, >>> the other one is much more arguable in the sense that >>> it can remains expressed purely by the end user without bloating >>> his productivity and quality (you don t really need it everywhere, just >>> where it is needed), >>> it seems to be highly software context dependent. >>> >>> Whereas cancellation, while it looks likes simple, is maybe more subtle. >>> After all it is about assigning uuid to a chain of call and >>> appropriately propagate, disjoint/rejoin new uuids with the previous >>> one, >>> so that we can ask to stop execution of a sub selection of a chain of >>> calls >>> via an handle. >>> Am i too simplistic? >>> >>> Its difficulty reside in its requirement to be passed absolutely >>> everywhere, >>> That reveals an important fact about the nature of cancellation, >>> it is there, it is everywhere, at all time (...within the program >>> lifetime), >>> it is somewhere in the background of every ops since the very beginning >>> the program began, >>> but not necessarily enabled, and certainly not a straight line. >>> >>> That is where the syntax might help to establish the plots >>> that the runtime consumes to connect the dots >>> and support what the developers want to achieve, >>> in my understanding so far. >>> >>> My best comparison to cancellation is >>> request rays tracing in micro services oriented architecture, >>> on both end it is a multi-tenant, >>> it always start with one ray, >>> the ray always split into multiple rays, >>> because we do programming, we need ways >>> to create, control distribute existing/new rays, >>> and possibly let the userland introduce a new behavior for the rays. >>> >>> So yeah basically main has a ray, >>> if you process an http request, >>> you to create a new ray >>> to be able to cancel only that request, >>> but it needs to be connected to the main ray >>> because if main should be canceled, >>> that signals should propagate to all rays connected with it, >>> probably. >>> >>> ** i want to put emphasis because in the description provided >>> by Sameer, as i tried to summarize, It has been tried to handle >>> the problem via the type system as if it was >>> a completely defined set of types. >>> Which in go is wrong (yeah i m very affirmative here :), >>> from my understanding, that might be correct in other oop languages. >>> Technically it is possible to rewrite interfaces, methods signatures, >>> taken individually, >>> as a whole, and from the consumer pov, >>> i d say it is an impossible task in go because it it goes against its >>> nature. >>> And i confirm/understand that by reading to Sameer feedback. >>> >>> Notes: i m a strong lover of go type system (not talking about values >>> and pointers ;) >>> >>> >>> On Friday, May 12, 2017 at 4:42:54 PM UTC+2, Henrik Johansson wrote: >>>> >>>> With the whole "Reactive" movement thread locals have started to vanish >>>> at least in the Java ecosystem. >>>> I agree with Sameer that, while convenient, it comes with a whole set >>>> of obscure bugs. >>>> >>>> On Fri, May 12, 2017, 14:57 Sameer Ajmani <sam...@golang.org> wrote: >>>> >>>>> Hmm, I'm not finding good open-source examples of ThreadLocal context >>>>> propagation in C++ and Java. My experience with this is based on what >>>>> Google uses internally. Perhaps someone more familiar with context use >>>>> (tracing?) outside Google can chime in? +Jaana Burcu Dogan >>>>> >>>>> On Thu, May 11, 2017 at 7:02 AM <mhh...@gmail.com> wrote: >>>>> >>>>>> thanks, >>>>>> >>>>>> ..integrating Context into the runtime.. >>>>>> >>>>>> >>>>>> 50% runtime, 50% syntax with explicit contextualization. >>>>>> >>>>>> ..The flow of request processing in Go may include multiple >>>>>> goroutines and may move across channels; >>>>>> >>>>>> >>>>>> yes big ? mark here. might the 50% of an handy and explicit syntax >>>>>> help with it? >>>>>> >>>>>> >>>>>> C++ and Java use thread-local Contexts, and while this is convenient, >>>>>> it is often a source of mysterious bugs. >>>>>> >>>>>> thanks! I don't know them, >>>>>> >>>>>> I quickly checked according to this >>>>>> >>>>>> https://dzone.com/articles/painless-introduction-javas-threadlocal-storage >>>>>> I may have totally wrong, the syntax does not look like, not even >>>>>> from a 100km, >>>>>> to how i represent go contexts in my head. >>>>>> >>>>>> This is more like the actor pattern dave cheney talks about in >>>>>> https://dave.cheney.net/2016/11/13/do-not-fear-first-class-functions >>>>>> (search for Let’s talk about actors) >>>>>> >>>>>> Is the dzone link correctly describe >>>>>> what you mentioned as being go context equivalent in java ? >>>>>> >>>>>> sorry my questions are so basic. >>>>>> >>>>>> >>>>>> On Thursday, May 11, 2017 at 12:29:11 PM UTC+2, Sameer Ajmani wrote: >>>>>> >>>>>>> I think you are asking whether we considered integrating Context >>>>>>> into the runtime, so that it does not need to be passed explicitly. >>>>>>> Yes, we >>>>>>> discussed this, but decided against it. The flow of request processing >>>>>>> in >>>>>>> Go may include multiple goroutines and may move across channels; we >>>>>>> decided >>>>>>> an explicit Context made this much easier to get right. C++ and Java use >>>>>>> thread-local Contexts, and while this is convenient, it is often a >>>>>>> source >>>>>>> of mysterious bugs. >>>>>>> >>>>>> On Thu, May 11, 2017 at 4:08 AM <mhh...@gmail.com> wrote: >>>>>>> >>>>>>>> Thanks a lot! >>>>>>>> >>>>>>>> Might i guess and try to generalize your explanations into >>>>>>>> "we tried to write a plumber for all cases possible" >>>>>>>> >>>>>>>> Which matters a lot, in my humble opinion. >>>>>>>> >>>>>>>> At least for the various reasons you put there, >>>>>>>> simply put, >>>>>>>> because it seems not technically achievable. >>>>>>>> >>>>>>>> Still i m happy you gave me those details, they are of interest >>>>>>>> indeed. >>>>>>>> >>>>>>>> I rather intent to solve the problem on a smaller surface >>>>>>>> with more predictability, less impossible solution to find. >>>>>>>> I hope so. >>>>>>>> And if it saves 90% of the time, that s already a win, imho. >>>>>>>> >>>>>>>> May i ask another question, >>>>>>>> have you considered to create the plumbing >>>>>>>> at runtime rather than statically ? >>>>>>>> With some intents from the syntax, necessarily >>>>>>>> (and yeah go 1 compat will be a problem, let s forget it for 1 >>>>>>>> minute). >>>>>>>> >>>>>>>> I suspect in some aspects in might be handier, >>>>>>>> because it might be all about chained func calls and not type >>>>>>>> system handling, >>>>>>>> and it might be easier to interleave the ctx.check in the flaw of >>>>>>>> ops, >>>>>>>> I don t know enough to realize for sure. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Wednesday, May 10, 2017 at 11:40:27 PM UTC+2, Sameer Ajmani >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Our approach was to identify function calls that consume a Context >>>>>>>>> (indicated by a call to context.TODO in the function body) and >>>>>>>>> function >>>>>>>>> calls that provide a Context (such as RPC and HTTP handlers). Then we >>>>>>>>> use >>>>>>>>> the guru callgraph tool to find all call paths from context providers >>>>>>>>> to >>>>>>>>> context consumers. These are the call paths that need to have a >>>>>>>>> context >>>>>>>>> plumbed through them. >>>>>>>>> >>>>>>>>> Starting from a context consumer, we can work upward to the >>>>>>>>> context provider, adding a context parameter to reach intervening >>>>>>>>> function >>>>>>>>> signature. Replace context.TODO in the consumer function body with >>>>>>>>> the new >>>>>>>>> ctx parameter, then update all the places that call the consumer to >>>>>>>>> pass >>>>>>>>> context.TODO. Now we have a new set of context consumers. Repeat >>>>>>>>> until you >>>>>>>>> reach the context providers (if you reach main or a Test function, >>>>>>>>> pass >>>>>>>>> context.Background instead). >>>>>>>>> >>>>>>>>> This works OK for static function calls but gets messy for dynamic >>>>>>>>> calls. If you need to add a context parameter to an interface method, >>>>>>>>> now >>>>>>>>> you have to update all implementations of that method, too (guru can >>>>>>>>> find >>>>>>>>> these for you). And if that interface is outside your control (like >>>>>>>>> io.Writer), you cannot change its signature, so you have to pass the >>>>>>>>> context some other way (such as via the method receiver). >>>>>>>>> >>>>>>>>> This gets yet more complicated if you cannot make atomic changes >>>>>>>>> to all callers of your functions, because callers may be in other >>>>>>>>> repositories. In this case, you must do an incremental refactoring in >>>>>>>>> multiple steps: each change to a function signature involves adding a >>>>>>>>> new >>>>>>>>> function that has the context parameter, then changing all existing >>>>>>>>> calls >>>>>>>>> to use the new function, while preventing new calls to the old >>>>>>>>> function, so >>>>>>>>> that you can finally delete it. >>>>>>>>> >>>>>>>>> Inside Google, we ended up not needing to build all this: Context >>>>>>>>> was introduced early enough that Go users could plumb it manually >>>>>>>>> where >>>>>>>>> needed. I think a context plumbing tool could still be interesting and >>>>>>>>> useful to other Go users. I'd love to see someone build it! >>>>>>>>> >>>>>>>>> S >>>>>>>>> >>>>>>>>> On Tue, May 9, 2017 at 10:54 AM <mhh...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> > I've done a limited form of this using awk ;-) >>>>>>>>>> >>>>>>>>>> if you have a minute, >>>>>>>>>> >>>>>>>>>> can you tell more about what limited you >>>>>>>>>> in your attempts and which trade made you stop (guessing), >>>>>>>>>> if any ? >>>>>>>>>> >>>>>>>>>> Do you still think it be awesome ? >>>>>>>>>> Or have you made your mind to an opposite position ? >>>>>>>>>> if so, For which reasons? >>>>>>>>>> >>>>>>>>>> My tool is very poor, consider it as on going, a place for >>>>>>>>>> inspiration to get started from absolutely no idea to lets get a >>>>>>>>>> dirty >>>>>>>>>> prototype. >>>>>>>>>> not sure yet how long is going to be the road, still digging :) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Tuesday, May 9, 2017 at 4:25:46 PM UTC+2, Sameer Ajmani wrote: >>>>>>>>>> >>>>>>>>>>> The eg tool can execute simple refactoring steps, but automating >>>>>>>>>>> context plumbing through a chain of calls is an open problem. Alan >>>>>>>>>>> Donovan >>>>>>>>>>> put some thought into this a few years ago, and I've done a limited >>>>>>>>>>> form of >>>>>>>>>>> this using awk ;-) >>>>>>>>>>> >>>>>>>>>> On Tue, May 9, 2017 at 6:10 AM <mhh...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>> I want something similar too. >>>>>>>>>>>> >>>>>>>>>>>> Automatic and smart insertion of context args in a chain of >>>>>>>>>>>> calls. >>>>>>>>>>>> >>>>>>>>>>>> Methods signature updates are easy, but how to appropriately >>>>>>>>>>>> insert context check in the ast ? >>>>>>>>>>>> I m not sure yet. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >The difficulty here seems to differentiate intra package calls >>>>>>>>>>>> from calls to standard/3rd party libraries which shouldn't be >>>>>>>>>>>> having new >>>>>>>>>>>> param. >>>>>>>>>>>> >>>>>>>>>>>> That does not sound too difficult, from the pkg identifier, >>>>>>>>>>>> lookup for the import path, for every import path, exists in >>>>>>>>>>>> GOROOT ? >>>>>>>>>>>> >>>>>>>>>>>> Please put updates here anything you want to share. >>>>>>>>>>>> >>>>>>>>>>>> At that moment i m using this package to help me with ast, >>>>>>>>>>>> https://github.com/mh-cbon/astutil >>>>>>>>>>>> >>>>>>>>>>>> might be a start even though it needs refactoring. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Monday, May 8, 2017 at 1:03:52 AM UTC+2, meir fischer wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> I'm adding tracing to an existing code base with many packages >>>>>>>>>>>>> and it seems the best way to have context's passed around is to >>>>>>>>>>>>> just have >>>>>>>>>>>>> every method take a context.Context. >>>>>>>>>>>>> >>>>>>>>>>>>> Is there any tooling for converting a code base/package to >>>>>>>>>>>>> have: >>>>>>>>>>>>> (a) context.Context as the first parameter in each function - >>>>>>>>>>>>> ctx context.Context >>>>>>>>>>>>> (b) for any function that has changed, have its callers >>>>>>>>>>>>> (within that package) pass ctx as the first arg >>>>>>>>>>>>> >>>>>>>>>>>>> The difficulty here seems to differentiate intra package calls >>>>>>>>>>>>> from calls to standard/3rd party libraries which shouldn't be >>>>>>>>>>>>> having new >>>>>>>>>>>>> param. >>>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>>> Google Groups "golang-nuts" group. >>>>>>>>>>>> >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>>> it, send an email to golang-nuts...@googlegroups.com. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "golang-nuts" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to golang-nuts...@googlegroups.com. >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "golang-nuts" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to golang-nuts...@googlegroups.com. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "golang-nuts" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to golang-nuts...@googlegroups.com. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "golang-nuts" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to golang-nuts...@googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
