For the specific case of managing long-running polling- or pubsub-based tasks, I would suggest basically the same pattern that others on the thread have already given: use a `sync.Map` to associate each task ID with a small struct containing a `context.CancelFunc` and a channel that can be used to confirm when the task has actually stopped (https://play.golang.org/p/Gq3SXuPFDvi).
That way you can have the invariant that after a successful call to Stop, the worker has completely finished all of its work. That property is extremely useful, for example, in writing tests and benchmarks for your implementation: if you can guarantee that the workers have actually stopped by the time Stop returns, your tests can detect unexpected deadlocks and your benchmarks can measure a controlled, well-defined amount of work. On Monday, August 30, 2021 at 7:52:59 AM UTC-4 Brian Candler wrote: > For that case, I'd suggest you pass a channel to each worker. It can > either close the channel, or send a specific message down it, to indicate > its termination. > > This may also be an X-Y problem. Generally speaking, "worker pools" are > an anti-pattern in Go. The startup and termination overhead of a goroutine > is so tiny that often it's better just to start a goroutine when there's > work to be done, and let it run to completion. The other reason you might > want to have a worker pool is to limit concurrency, but there are better > ways to do that in Go: e.g. you can have a buffered channel with N slots, > which the goroutine writes to before starting work and reads from when it's > finished. That will limit your concurrency to N tasks. > > Bryan C Mills discusses this in detail in this excellent video on > "Rethinking Classical Concurrency Patterns": > https://www.youtube.com/watch?v=5zXAHh5tJqQ > It took me several views with pauses to fully understand it, but it was > worth the effort :-) > > On Monday, 30 August 2021 at 12:29:26 UTC+1 sansaid wrote: > >> Appreciated, thanks Brian and Ian. >> >> On the subject of sync.WaitGroups, I haven't tackled this yet for my use >> case, but will use the opportunity to brainstorm some options, if that's >> okay. I want to be able to terminate each worker independently of others >> (i.e. close on worker, wait for it to finish its tasks, but allow other >> workers to run if they haven't been signalled to stop). Should I create a >> wait group per worker and store that in a map? This sounds bloated to me, >> especially since it's always going to be a wait group of 1, but I haven't >> come across better alternatives yet (relatively new to concurrency myself). >> >> If that's too broad a question, can come back once I've explored some >> options. >> >> On Mon, 30 Aug 2021, 12:20 Brian Candler, <b.ca...@pobox.com> wrote: >> >>> Also: you can use the same context for all the workers (if that's >>> appropriate); or if you want each worker to have a different context, you >>> can make those contexts children of the same top-level context. In that >>> case, you only need to cancel one context and all the workers will stop, >>> without having to keep a map of them. >>> >>> Do bear in mind that you may wish to wait for the workers to finish what >>> they're currently doing, before the program exits. One way to do that is >>> with a sync.WaitGroup, which you increment before starting each worker, and >>> each worker decrements just before it terminates. Then in your main >>> program you wait on the WaitGroup. >>> >>> Typical pattern: https://play.golang.org/p/51oy7ySbL4T >>> >>> On Monday, 30 August 2021 at 10:01:08 UTC+1 Ian Davis wrote: >>> >>>> >>>> On Sun, 29 Aug 2021, at 10:45 PM, sansaid wrote: >>>> >>>> Hello, >>>> >>>> Does anybody know of some reference code or common patterns I can use >>>> to keep track of worker goroutines? For context, I want a user to be able >>>> to issue a "stop" command using a CLI tool which will prompt my server to >>>> gracefully terminate one of my workers. >>>> >>>> For example, when a user issues a "start" command through a CLI tool, >>>> this will signal my server to spawn a goroutine that perpetually runs and >>>> polls an HTTP endpoint until a user initiates a "stop" command. When the >>>> user issues a "stop" command through the same CLI tool, I want the server >>>> to be able to signal the goroutine to stop. Any reference code would be >>>> much appreciated! >>>> >>>> One approach I thought of was by passing a `context.WithCancel()` and >>>> holding a reference to the cancel function in a global map (with the >>>> worker >>>> ID as keys). When the user issues a "stop" command against the worker ID, >>>> another function is executed which calls the context's cancel function. >>>> Example code below (appreciate this is horrendously breaking a lot of >>>> rules, but I want to focus only on the elements around passing the cancel >>>> function around - happy to discuss anything else that is of concern to >>>> someone though): >>>> >>>> ``` >>>> // invoked when a CLI start subcommand is issued to my main CLI tool >>>> func (w *WorkerGroup) Start(ctx context.Context, id string) { >>>> _, cancel := context.WithCancel(ctx) >>>> >>>> w.Workers.Lock() >>>> w.Workers[id] := cancel >>>> w.Workers.Unlock() >>>> >>>> go startWorker(id) >>>> } >>>> >>>> // invoked when a CLI stop subcommand is issued to my main CLI tool >>>> func (w *WorkerGroup) Stop(id string) { >>>> cancelWorker := w.Workers[id] >>>> >>>> cancelWorker() >>>> } >>>> ``` >>>> >>>> I have read that passing a context's cancel function around is a bad >>>> idea, but in an example like this, is it justified? >>>> >>>> >>>> Passing the cancel function is fine and storing it for use later is >>>> fine too. >>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to golang-nuts...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/golang-nuts/fdff94c4-c0e4-4418-abb3-eba91298fbb7n%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/golang-nuts/fdff94c4-c0e4-4418-abb3-eba91298fbb7n%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/7246761d-28a2-420b-a04b-c869c85194ccn%40googlegroups.com.
