You can add goroutines to a WaitGroup as you create them. There is nothing that keeps you from calling Add more than once.
Andy > On Nov 1, 2017, at 11:10 PM, kanth...@gmail.com wrote: > > I am new to Go and I had read numerous times in various articles that one can > technically create any number of Go routines and the runtime takes care of > them mapping to kernel threads. If so, why does waitGroup designed in a way > such that one has to specify how many go routines one need to create ahead of > time ? why can't waitGroup be more dynamic such that I can add a Goroutine to > waitgroup once I create it ? To put the question in another way why are > people even thinking about pool of go routines in the first place given that > goroutines are user level threads? If someone talks about pool of goroutines > I feel like I am back to C++ or any other language that creates pool of > kernel threads. Does Go runtimes doesn't keep its promise? Again, I am new so > please enlighten me. > > On Wednesday, June 29, 2016 at 3:35:31 PM UTC-7, Inspectre Gadget wrote: > Hey everyone, > > Here’s my issue, I will try to keep this short and concise: > > I have written a program that will accept a URL, spider that URL’s domain and > scheme (http/https), and return back all input fields found throughout to the > console. The purpose is largely for web application security testing, as > input fields are the most common vulnerability entry points (sinks), and this > program automates that part of the reconnaissance phase. > > Here is the problematic code: > https://github.com/insp3ctre/input-field-finder/blob/ce7983bd336ad59b2e2b613868e49dfb44110d09/main.go > > <https://github.com/insp3ctre/input-field-finder/blob/ce7983bd336ad59b2e2b613868e49dfb44110d09/main.go> > > The issue lies in the last for loop in the main() function. If you were to > run this program, it would check the queue and workers so frequently that it > is bound to find a point where there are both no workers working, and no URLs > in the queue (as proved by the console output statements before it exits). > Nevermind that the problem is exacerbated by network latency. The number of > URLs actually checked varies on every run, which causes some serious > inconsistencies, and prevents the program from being at all reliable. > > The issue was fixed here: > https://github.com/insp3ctre/input-field-finder/blob/f0032bb550ced0b323e63be9c4f40d644257abcd/main.go > > <https://github.com/insp3ctre/input-field-finder/blob/f0032bb550ced0b323e63be9c4f40d644257abcd/main.go> > > I fixed it by removing all concurrency from network requests, leaving it only > in the internal HTML processing functions. > > So, the question is- how does one run efficient concurrent code when the > number of wait groups is dynamic, and unknown at program initialization? > > I have tried: > Using “worker pools”, which consist of channels of workers. The for loop > checks the length of the URL queue and the number of workers available. If > the URL queue is empty and all the workers are available, then it exits the > loop. > Dynamically adding wait groups (wg.Add(1)) every time I pull a URL from the > URL queue. I can’t set the wait group numbers before the loop, because I can > never know how many URLs are going to be checked. > > So I have tried using both channels and wait groups to check alongside the > URL queue length to determine whether more concurrent network requests are > needed. In both cases, the for loop checks the values so fast that it > eventually stumbles upon a non-satisfied loop condition, and exits. This > usually results in either the program hanging as it waits for wait groups to > exit that never do, or it simply exits prematurely, as more URLs are added to > the queue after the fact. > > I would really like to know if there is a way to actually do this well in Go. > > Cheers, > > Inspectre > > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com > <mailto:golang-nuts+unsubscr...@googlegroups.com>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
