I am not sure if that is indeed a better solution. I am probably better off delegating the responsibility of writing logs to a dedicated goroutine.
On Sun, Aug 5, 2018 at 10:51 PM, <grap...@berkeley.edu> wrote: > You can use a semaphore-like pattern described in > https://tip.golang.org/ref/mem to limit the number of goroutines trying > to concurrently write to the log file. This remove the need for the > buffering scheme. > > > On Sunday, August 5, 2018 at 9:32:06 AM UTC-7, Anuj Agrawal wrote: >> >> I have an application with more than 50k goroutines. I pass a context to >> each of these goroutines which contains a logger and that logger writes to >> a file. At some point, the number of goroutines trying to write a log >> peaked and my application crashed with - runtime: program exceeds >> 10000-thread limitfatal error: thread exhaustion. >> >> To provide a quick fix, I added a writer in between the logger and the >> file writer which would buffer all logs in a channel and a separate >> goroutine can then read from the channel and write to the file, thus, >> avoiding having to create so many OS threads. Something like: >> >> type writer struct { >> wg sync.WaitGroup >> buffer chan []byte >> out io.WriteCloser >> pool sync.Pool >> } >> >> func (w *writer) Write(p []byte) (int, error) { >> tmp := w.pool.Get().([]byte) >> tmp = append(tmp[:0], p...) >> w.buffer <- tmp >> return len(tmp), nil >> } >> >> go func() { >> for p := range w.buffer { >> if _, err := w.out.Write(p); err != nil { >> log.Printf("error writing to log file: %s\n%s\n", err.Error(), string(p)) >> } >> w.pool.Put(p) >> } >> w.wg.Done() >> }() >> >> >> >> I want to have suggestions if it can be fixed in a better way. >> > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
