How can I debug the number of threads and go routines running and checking if the Wait()'s finish? Because I believe that may be problem, that they hang.
And when you say append the output, are you saying make a go routine to write to the headers? If you have an example I would appreciate it On Wednesday, February 5, 2020 at 8:47:05 AM UTC-7, Robert Engels wrote: > > I think your problem may be > > "Depending on the HTTP protocol version and the client, calling > // Write or WriteHeader may prevent future reads on the > // Request.Body. For HTTP/1.x requests, handlers should read any > // needed request body data before writing the response. Once the > // headers have been flushed (due to either an explicit Flusher.Flush > // call or writing enough data to trigger a flush), the request body > // may be unavailable. For HTTP/2 requests, the Go HTTP server permits > // handlers to continue to read the request body while concurrently > // writing the response. However, such behavior may not be supported > // by all HTTP/2 clients. Handlers should read before writing if > // possible to maximize compatibility." > > You may need to write the ResponseHeader as a final stage and append the > output - if you write the header you may be hanging the input stages. If > the input stage hangs (you tube download hangs, etc.), the whole process is > going to hang. > > Did you debug the number of threads and go routines the process has while > running? I am betting these are continually increasing. (Another check > would be that all Waits() complete). > > Finally, I would use a CommandContext with a Deadline to ensure stragglers > are cleaned-up. > > > > -----Original Message----- > >From: Ian Lance Taylor <[email protected] <javascript:>> > >Sent: Feb 5, 2020 8:32 AM > >To: [email protected] <javascript:> > >Cc: golang-nuts <[email protected] <javascript:>> > >Subject: Re: [go-nuts] runtime/cgo: pthread_create failed: Resource > temporarily unavailable > > > >On Tue, Feb 4, 2020 at 11:22 PM <[email protected] <javascript:>> > wrote: > >> > >> I don't think that is the issue. I have tried it on a few different > servers. Most recent one with 100 gig's of ram and 50 cores. The load > average never goes above 9, but the ram slowly but surely on htop starts to > go up. The go binary ends up climbing slowly in it's ram use over time, > then after a hour or so, it reaches around 30 gigs of ram and then crashes, > and restarts. > >> > >> I have it under supervisor. > > > >That is not inconsistent with Robert's suggestion. If you are > >starting C threads that don't do any work but never exit, that is > >exactly what you would see. > > > >It's not the only possible cause of this. There could also be a space > >leak, either in C code with memory that is malloced but never freed, > >or in Go code with memory that something keeps a permanent reference > >to. > > > >Ian > > > > > >> On Tuesday, February 4, 2020 at 2:00:55 PM UTC-7, Robert Engels wrote: > >>> > >>> Are you certain you are not just starting too many processes? Ie use a > “worker pool” so you have at most N conversions happening at the same time. > >>> > >>> On Feb 4, 2020, at 2:34 PM, Robert Engels <[email protected]> > wrote: > >>> > >>> > >>> I will take a more in-depth look this evening. > >>> > >>> On Feb 4, 2020, at 2:19 PM, [email protected] wrote: > >>> > >>> > >>> I also thought the Wait() took care of closing the file descriptors? > Are you saying I should add a pipe3.Close()? Or a youtube.Close()? > >>> > >>> -- > >>> You received this message because you are subscribed to the Google > Groups "golang-nuts" group. > >>> To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > >>> To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/2cbf0d51-0b36-4c37-a324-8a343193769a%40googlegroups.com. > > > >>> > >>> -- > >>> You received this message because you are subscribed to the Google > Groups "golang-nuts" group. > >>> To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected]. > >>> To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/A56FB7F5-95D2-4B0E-B90C-B335B8714009%40ix.netcom.com. > > > >> > >> -- > >> You received this message because you are subscribed to the Google > Groups "golang-nuts" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an email to [email protected] <javascript:>. > >> To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/0e7e67a6-6ad4-4187-9f15-0d9f278b55a2%40googlegroups.com. > > > > > >-- > >You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > >To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected] <javascript:>. > >To view this discussion on the web visit > https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVeSsLHe92m8eQue4Lfyk3uf%2B%3D94Qy7ZapHdjsgA1RjhA%40mail.gmail.com. > > > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/d5b5800c-5170-4b1f-a760-7ab9def549ab%40googlegroups.com.
