I think your problem may be
"Depending on the HTTP protocol version and the client, calling
// Write or WriteHeader may prevent future reads on the
// Request.Body. For HTTP/1.x requests, handlers should read any
// needed request body data before writing the response. Once the
// headers have been flushed (due to either an explicit Flusher.Flush
// call or writing enough data to trigger a flush), the request body
// may be unavailable. For HTTP/2 requests, the Go HTTP server permits
// handlers to continue to read the request body while concurrently
// writing the response. However, such behavior may not be supported
// by all HTTP/2 clients. Handlers should read before writing if
// possible to maximize compatibility."
You may need to write the ResponseHeader as a final stage and append the output
- if you write the header you may be hanging the input stages. If the input
stage hangs (you tube download hangs, etc.), the whole process is going to hang.
Did you debug the number of threads and go routines the process has while
running? I am betting these are continually increasing. (Another check would be
that all Waits() complete).
Finally, I would use a CommandContext with a Deadline to ensure stragglers are
cleaned-up.
-----Original Message-----
>From: Ian Lance Taylor <i...@golang.org>
>Sent: Feb 5, 2020 8:32 AM
>To: jnaderl...@gmail.com
>Cc: golang-nuts <golang-nuts@googlegroups.com>
>Subject: Re: [go-nuts] runtime/cgo: pthread_create failed: Resource
>temporarily unavailable
>
>On Tue, Feb 4, 2020 at 11:22 PM <jnaderl...@gmail.com> wrote:
>>
>> I don't think that is the issue. I have tried it on a few different
>> servers. Most recent one with 100 gig's of ram and 50 cores. The load
>> average never goes above 9, but the ram slowly but surely on htop starts to
>> go up. The go binary ends up climbing slowly in it's ram use over time,
>> then after a hour or so, it reaches around 30 gigs of ram and then crashes,
>> and restarts.
>>
>> I have it under supervisor.
>
>That is not inconsistent with Robert's suggestion. If you are
>starting C threads that don't do any work but never exit, that is
>exactly what you would see.
>
>It's not the only possible cause of this. There could also be a space
>leak, either in C code with memory that is malloced but never freed,
>or in Go code with memory that something keeps a permanent reference
>to.
>
>Ian
>
>
>> On Tuesday, February 4, 2020 at 2:00:55 PM UTC-7, Robert Engels wrote:
>>>
>>> Are you certain you are not just starting too many processes? Ie use a
>>> “worker pool” so you have at most N conversions happening at the same time.
>>>
>>> On Feb 4, 2020, at 2:34 PM, Robert Engels <ren...@ix.netcom.com> wrote:
>>>
>>>
>>> I will take a more in-depth look this evening.
>>>
>>> On Feb 4, 2020, at 2:19 PM, jnade...@gmail.com wrote:
>>>
>>>
>>> I also thought the Wait() took care of closing the file descriptors? Are
>>> you saying I should add a pipe3.Close()? Or a youtube.Close()?
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to golan...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/2cbf0d51-0b36-4c37-a324-8a343193769a%40googlegroups.com.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to golan...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/A56FB7F5-95D2-4B0E-B90C-B335B8714009%40ix.netcom.com.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/0e7e67a6-6ad4-4187-9f15-0d9f278b55a2%40googlegroups.com.
>
>--
>You received this message because you are subscribed to the Google Groups
>"golang-nuts" group.
>To unsubscribe from this group and stop receiving emails from it, send an
>email to golang-nuts+unsubscr...@googlegroups.com.
>To view this discussion on the web visit
>https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVeSsLHe92m8eQue4Lfyk3uf%2B%3D94Qy7ZapHdjsgA1RjhA%40mail.gmail.com.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/2131046265.665.1580917592645%40wamui-scooby.atl.sa.earthlink.net.
