>
> What cgo calls were those? Are you using an alternative crypto
> implentation?
Sadly, I didn't save any of the metrics I gathered any more, so I can't say
specifically what CGO calls were being used by my application. I do recall
a good bit of the memory being spent on checking the certificate with the
system the application was running on... but my memory is foggy after a
long day.
I was not using any alternative crypo implemenations. Straight out of the
box native http.Client with no alterations.
Also quite curious about this. My assumption was that all resources should
> eventually be cleaned up when http.Client goes out of scope?
...Honestly, I'm not sure entirely what was going on, as I'm not too
familiar with Go's GC. Running my build with -gcflags -m shows that the
HTTP client in question is escaping to the heap. My expectation was also
that it would be cleared eventually. From what I could tell from profiling
and a few other things, the memory gets allocated to the application and
*maybe* gets released back to the application, but not the host. I say
maybe because profiling does not get everything, and upon profiling I would
see about 300mb being used but my host (again, Windows) was showing
500mb-1.3gb being dedicated at the same time. Now, why this memory doesn't
get reused by the application is, frankly, a mystery to me. It seems to
just provision more and more, even though it doesn't need it.
If you want to replicate... My setup essentially had multiple goroutines
calling a single method that creates a new http.Client with every call to
execute the request. No extra configuration, no customized TLS, just your
basic &http.Client{} or something similar. The only thing you need to do is
ensure that the client uses HTTPS. I also used a Redis client for the job
queue and Mongo for the database, but the resources used there were
negligible in comparison.
If you want something you can throw the might of Go at without having to
worry about rate limiting, here's the home page for the API
<https://esi.tech.ccp.is/latest/> I was consuming in this case. Only thing
you have to worry about is obeying cache rules from the headers.
And no, I wasn't using swagger... mostly because windows doesn't like long
file paths.
With that setup, and 5 worker threads, I would quickly reach 300-500mb
withing a few minutes, and peak at 1.2-1.3gb before it would stop
functioning entirely.
Ultimately, it just comes down to me doing a quick-n-dirty setup without
really thinking. I fixed it by making a client that is reused for every
call, and happily sit at 9-12mb in memory without issue.
P.S. Roughly the same application didn't have this issue on older Go
versions... 1.8 is when I really started noticing it. Possibly a new bug?
P.S.S. Sorry for typos/grammar etc... long day and the T key is acting up.
On Wed, Mar 15, 2017 at 10:11 PM, Tom <hyper...@gmail.com> wrote:
> Also quite curious about this. My assumption was that all resources should
> eventually be cleaned up when http.Client goes out of scope?
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/
> topic/golang-nuts/DIXihO2sLRE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
