Hi I am back with update. There is most certainly was memory leak connected with openCv. I don't know where is the problem yet. I can confirm this, because I have run test while mocking openCv out, there were no leaks of the type that was desbribed before. RSS was constant. Except there was one sudden rise of RSS to 6GB, but this was the problem in go code that I was able to profile with profiler. This was different than before.
I wonder why Valgrind doesn’t report those leaks? I will need some other methodology to track the problem down. But this is out of the scope of this discussion. Any ideas are welcome anyway :) Thanks. On Monday, May 8, 2017 at 8:40:11 AM UTC+2, miha....@visionect.com wrote: > > Hi Justin > > I don't think I am leaking goroutines or any other Go resources. Check > graphs in attachment. > > OpenCV and Zmq cgo bindings are used quite heavily. > About those I am sure about leaks, as, much I can relay on valgrind. Does > valgrind work reliably in Cgo environment? > > For the start I will stub out OpenCV dependency, to see what happens. Will > report back > > We have tried GraphicMagick and ImageMagic in the past and we have found > both them quite slow. > I don't know about OpenImageIO, but I will certainly check it out. > > Thanks > > Miha > > On Saturday, May 6, 2017 at 12:41:33 AM UTC+2, Justin Israel wrote: >> >> I also have done an image processing server in the past and used two >> different image library cgo bindings. Each one has its own specific way it >> manages the C memory and you can leak of you don't observe them. I am not >> familiar with the OpenCV binging though. >> >> Are you leaking any goroutines? How extensively are you using the OpenCV >> features, and can you swap to another library to confirm whether it is >> OpenCV or not? >> >> I used to use ImageMagick bindings (which I still maintain) and now I use >> OpenImageIO bindings (which I authored). I haven't had memory leak issues >> with these libs in current production use. >> >> Justin >> >> On Sat, May 6, 2017, 12:27 AM <miha....@visionect.com> wrote: >> >>> Hi >>> >>> >>> >>> <https://lh3.googleusercontent.com/-rQzo0h1nFmc/WQxWfRtgdaI/AAAAAAAAAEw/fyQ4N86p2csv0ktKCb5xaIB00hNSBrgagCLcB/s1600/rss.png> >>> >>> 1. System >>> >>> >>> Ubuntu 16.04.1 LTS \n \l >>> Linux G-00017 4.4.0-57-generic #78-Ubuntu SMP Fri Dec 9 23:50:32 UTC >>> 2016 x86_64 x86_64 x86_64 GNU/Linux >>> >>> <https://lh3.googleusercontent.com/-rQzo0h1nFmc/WQxWfRtgdaI/AAAAAAAAAEw/fyQ4N86p2csv0ktKCb5xaIB00hNSBrgagCLcB/s1600/rss.png> >>> >>> 16GB of RAM >>> >>> App runs inside docker: Docker version 17.03.1-ce, build c6d412e >>> >>> >>> >>> <https://lh3.googleusercontent.com/-rQzo0h1nFmc/WQxWfRtgdaI/AAAAAAAAAEw/fyQ4N86p2csv0ktKCb5xaIB00hNSBrgagCLcB/s1600/rss.png> >>> >>> 2. Memory usage problem >>> >>> I am having memory usage problems in production, where things do fail >>> because some Go processes use much more memory than expected. >>> >>> I have prepared a test, where I am running my my server application, >>> and I am measuring, how memory is consumed for one process. >>> >>> I fount out that more and more system memory is consumed with time. On >>> the graph below, you can see how RSS has risen from ~0.5GB up to ~6Gb in a >>> few days. >>> >>> >>> >>> <https://lh3.googleusercontent.com/-rQzo0h1nFmc/WQxWfRtgdaI/AAAAAAAAAEw/fyQ4N86p2csv0ktKCb5xaIB00hNSBrgagCLcB/s1600/rss.png> >>> >>> >>> >>> <https://lh3.googleusercontent.com/-rQzo0h1nFmc/WQxWfRtgdaI/AAAAAAAAAEw/fyQ4N86p2csv0ktKCb5xaIB00hNSBrgagCLcB/s1600/rss.png> >>> >>> >>> <https://lh3.googleusercontent.com/-rQzo0h1nFmc/WQxWfRtgdaI/AAAAAAAAAEw/fyQ4N86p2csv0ktKCb5xaIB00hNSBrgagCLcB/s1600/rss.png> >>> >>> >>> >>> >>> >>> >>> >>> The app does a lot of image processing for the images from remote source >>> (in the range of 100 images each second). All this processing, means a lot >>> of allocations, so naturally I tough, this must be memory a memory leak. >>> >>> So I run pprof (see attachment: pprof.log), which reports heap usage >>> below 300Mb. This heap usage, doesn't seam to change much over the time. >>> >>> Then, there are some CGO bindings with OpenCV, so I thought there must >>> be some leak in C code. I run the thing with Valgrind, which didn't find >>> anything except some minor leak, which I had already fixed by now. >>> >>> >>> >>> <https://lh3.googleusercontent.com/-rQzo0h1nFmc/WQxWfRtgdaI/AAAAAAAAAEw/fyQ4N86p2csv0ktKCb5xaIB00hNSBrgagCLcB/s1600/rss.png> >>> >>> I also run the app with GOGC=50, but it didn't help. >>> >>> >>> I am looking for the explanation what is happening, and how this can be >>> solved? >>> >>> >>> >>> Thanks! >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to golang-nuts...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
