Thanks Leif for responding, some questions...
How do I turn off the freelist? Could you please elaborate on your RAM disk reference? In my setup, HTTP cache is turned off. Would ram disk still come into play? Is your suggestion 'kill -USR1' different from enabling 'proxy.config.dump_mem_info_frequency' config? The graphs I posted where made by enabling the dump memory config set to 15s frequency and turning the dumped stats into time series data. Again, thanks for taking the time to respond... On Sun, Jun 2, 2019 at 12:52 PM Leif Hedstrom <zw...@apache.org> wrote: > Did you try turning off the freelist? If you do, you likely want to use > jemalloc or tcmalloc instead. > > If that stops the “leak”, then it’s likely related to how the RAM disk. > > The other thing to do is to kill -USR1 and look at the allocator usage. Do > that a few times, with a few hours between and compare. There’s a script in > the tools directory that lets you “diff” two such memory usage dumps. > > Cheers, > > — Leif > > > On Jun 2, 2019, at 12:41, Dk Jack <dnj0...@gmail.com> wrote: > > > > Does anyone have an idea of how much memory is allocated for processing > > each request? In this particular environment (from the graphs), it looks > > like none of the memory allocated is being freed. Strange thing is, with > no > > change to software of configuration (besides a restart of ats), the > memory > > consumption has gone up over the past couple of week with no appreciable > > increase in traffic volumes. It used to leak about 2-3g a day and now it > > has gone up to 12-14G a day. > > > > Any idea on where to start looking at and what to look for. I've scoured > my > > plugin code a few times and it looks clean. The memory dumps show it's > > happening in the ATS code, using ATS allocators. Any help is appreciated. > > Thanks. > > > >> On Fri, May 31, 2019 at 3:42 PM Dk Jack <dnj0...@gmail.com> wrote: > >> > >> stats collected via 'traffic_ctl metric ...' commands... > >> > >> https://www.dropbox.com/s/fmamnvrk5v1dq82/ats_6.2.1.txt?dl=0 > >> > >>> On Fri, May 31, 2019 at 3:41 PM Dk Jack <dnj0...@gmail.com> wrote: > >>> > >>> No. I only have healthcheck plugin, stats plugin and my plugin. > >>> > >>> On Fri, May 31, 2019 at 2:45 PM Steve Malenfant <smalenf...@gmail.com> > >>> wrote: > >>> > >>>> Do you have stale while revalidation plug-in? If so, disable. > >>>> > >>>>> On Fri, May 31, 2019 at 5:42 PM Dk Jack <dnj0...@gmail.com> wrote: > >>>>> > >>>>> Hi, > >>>>> I am running ATS 6.2.1 and I am seeing memory leaks. The link below > >>>> shows > >>>>> memory dump graphs for a half-hour period (dump freq is 15s). I have > a > >>>>> custom plugin that's using atscppapi. These graphs are from our > >>>> production > >>>>> setup where the traffic volume is very high (120M+ req.s/day). We are > >>>>> seeing memory growth of 6-8M every minute. > >>>>> > >>>>> https://www.dropbox.com/s/m03qdzm5iwl7y0w/ats_stats_6.2.1.pdf?dl=0 > >>>>> > >>>>> In my test setup, I don't see the issue. Although, the volume is a > lot > >>>> less > >>>>> in my test setup. I've put debug logs in all places where ATS > >>>> allocators > >>>>> are showing growth and I see them properly being release in my test > >>>> setup. > >>>>> Is it possible, ATS is running into some error conditions causing > this > >>>>> leak? Any pointers on where or how to go about debugging this issue > is > >>>>> greatly appreciated... > >>>>> > >>>>> Dk. > >>>>> > >>>>> PS: I've tried to upgrade to 7.1.6, but I am running into some other > >>>>> crashes with my custom plugin enabled. > >>>>> > >>>> > >>> > >
