Yeah, I tried moving to 7.1.6. However, I ran into a couple of crashes with the new releases...
Bhasker. On Mon, Jun 3, 2019 at 12:20 PM Leif Hedstrom <zw...@apache.org> wrote: > > > > On Jun 2, 2019, at 6:24 PM, Dk Jack <dnj0...@gmail.com> wrote: > > > > Thanks Leif for responding, > > > > some questions... > > > > How do I turn off the freelist? Could you please elaborate on your RAM > disk > > reference? In my setup, HTTP cache is turned off. Would ram disk still > come > > into play? > > Optino to traffic_server: > > -f, --disable_freelist tog false Disable the freelist memory > allocator > > > > > > Is your suggestion 'kill -USR1' different from enabling > > 'proxy.config.dump_mem_info_frequency' config? The graphs I posted where > > made by enabling the dump memory config set to 15s frequency and turning > > the dumped stats into time series data. > > Yeh, same things, sorry. > > Also, note that ATS v6.2 is no longer supported, really need you to test > it on 7.x or 8.x I think to be able to properly debug. > > — Leif > > > > > Again, thanks for taking the time to respond... > > > > On Sun, Jun 2, 2019 at 12:52 PM Leif Hedstrom <zw...@apache.org> wrote: > > > >> Did you try turning off the freelist? If you do, you likely want to use > >> jemalloc or tcmalloc instead. > >> > >> If that stops the “leak”, then it’s likely related to how the RAM disk. > >> > >> The other thing to do is to kill -USR1 and look at the allocator usage. > Do > >> that a few times, with a few hours between and compare. There’s a > script in > >> the tools directory that lets you “diff” two such memory usage dumps. > >> > >> Cheers, > >> > >> — Leif > >> > >>> On Jun 2, 2019, at 12:41, Dk Jack <dnj0...@gmail.com> wrote: > >>> > >>> Does anyone have an idea of how much memory is allocated for processing > >>> each request? In this particular environment (from the graphs), it > looks > >>> like none of the memory allocated is being freed. Strange thing is, > with > >> no > >>> change to software of configuration (besides a restart of ats), the > >> memory > >>> consumption has gone up over the past couple of week with no > appreciable > >>> increase in traffic volumes. It used to leak about 2-3g a day and now > it > >>> has gone up to 12-14G a day. > >>> > >>> Any idea on where to start looking at and what to look for. I've > scoured > >> my > >>> plugin code a few times and it looks clean. The memory dumps show it's > >>> happening in the ATS code, using ATS allocators. Any help is > appreciated. > >>> Thanks. > >>> > >>>> On Fri, May 31, 2019 at 3:42 PM Dk Jack <dnj0...@gmail.com> wrote: > >>>> > >>>> stats collected via 'traffic_ctl metric ...' commands... > >>>> > >>>> https://www.dropbox.com/s/fmamnvrk5v1dq82/ats_6.2.1.txt?dl=0 > >>>> > >>>>> On Fri, May 31, 2019 at 3:41 PM Dk Jack <dnj0...@gmail.com> wrote: > >>>>> > >>>>> No. I only have healthcheck plugin, stats plugin and my plugin. > >>>>> > >>>>> On Fri, May 31, 2019 at 2:45 PM Steve Malenfant < > smalenf...@gmail.com> > >>>>> wrote: > >>>>> > >>>>>> Do you have stale while revalidation plug-in? If so, disable. > >>>>>> > >>>>>>> On Fri, May 31, 2019 at 5:42 PM Dk Jack <dnj0...@gmail.com> wrote: > >>>>>>> > >>>>>>> Hi, > >>>>>>> I am running ATS 6.2.1 and I am seeing memory leaks. The link below > >>>>>> shows > >>>>>>> memory dump graphs for a half-hour period (dump freq is 15s). I > have > >> a > >>>>>>> custom plugin that's using atscppapi. These graphs are from our > >>>>>> production > >>>>>>> setup where the traffic volume is very high (120M+ req.s/day). We > are > >>>>>>> seeing memory growth of 6-8M every minute. > >>>>>>> > >>>>>>> https://www.dropbox.com/s/m03qdzm5iwl7y0w/ats_stats_6.2.1.pdf?dl=0 > >>>>>>> > >>>>>>> In my test setup, I don't see the issue. Although, the volume is a > >> lot > >>>>>> less > >>>>>>> in my test setup. I've put debug logs in all places where ATS > >>>>>> allocators > >>>>>>> are showing growth and I see them properly being release in my test > >>>>>> setup. > >>>>>>> Is it possible, ATS is running into some error conditions causing > >> this > >>>>>>> leak? Any pointers on where or how to go about debugging this issue > >> is > >>>>>>> greatly appreciated... > >>>>>>> > >>>>>>> Dk. > >>>>>>> > >>>>>>> PS: I've tried to upgrade to 7.1.6, but I am running into some > other > >>>>>>> crashes with my custom plugin enabled. > >>>>>>> > >>>>>> > >>>>> > >> > >> > >
