Opened... https://github.com/apache/trafficserver/issues/5596
and https://github.com/apache/trafficserver/issues/5595 On Mon, Jun 3, 2019 at 1:25 PM Leif Hedstrom <zw...@apache.org> wrote: > > > > On Jun 3, 2019, at 2:23 PM, Dk Jack <dnj0...@gmail.com> wrote: > > > > Yeah, I tried moving to 7.1.6. However, I ran into a couple of crashes > with > > the new releases… > > Those would be good to get reports on, and fixes :) Can you try the latest > 7.1.1 RC0 that I just posted about? > > — leif > > > > > Bhasker. > > > > On Mon, Jun 3, 2019 at 12:20 PM Leif Hedstrom <zw...@apache.org> wrote: > > > >> > >> > >>> On Jun 2, 2019, at 6:24 PM, Dk Jack <dnj0...@gmail.com> wrote: > >>> > >>> Thanks Leif for responding, > >>> > >>> some questions... > >>> > >>> How do I turn off the freelist? Could you please elaborate on your RAM > >> disk > >>> reference? In my setup, HTTP cache is turned off. Would ram disk still > >> come > >>> into play? > >> > >> Optino to traffic_server: > >> > >> -f, --disable_freelist tog false Disable the freelist memory > >> allocator > >> > >> > >>> > >>> Is your suggestion 'kill -USR1' different from enabling > >>> 'proxy.config.dump_mem_info_frequency' config? The graphs I posted > where > >>> made by enabling the dump memory config set to 15s frequency and > turning > >>> the dumped stats into time series data. > >> > >> Yeh, same things, sorry. > >> > >> Also, note that ATS v6.2 is no longer supported, really need you to test > >> it on 7.x or 8.x I think to be able to properly debug. > >> > >> — Leif > >> > >>> > >>> Again, thanks for taking the time to respond... > >>> > >>> On Sun, Jun 2, 2019 at 12:52 PM Leif Hedstrom <zw...@apache.org> > wrote: > >>> > >>>> Did you try turning off the freelist? If you do, you likely want to > use > >>>> jemalloc or tcmalloc instead. > >>>> > >>>> If that stops the “leak”, then it’s likely related to how the RAM > disk. > >>>> > >>>> The other thing to do is to kill -USR1 and look at the allocator > usage. > >> Do > >>>> that a few times, with a few hours between and compare. There’s a > >> script in > >>>> the tools directory that lets you “diff” two such memory usage dumps. > >>>> > >>>> Cheers, > >>>> > >>>> — Leif > >>>> > >>>>> On Jun 2, 2019, at 12:41, Dk Jack <dnj0...@gmail.com> wrote: > >>>>> > >>>>> Does anyone have an idea of how much memory is allocated for > processing > >>>>> each request? In this particular environment (from the graphs), it > >> looks > >>>>> like none of the memory allocated is being freed. Strange thing is, > >> with > >>>> no > >>>>> change to software of configuration (besides a restart of ats), the > >>>> memory > >>>>> consumption has gone up over the past couple of week with no > >> appreciable > >>>>> increase in traffic volumes. It used to leak about 2-3g a day and now > >> it > >>>>> has gone up to 12-14G a day. > >>>>> > >>>>> Any idea on where to start looking at and what to look for. I've > >> scoured > >>>> my > >>>>> plugin code a few times and it looks clean. The memory dumps show > it's > >>>>> happening in the ATS code, using ATS allocators. Any help is > >> appreciated. > >>>>> Thanks. > >>>>> > >>>>>> On Fri, May 31, 2019 at 3:42 PM Dk Jack <dnj0...@gmail.com> wrote: > >>>>>> > >>>>>> stats collected via 'traffic_ctl metric ...' commands... > >>>>>> > >>>>>> https://www.dropbox.com/s/fmamnvrk5v1dq82/ats_6.2.1.txt?dl=0 > >>>>>> > >>>>>>> On Fri, May 31, 2019 at 3:41 PM Dk Jack <dnj0...@gmail.com> wrote: > >>>>>>> > >>>>>>> No. I only have healthcheck plugin, stats plugin and my plugin. > >>>>>>> > >>>>>>> On Fri, May 31, 2019 at 2:45 PM Steve Malenfant < > >> smalenf...@gmail.com> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Do you have stale while revalidation plug-in? If so, disable. > >>>>>>>> > >>>>>>>>> On Fri, May 31, 2019 at 5:42 PM Dk Jack <dnj0...@gmail.com> > wrote: > >>>>>>>>> > >>>>>>>>> Hi, > >>>>>>>>> I am running ATS 6.2.1 and I am seeing memory leaks. The link > below > >>>>>>>> shows > >>>>>>>>> memory dump graphs for a half-hour period (dump freq is 15s). I > >> have > >>>> a > >>>>>>>>> custom plugin that's using atscppapi. These graphs are from our > >>>>>>>> production > >>>>>>>>> setup where the traffic volume is very high (120M+ req.s/day). We > >> are > >>>>>>>>> seeing memory growth of 6-8M every minute. > >>>>>>>>> > >>>>>>>>> > https://www.dropbox.com/s/m03qdzm5iwl7y0w/ats_stats_6.2.1.pdf?dl=0 > >>>>>>>>> > >>>>>>>>> In my test setup, I don't see the issue. Although, the volume is > a > >>>> lot > >>>>>>>> less > >>>>>>>>> in my test setup. I've put debug logs in all places where ATS > >>>>>>>> allocators > >>>>>>>>> are showing growth and I see them properly being release in my > test > >>>>>>>> setup. > >>>>>>>>> Is it possible, ATS is running into some error conditions causing > >>>> this > >>>>>>>>> leak? Any pointers on where or how to go about debugging this > issue > >>>> is > >>>>>>>>> greatly appreciated... > >>>>>>>>> > >>>>>>>>> Dk. > >>>>>>>>> > >>>>>>>>> PS: I've tried to upgrade to 7.1.6, but I am running into some > >> other > >>>>>>>>> crashes with my custom plugin enabled. > >>>>>>>>> > >>>>>>>> > >>>>>>> > >>>> > >>>> > >> > >> > >
