> On Jun 3, 2019, at 2:23 PM, Dk Jack <dnj0...@gmail.com> wrote:
>
> Yeah, I tried moving to 7.1.6. However, I ran into a couple of crashes with
> the new releases…
Those would be good to get reports on, and fixes :) Can you try the latest
7.1.1 RC0 that I just posted about?
— leif
>
> Bhasker.
>
> On Mon, Jun 3, 2019 at 12:20 PM Leif Hedstrom <zw...@apache.org> wrote:
>
>>
>>
>>> On Jun 2, 2019, at 6:24 PM, Dk Jack <dnj0...@gmail.com> wrote:
>>>
>>> Thanks Leif for responding,
>>>
>>> some questions...
>>>
>>> How do I turn off the freelist? Could you please elaborate on your RAM
>> disk
>>> reference? In my setup, HTTP cache is turned off. Would ram disk still
>> come
>>> into play?
>>
>> Optino to traffic_server:
>>
>> -f, --disable_freelist tog false Disable the freelist memory
>> allocator
>>
>>
>>>
>>> Is your suggestion 'kill -USR1' different from enabling
>>> 'proxy.config.dump_mem_info_frequency' config? The graphs I posted where
>>> made by enabling the dump memory config set to 15s frequency and turning
>>> the dumped stats into time series data.
>>
>> Yeh, same things, sorry.
>>
>> Also, note that ATS v6.2 is no longer supported, really need you to test
>> it on 7.x or 8.x I think to be able to properly debug.
>>
>> — Leif
>>
>>>
>>> Again, thanks for taking the time to respond...
>>>
>>> On Sun, Jun 2, 2019 at 12:52 PM Leif Hedstrom <zw...@apache.org> wrote:
>>>
>>>> Did you try turning off the freelist? If you do, you likely want to use
>>>> jemalloc or tcmalloc instead.
>>>>
>>>> If that stops the “leak”, then it’s likely related to how the RAM disk.
>>>>
>>>> The other thing to do is to kill -USR1 and look at the allocator usage.
>> Do
>>>> that a few times, with a few hours between and compare. There’s a
>> script in
>>>> the tools directory that lets you “diff” two such memory usage dumps.
>>>>
>>>> Cheers,
>>>>
>>>> — Leif
>>>>
>>>>> On Jun 2, 2019, at 12:41, Dk Jack <dnj0...@gmail.com> wrote:
>>>>>
>>>>> Does anyone have an idea of how much memory is allocated for processing
>>>>> each request? In this particular environment (from the graphs), it
>> looks
>>>>> like none of the memory allocated is being freed. Strange thing is,
>> with
>>>> no
>>>>> change to software of configuration (besides a restart of ats), the
>>>> memory
>>>>> consumption has gone up over the past couple of week with no
>> appreciable
>>>>> increase in traffic volumes. It used to leak about 2-3g a day and now
>> it
>>>>> has gone up to 12-14G a day.
>>>>>
>>>>> Any idea on where to start looking at and what to look for. I've
>> scoured
>>>> my
>>>>> plugin code a few times and it looks clean. The memory dumps show it's
>>>>> happening in the ATS code, using ATS allocators. Any help is
>> appreciated.
>>>>> Thanks.
>>>>>
>>>>>> On Fri, May 31, 2019 at 3:42 PM Dk Jack <dnj0...@gmail.com> wrote:
>>>>>>
>>>>>> stats collected via 'traffic_ctl metric ...' commands...
>>>>>>
>>>>>> https://www.dropbox.com/s/fmamnvrk5v1dq82/ats_6.2.1.txt?dl=0
>>>>>>
>>>>>>> On Fri, May 31, 2019 at 3:41 PM Dk Jack <dnj0...@gmail.com> wrote:
>>>>>>>
>>>>>>> No. I only have healthcheck plugin, stats plugin and my plugin.
>>>>>>>
>>>>>>> On Fri, May 31, 2019 at 2:45 PM Steve Malenfant <
>> smalenf...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Do you have stale while revalidation plug-in? If so, disable.
>>>>>>>>
>>>>>>>>> On Fri, May 31, 2019 at 5:42 PM Dk Jack <dnj0...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> I am running ATS 6.2.1 and I am seeing memory leaks. The link below
>>>>>>>> shows
>>>>>>>>> memory dump graphs for a half-hour period (dump freq is 15s). I
>> have
>>>> a
>>>>>>>>> custom plugin that's using atscppapi. These graphs are from our
>>>>>>>> production
>>>>>>>>> setup where the traffic volume is very high (120M+ req.s/day). We
>> are
>>>>>>>>> seeing memory growth of 6-8M every minute.
>>>>>>>>>
>>>>>>>>> https://www.dropbox.com/s/m03qdzm5iwl7y0w/ats_stats_6.2.1.pdf?dl=0
>>>>>>>>>
>>>>>>>>> In my test setup, I don't see the issue. Although, the volume is a
>>>> lot
>>>>>>>> less
>>>>>>>>> in my test setup. I've put debug logs in all places where ATS
>>>>>>>> allocators
>>>>>>>>> are showing growth and I see them properly being release in my test
>>>>>>>> setup.
>>>>>>>>> Is it possible, ATS is running into some error conditions causing
>>>> this
>>>>>>>>> leak? Any pointers on where or how to go about debugging this issue
>>>> is
>>>>>>>>> greatly appreciated...
>>>>>>>>>
>>>>>>>>> Dk.
>>>>>>>>>
>>>>>>>>> PS: I've tried to upgrade to 7.1.6, but I am running into some
>> other
>>>>>>>>> crashes with my custom plugin enabled.
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>
>>>>
>>
>>
