I’ve been thinking about this some more, and I think that LockOSThread() should
not be needed - that the Go thread multiplexing must perform memory fences
otherwise the simplest of Go apps would have concurrency issues.
So, that leads me to believe that your “single routine” is not correct. I would
add code on the Go side that does similar Go global variable handling at the
call site for the C call. Then run under the race detector - I’m guessing that
it will report a race on the Go global.
> On Mar 16, 2020, at 2:46 PM, Robert Engels <reng...@ix.netcom.com> wrote:
>
> In the single Go routine, use LockOSThread(). Then it was always be accessed
> on the same thread removing the memory synchronization problems.
>
>> On Mar 16, 2020, at 11:28 AM, Nitish Saboo <nitish.sabo...@gmail.com> wrote:
>>
>>
>> Hi,
>>
>> So finally I got a little hint of the problem from what Robert described
>> earlier in the mail. Thank you so much Robert.
>> Looks like patterndb instance is not getting freed.
>>
>> node.c
>> ---------
>>
>> PatternDB *patterndb;
>>
>> int load_pattern_db(const gchar* file, key_value_cb cb)
>> {
>> if(patterndb != NULL){
>> printf("Patterndb Free called\n"); <<< Not getting printed
>> pattern_db_free(patterndb);
>> }
>> patterndb = pattern_db_new();
>> pattern_db_reload_ruleset(patterndb, configuration, file);
>> pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>> return 0;
>> }
>>
>>
>> patterndb is a global variable in C wrapper code that internally calls some
>> syslog-ng library api's.Since load_pattern_db() method is getting called
>> from a single goroutine every 3 mins, patterndb instance is not getting free
>> because the statement inside if clause ('if(patterndb != NULL)') is not
>> getting printed when I call 'load_pattern_db()' method.Looks like that is
>> the leak here.
>>
>>
>> 1)Can someone please help me understand the problem in detail as in why am I
>> facing this issue?
>>
>> 2)Though patterndb instance is a global variable in the C wrapper code, why
>> is it not getting freed?
>>
>> 3)How can I fix this issue?
>>
>> Thanks,
>> Nitish
>>
>>> On Mon, Mar 16, 2020 at 8:17 PM Nitish Saboo <nitish.sabo...@gmail.com>
>>> wrote:
>>> Hi Robert,
>>>
>>> Sorry I did not understand your point completely.
>>> I have a global variable patterndb on C side and It is getting called from
>>> a single goroutine every 3 mins. Why do I need to synchronize it?
>>> Even though the goroutine gets pinned to different threads, it can access
>>> the same global variable every time and free it ...right ?
>>>
>>> Thanks,
>>> Nitish
>>>
>>>
>>>> On Mon, Mar 16, 2020 at 8:10 PM Robert Engels <reng...@ix.netcom.com>
>>>> wrote:
>>>> Yes, you have a shared global variable you need to synchronize.
>>>>
>>>>> On Mar 16, 2020, at 9:35 AM, Nitish Saboo <nitish.sabo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> Are you saying it is working as expected?
>>>>>
>>>>> Thanks,
>>>>> Nitish
>>>>>
>>>>>> On Mon, Mar 16, 2020 at 7:42 PM Volker Dobler
>>>>>> <dr.volker.dob...@gmail.com> wrote:
>>>>>>> On Monday, 16 March 2020 14:25:52 UTC+1, Nitish Saboo wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I upgraded the go version and compiled the binary against go version
>>>>>>> 'go version go1.12.4 linux/amd64'.
>>>>>>> I ran the program for some time. I made almost 30-40 calls to the
>>>>>>> method Load_Pattern_Db().
>>>>>>> The program starts with 6% Mem Usage. The memory usage increases only
>>>>>>> when I call 'LoadPatternDb()' method and LoadPatternDb() method is
>>>>>>> called by a goroutine at regular intervals of 3 minutes(making use of
>>>>>>> ticker here ).
>>>>>>>
>>>>>>> What I observed is:
>>>>>>>
>>>>>>> 1)After almost 16-17 calls to the method 'LoadPatternDb(), the memory
>>>>>>> usage got almost constant at 29%. But I did not expect the program to
>>>>>>> take this much memory.
>>>>>>> When I restart the service the Mem Usage again starts with 6%.
>>>>>>>
>>>>>>> a) Is this the sign of memory leaking?
>>>>>>
>>>>>> No, as explained above.
>>>>>>
>>>>>>>
>>>>>>> b) Till this moment I did not see memory getting reclaimed or going
>>>>>>> down but it did become constant.
>>>>>>> As mentioned by experts above, the same sort of behavior is seen here.
>>>>>>> But I did not expect the memory usage to grow this much. Is this
>>>>>>> expected?
>>>>>> Yes. (Well, no. But your gut feeling of how much memory
>>>>>> should grow is not a suitable benchmark to compare
>>>>>> actual growth to.)
>>>>>>
>>>>>>>
>>>>>>> 2)I will run mem-profiling at intervals(10 minutes, 100 minutes..etc)
>>>>>>> as mentioned in the earlier email.
>>>>>>>
>>>>>>> a) Which all mem-stats variables should I look into for debugging this
>>>>>>> kind of behavior?
>>>>>> Alloc/HeapAlloc
>>>>>> But probably this is plain useless as nothing here indicates
>>>>>> that you do have any memory issues.
>>>>>>
>>>>>> V.
>>>>>>
>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "golang-nuts" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>> To view this discussion on the web visit
>>>>>> https://groups.google.com/d/msgid/golang-nuts/e664151d-474d-4c1d-ae1d-979dc6975469%40googlegroups.com.
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google Groups
>>>>> "golang-nuts" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send an
>>>>> email to golang-nuts+unsubscr...@googlegroups.com.
>>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/golang-nuts/CALjMrq7EuvpFBaAQCJfO_QhkW8ceac8oEv-oFq9GPsik%3D5GNkw%40mail.gmail.com.
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/2BD0A731-0F46-44DF-AEDE-8CC2F182D1B3%40ix.netcom.com.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CF994B1C-F870-4755-830C-AD067FF0CA36%40ix.netcom.com.
