My only thought was that maybe you had more Go routines accessing it than you
thought.
It is remains constant after a while it is most likely not a memory leak. It is
done what surprising that the memory consumption in a steady state would be 4x
the equivalent C program.
> On Mar 17, 2020, at 9:21 AM, Nitish Saboo <nitish.sabo...@gmail.com> wrote:
>
>
> Hi Robert,
>
> Thanks for your response.
> Since patterndb is a global variable(not a thread-local variable) and I have
> a single goroutine that calls load_pattern_db() method, therefore it was not
> looking correct to me to pin a goroutine to a thread.
> I once again tested the code flow. Apologies for making confusion in my last
> mail. When I called load_pattern_db() for about 6-7 times, every time the
> following lines were getting printed. It looks like patterndb instance is
> getting freed and the memory is becoming constant at around 29%.
>
> Patterndb Free Entered
> Patterndb Free called
> Patterndb New called
>
> node.c
> ---------
>
> PatternDB *patterndb;
>
> int load_pattern_db(const gchar* file, key_value_cb cb)
> {
> printf("Patterndb Free Entered\n")
> if(patterndb != NULL){
> printf("Patterndb Free called\n"); <<< It is getting printed
> pattern_db_free(patterndb);
> }
> printf("Patterndb New called\n")
> patterndb = pattern_db_new();
> pattern_db_reload_ruleset(patterndb, configuration, file);
> pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
> return 0;
> }
>
> But, what made you feel that Go global variable would report a race
> condition? Since it is a single goroutine what would cause a race condition
> here ?
>
> Thanks,
> Nitish
>
>> On Tue, Mar 17, 2020 at 6:31 PM Robert Engels <reng...@ix.netcom.com> wrote:
>> I’ve been thinking about this some more, and I think that LockOSThread()
>> should not be needed - that the Go thread multiplexing must perform memory
>> fences otherwise the simplest of Go apps would have concurrency issues.
>>
>> So, that leads me to believe that your “single routine” is not correct. I
>> would add code on the Go side that does similar Go global variable handling
>> at the call site for the C call. Then run under the race detector - I’m
>> guessing that it will report a race on the Go global.
>>
>>> On Mar 16, 2020, at 2:46 PM, Robert Engels <reng...@ix.netcom.com> wrote:
>>>
>>> In the single Go routine, use LockOSThread(). Then it was always be
>>> accessed on the same thread removing the memory synchronization problems.
>>>
>>>>> On Mar 16, 2020, at 11:28 AM, Nitish Saboo <nitish.sabo...@gmail.com>
>>>>> wrote:
>>>>>
>>>>
>>>> Hi,
>>>>
>>>> So finally I got a little hint of the problem from what Robert described
>>>> earlier in the mail. Thank you so much Robert.
>>>> Looks like patterndb instance is not getting freed.
>>>>
>>>> node.c
>>>> ---------
>>>>
>>>> PatternDB *patterndb;
>>>>
>>>> int load_pattern_db(const gchar* file, key_value_cb cb)
>>>> {
>>>> if(patterndb != NULL){
>>>> printf("Patterndb Free called\n"); <<< Not getting printed
>>>> pattern_db_free(patterndb);
>>>> }
>>>> patterndb = pattern_db_new();
>>>> pattern_db_reload_ruleset(patterndb, configuration, file);
>>>> pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>>>> return 0;
>>>> }
>>>>
>>>>
>>>> patterndb is a global variable in C wrapper code that internally calls
>>>> some syslog-ng library api's.Since load_pattern_db() method is getting
>>>> called from a single goroutine every 3 mins, patterndb instance is not
>>>> getting free because the statement inside if clause ('if(patterndb !=
>>>> NULL)') is not getting printed when I call 'load_pattern_db()'
>>>> method.Looks like that is the leak here.
>>>>
>>>>
>>>> 1)Can someone please help me understand the problem in detail as in why am
>>>> I facing this issue?
>>>>
>>>> 2)Though patterndb instance is a global variable in the C wrapper code,
>>>> why is it not getting freed?
>>>>
>>>> 3)How can I fix this issue?
>>>>
>>>> Thanks,
>>>> Nitish
>>>>
>>>>> On Mon, Mar 16, 2020 at 8:17 PM Nitish Saboo <nitish.sabo...@gmail.com>
>>>>> wrote:
>>>>> Hi Robert,
>>>>>
>>>>> Sorry I did not understand your point completely.
>>>>> I have a global variable patterndb on C side and It is getting called
>>>>> from a single goroutine every 3 mins. Why do I need to synchronize it?
>>>>> Even though the goroutine gets pinned to different threads, it can access
>>>>> the same global variable every time and free it ...right ?
>>>>>
>>>>> Thanks,
>>>>> Nitish
>>>>>
>>>>>
>>>>>> On Mon, Mar 16, 2020 at 8:10 PM Robert Engels <reng...@ix.netcom.com>
>>>>>> wrote:
>>>>>> Yes, you have a shared global variable you need to synchronize.
>>>>>>
>>>>>>>> On Mar 16, 2020, at 9:35 AM, Nitish Saboo <nitish.sabo...@gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Are you saying it is working as expected?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Nitish
>>>>>>>
>>>>>>>> On Mon, Mar 16, 2020 at 7:42 PM Volker Dobler
>>>>>>>> <dr.volker.dob...@gmail.com> wrote:
>>>>>>>>> On Monday, 16 March 2020 14:25:52 UTC+1, Nitish Saboo wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I upgraded the go version and compiled the binary against go version
>>>>>>>>> 'go version go1.12.4 linux/amd64'.
>>>>>>>>> I ran the program for some time. I made almost 30-40 calls to the
>>>>>>>>> method Load_Pattern_Db().
>>>>>>>>> The program starts with 6% Mem Usage. The memory usage increases only
>>>>>>>>> when I call 'LoadPatternDb()' method and LoadPatternDb() method is
>>>>>>>>> called by a goroutine at regular intervals of 3 minutes(making use of
>>>>>>>>> ticker here ).
>>>>>>>>>
>>>>>>>>> What I observed is:
>>>>>>>>>
>>>>>>>>> 1)After almost 16-17 calls to the method 'LoadPatternDb(), the memory
>>>>>>>>> usage got almost constant at 29%. But I did not expect the program to
>>>>>>>>> take this much memory.
>>>>>>>>> When I restart the service the Mem Usage again starts with 6%.
>>>>>>>>>
>>>>>>>>> a) Is this the sign of memory leaking?
>>>>>>>>
>>>>>>>> No, as explained above.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> b) Till this moment I did not see memory getting reclaimed or going
>>>>>>>>> down but it did become constant.
>>>>>>>>> As mentioned by experts above, the same sort of behavior is seen
>>>>>>>>> here. But I did not expect the memory usage to grow this much. Is
>>>>>>>>> this expected?
>>>>>>>> Yes. (Well, no. But your gut feeling of how much memory
>>>>>>>> should grow is not a suitable benchmark to compare
>>>>>>>> actual growth to.)
>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2)I will run mem-profiling at intervals(10 minutes, 100 minutes..etc)
>>>>>>>>> as mentioned in the earlier email.
>>>>>>>>>
>>>>>>>>> a) Which all mem-stats variables should I look into for debugging
>>>>>>>>> this kind of behavior?
>>>>>>>> Alloc/HeapAlloc
>>>>>>>> But probably this is plain useless as nothing here indicates
>>>>>>>> that you do have any memory issues.
>>>>>>>>
>>>>>>>> V.
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "golang-nuts" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>>>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/e664151d-474d-4c1d-ae1d-979dc6975469%40googlegroups.com.
>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "golang-nuts" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>>>>>> To view this discussion on the web visit
>>>>>>> https://groups.google.com/d/msgid/golang-nuts/CALjMrq7EuvpFBaAQCJfO_QhkW8ceac8oEv-oFq9GPsik%3D5GNkw%40mail.gmail.com.
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an
>>> email to golang-nuts+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/2BD0A731-0F46-44DF-AEDE-8CC2F182D1B3%40ix.netcom.com.
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/798AC492-766D-4C71-8215-8AB2B6691E83%40ix.netcom.com.
