Hi Robert,
Looks like pattern_db is a global. Are you sure you don’t have multiple Go
routines calling the load? The global changes may not be visible- so the
free is doing nothing.
patterndb is a global variable on C side.load method is getting called from
a single go routine every 3 mins.I used gdb to put a break point on
pattern_db_free and that is getting called.
The global changes may not be visible- so the free is doing nothing.
>>Since it is a global variable it should be visible..right?
You probably need synchronization or pinning the access routine to a
thread.
>>Why we need to pin routine to a thread ?
Thanks,
Nitish
On Mon, Mar 16, 2020 at 7:41 PM Robert Engels <reng...@ix.netcom.com> wrote:
> Looks like pattern_db is a global. Are you sure you don’t have multiple Go
> routines calling the load? The global changes may not be visible- so the
> free is doing nothing.
>
> You probably need synchronization or pinning the access routine to a
> thread.
>
> On Mar 16, 2020, at 9:03 AM, Nitish Saboo <nitish.sabo...@gmail.com>
> wrote:
>
>
> Hi,
>
> From reading the code, I assume that the `pattern_db_new()` does some sort
> of allocation of a new pattern DB. I don't see any code releasing the
> pattern DB. Is that just missing from your post or something that some
> automatic mechanism does?
>
> >>Apologies, that part got missed in the code snippet. I am making the
> free call in the C code.
>
> node.c
> ---------
> int load_pattern_db(const gchar* file, key_value_cb cb)
> {
> if(patterndb != NULL){
> pattern_db_free(patterndb);
> }
> patterndb = pattern_db_new();
> pattern_db_reload_ruleset(patterndb, configuration, file);
> pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
> return 0;
> }
>
> You can try a similar C program that instantiates and frees the structure
> to check for similar behavior.
>
> >> To verify if the C code had some issue, I called the C wrapper code
> method '*load_pattern_db*' from my main.c to completely eliminate Go code
> here. What I found is there is no increase in memory consumption after
> every call ('load_pattern_db' was called 5 times). Hence there is no issue
> from C code.
>
> Thanks,
> Nitish
>
> On Mon, Mar 16, 2020 at 7:04 PM Gregor Best <b...@pferdewetten.de> wrote:
>
>> This might be a dumb question but...
>>
>> From reading the code, I assume that the `pattern_db_new()` does some
>> sort of allocation of a new pattern DB. I don't see any code releasing the
>> pattern DB. Is that just missing from your post or something that some
>> automatic mechanism does?
>>
>> If not, that might be your leak.
>> On 09.03.20 12:33, Nitish Saboo wrote:
>>
>> Hi
>>
>> Following are my Go code and C header file and C wrapper code
>>
>> parser.go
>> ==========
>> var f *os.File
>>
>> func LoadPatternDB(patterndb string) {
>> path := C.CString(patterndb)
>> defer C.free(unsafe.Pointer(path))
>> C.load_pattern_db(path,
>> (C.key_value_cb)(unsafe.Pointer(C.callOnMeGo_cgo)))
>> }
>>
>> //export ParsedData
>> func ParsedData(k *C.char, val *C.char, val_len C.size_t) {
>> f.WriteString(C.GoString(k))
>> f.WriteString("\n")
>> }
>>
>> cfunc.go
>> ========
>> /*
>> #include <stdio.h>
>> // The gateway function
>> void callOnMeGo_cgo(char *k, char *val, size_t val_len)
>> {
>> void ParsedData(const char *k, const char *val, size_t val_len);
>> ParsedData(k, val, val_len);
>> }
>> */
>> import "C"
>>
>> node.h
>> =======
>>
>> #ifndef TEST_H_INCLUDED
>> #define TEST_H_INCLUDED
>>
>> #include <stdlib.h>
>>
>> typedef void (*key_value_cb)(const char* k, const char* val, size_t
>> val_len);
>> int load_pattern_db(const char* file, key_value_cb cb);
>>
>> #endif
>>
>> node.c
>> ---------
>> int load_pattern_db(const gchar* file, key_value_cb cb)
>> {
>> patterndb = pattern_db_new();
>> pattern_db_reload_ruleset(patterndb, configuration, file);
>> pattern_db_set_emit_func(patterndb, pdbtool_pdb_emit_accumulate, cb);
>> return 0;
>> }
>>
>>
>> I am calling '*LoadPatternDB*' method in my parser.go file that makes a
>> cgo call '*C.load_pattern_db'* where I am passing a callback function to
>> the C code.
>> The C code is a wrapper code that internally calls some syslog-ng library
>> apis'
>>
>> What I observed is:
>>
>> 1)As soon as I call LoadPatternDB() method in parser.go there is some
>> increase in memory consumption(some memory leak).Ideally that should not
>> have happened.
>>
>> 2)To verify if the C code had some issue, I called the C wrapper code
>> method '*load_pattern_db*' from my main.c in the following manner to
>> completely eliminate Go code here.What I found is there is no increase in
>> memory consumption after every call ('load_pattern_db' was called 5
>> times).Hence there is no memory leak from C code.So the issue lies in the
>> Go code in '*LoadPatternDB*' method in parser.go
>>
>> main.c
>> =======
>>
>> void check(char *key, char *value, size_t value_len)
>> {
>> printf("I am in function check\n");
>> }
>>
>> int main(void){
>> char* filename = "/home/nitish/default.xml";
>> key_value_cb s = check;
>> int i;
>> for (i=1; i<=5; i++)
>> {
>> load_pattern_db(filename, s);
>> printf("Sleeping for 5 second.\n");
>> sleep(5);
>> }
>> printf("Loading done 5 times.\n");
>> return 0;
>> }
>>
>> 3)Can someone please guide me and help me figure out the mem-leak in
>> 'LoadPatternDB' method in parser.go at very first glance? Is the callback
>> function pointer an issue here ?
>>
>> 4)What tool can I use to check this mem-leak ?
>>
>> Thanks,
>> Nitish
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/CALjMrq5cAJ19CQ8OmMPSJmVB1P3t4hE0CKZ4HsEsH-mp6zm3Ng%40mail.gmail.com
>> <https://groups.google.com/d/msgid/golang-nuts/CALjMrq5cAJ19CQ8OmMPSJmVB1P3t4hE0CKZ4HsEsH-mp6zm3Ng%40mail.gmail.com?utm_medium=email&utm_source=footer>
>> .
>>
>> --
>> --
>> Gregor Best
>> b...@pferdewetten.de
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/fee07a13-e3d7-4a2e-6a6c-fdb11140ce21%40pferdewetten.de
>> <https://groups.google.com/d/msgid/golang-nuts/fee07a13-e3d7-4a2e-6a6c-fdb11140ce21%40pferdewetten.de?utm_medium=email&utm_source=footer>
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CALjMrq4-%3DcsuKRP4Xa-MVgrm5MJdWLNVe0v5_i3jswjesHQTqA%40mail.gmail.com
> <https://groups.google.com/d/msgid/golang-nuts/CALjMrq4-%3DcsuKRP4Xa-MVgrm5MJdWLNVe0v5_i3jswjesHQTqA%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CALjMrq6zRGBykH1s0EvzbsByxd%2Bfk4caowizkT-UNAMCZYVJUg%40mail.gmail.com.
