Hi Daniel, The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server release 5.4. If I remember right programs running under Valgrind can have issues, so I'm not sure if the customer will want to do that. Ideally we'd do it on a test system, but I'm not sure if we have any RHEL available. I'll see what we can do. Thanks again.
On 25 July 2013 04:55, Daniel-Constantin Mierla <mico...@gmail.com> wrote: > Hello, > > I would say that perl_exec() is the one with the highest chances to be the > reason for the leak. Next is line would be db_mysql module, if liked with > some custom mysql client library, although even in this case will be > unlikely. > > Back to perl, the module itself does not call any malloc, so it might be > the embedding Perl API that is not used properly in the module. > > Can you use some testbed, set children=1 and run kamailio under valgrind, > then do some calls and see if it detects the source of the leak? > > I'm not using the perl module, I will try to check it whenever I get a > chance in the next days. What version of perl do you have installed? > > Cheers, > Daniel > > > On 7/24/13 10:31 AM, David Cunningham wrote: > > Hello, > > We don't do any kamctl commands at all. We do have various modules loaded, > as follows. The primary functions we use Kamailio for are phone > registrations through usrloc, and routing calls to Asterisk through logic > contained in Perl via perl_exec(). > Thanks for all your advice so far! > > loadmodule "tm.so" > loadmodule "tmx.so" > loadmodule "usrloc.so" > loadmodule "auth.so" > loadmodule "auth_db.so" > loadmodule "ctl.so" > loadmodule "db_mysql.so" > loadmodule "kex.so" > loadmodule "maxfwd.so" > loadmodule "mi_fifo.so" > loadmodule "mi_rpc.so" > loadmodule "nathelper.so" > loadmodule "perl.so" > loadmodule "pv.so" > loadmodule "registrar.so" > loadmodule "rr.so" > loadmodule "sanity.so" > loadmodule "siputils.so" > loadmodule "sl.so" > loadmodule "textops.so" > loadmodule "xlog.so" > > > On 24 July 2013 16:33, Daniel-Constantin Mierla <mico...@gmail.com> wrote: > >> Hello, >> >> >> On 7/24/13 4:24 AM, David Cunningham wrote: >> >> Hello, >> >> Thank you very much for the email. In reply: >> >> 1. The system ran out of memory. Linux's oom-killer killed Kamailio. >> >> then all the instructions I gave are useless, they are for debugging >> kamailio's internal memory manager, which handles pkg and shm mallocs. >> >> The chances to be from kamailio itself are very low now. Do you do lot of >> mi commands (e.g., kamctl ...)? The mi api uses system malloc, but the rest >> of code should use internal memory manager which does not go beyond the >> limits set with -m and -M, thus not causing an OS memory exhaustion. >> >> Can you list what modules are you loading? At some point it was a leak in >> libssl, in case you use tls a lot. But could be another external library... >> >> Cheers, >> Daniel >> >> >> >> 2. You're right, DEBUG_MEMORY is a local configuration setting. If >> defined it sets memdbg to -2, and memlog to -2. The debug setting is -1. >> >> 3. We'll try setting mem_summary=12, thanks. >> >> 4. We'll try setting asynchronous syslog, thanks. >> >> 5. Our configuration totals 338 lines, or approx 8.5kb. Is that a lot? >> >> 6. We'll try setting mem_join=1, thanks. >> >> >> >> On 23 July 2013 16:53, Daniel-Constantin Mierla <mico...@gmail.com>wrote: >> >>> Hello, >>> >>> first, to clarify, is the system memory or kamailio's pkg/shm memory >>> running out? If the operating system runs out of memory, then should be a >>> leak in a library, because kamailio modules uses only from a pre-allocated >>> chunk, not going over it. >>> >>> >>> On 7/23/13 7:33 AM, David Cunningham wrote: >>> >>>> Hello, >>>> >>>> We're running a Kamailio 3.3.4 system, and Kamailio is slowly using >>>> more and more memory. Over a couple of weeks it will run out of system >>>> memory. >>>> >>>> We tried to enable memory debugging doing the following, but it >>>> resulted in Kamailio not responding to any SIP packets. Would anyone have >>>> advice please on how to debug the situation? >>>> >>>> 1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio. >>>> 2. In kamailio.cfg add the line: >>>> #!define DEBUG_MEMORY 1 >>>> >>> do you set something special in config when DEBUG_MEMORY is 1? It is >>> not by default there, so I assume you added some rules based on this >>> pre-processor directive. >>> >>> For memory troubleshooting, set memlog to a value lower than debug >>> parameter in config file and try with mem_summary=12 for a more compact >>> output. See more about these parameters in the wiki: >>> >>> - http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog >>> >>> Run kamailio for a while in normal conditions, then restart it to get >>> the memory usage summaries. There should be indication if there is some >>> leak, by seeing memory chunks allocated many times from a function used at >>> runtime. You can send the memory summary for a process here, we can look at >>> it. >>> >>> >>> >>>> While this was running and Kamailio didn't respond to packets, it >>>> logged lots of lines like this: >>>> >>> >>> Do you have syslog to be configured in asynchronous mode? See the notes >>> from: >>> >>> - http://www.kamailio.org/wiki/tutorials/3.2.x/syslog >>> >>> The memdbg is less than debug value, that means printing few log >>> messages for each memory operation. You can make memdbg higher and rely on >>> memlog for memory summaries, otherwise will be lot of log messages related >>> to memory. >>> >>> >>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]: >>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438) >>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]: >>>> qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900 >>>> (size=128) on 1 -th hit >>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]: >>>> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438) >>>> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]: >>>> qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0 >>>> (size=128) on 1 -th hit >>>> >>> addstr() is a function used only for parsing configuration file, as >>> long as you can still see them, the configuration file parsing was not >>> finish. addstr() is not a source of leaks because it is not used at runtime. >>> >>> If you have large config file, then you can get close to the limits of >>> the private memory, which is set to 4MB. You can increase its value using >>> -M parameter (e.g., start kamailio with -M 8 to set it to use 8MB of >>> memory). >>> >>> Over the time, the private memory can get used due to fragmentation, you >>> can set the mem_join parameter in config file to avoid it (works when >>> compiled with MEMDBG=1). >>> >>> To monitor usage of internal pkg memory, then you can use sercmd with >>> pkg.stats command: >>> >>> http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640 >>> >>> Shared memory stats are printed by 'kamctl fifo get_statistics shmem:' >>> >>> When you see significant increase of the memory usage, then you can >>> restart to get the summaries. >>> >>> You should run these commands after start, just to see the initial usage >>> of memory. >>> >>> Cheers, >>> Daniel >>> >>> -- >>> Daniel-Constantin Mierla - http://www.asipto.com >>> http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda >>> >>> >>> _______________________________________________ >>> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list >>> sr-users@lists.sip-router.org >>> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users >>> >> >> >> >> -- >> David Cunningham, Voisonics >> http://voisonics.com/ >> USA: +1 213 221 1092 >> UK: +44 (0) 20 3298 1642 <%2B44%20%280%29%2020%203298%201642> >> Australia: +61 (0) 2 8063 9019 <%2B61%20%280%29%202%208063%209019> >> >> >> -- >> Daniel-Constantin Mierla - >> http://www.asipto.comhttp://twitter.com/#!/miconda - >> http://www.linkedin.com/in/miconda >> >> > > > -- > David Cunningham, Voisonics > http://voisonics.com/ > USA: +1 213 221 1092 > UK: +44 (0) 20 3298 1642 > Australia: +61 (0) 2 8063 9019 > > > -- > Daniel-Constantin Mierla - http://www.asipto.comhttp://twitter.com/#!/miconda > - http://www.linkedin.com/in/miconda > > -- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
