Hello, Thank you very much for the email. In reply:
1. The system ran out of memory. Linux's oom-killer killed Kamailio. 2. You're right, DEBUG_MEMORY is a local configuration setting. If defined it sets memdbg to -2, and memlog to -2. The debug setting is -1. 3. We'll try setting mem_summary=12, thanks. 4. We'll try setting asynchronous syslog, thanks. 5. Our configuration totals 338 lines, or approx 8.5kb. Is that a lot? 6. We'll try setting mem_join=1, thanks. On 23 July 2013 16:53, Daniel-Constantin Mierla <mico...@gmail.com> wrote: > Hello, > > first, to clarify, is the system memory or kamailio's pkg/shm memory > running out? If the operating system runs out of memory, then should be a > leak in a library, because kamailio modules uses only from a pre-allocated > chunk, not going over it. > > > On 7/23/13 7:33 AM, David Cunningham wrote: > >> Hello, >> >> We're running a Kamailio 3.3.4 system, and Kamailio is slowly using more >> and more memory. Over a couple of weeks it will run out of system memory. >> >> We tried to enable memory debugging doing the following, but it resulted >> in Kamailio not responding to any SIP packets. Would anyone have advice >> please on how to debug the situation? >> >> 1. In Makefile.defs set MEMDBG to 1 and recompile Kamailio. >> 2. In kamailio.cfg add the line: >> #!define DEBUG_MEMORY 1 >> > do you set something special in config when DEBUG_MEMORY is 1? It is not > by default there, so I assume you added some rules based on this > pre-processor directive. > > For memory troubleshooting, set memlog to a value lower than debug > parameter in config file and try with mem_summary=12 for a more compact > output. See more about these parameters in the wiki: > > - > http://www.kamailio.org/wiki/**cookbooks/3.3.x/core#memlog<http://www.kamailio.org/wiki/cookbooks/3.3.x/core#memlog> > > Run kamailio for a while in normal conditions, then restart it to get the > memory usage summaries. There should be indication if there is some leak, > by seeing memory chunks allocated many times from a function used at > runtime. You can send the memory summary for a process here, we can look at > it. > > > >> While this was running and Kamailio didn't respond to packets, it logged >> lots of lines like this: >> > > Do you have syslog to be configured in asynchronous mode? See the notes > from: > > - > http://www.kamailio.org/wiki/**tutorials/3.2.x/syslog<http://www.kamailio.org/wiki/tutorials/3.2.x/syslog> > > The memdbg is less than debug value, that means printing few log messages > for each memory operation. You can make memdbg higher and rely on memlog > for memory summaries, otherwise will be lot of log messages related to > memory. > > >> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]: >> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438) >> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]: >> qm_malloc(0x4000e008, 128) returns address 0x40048918 frag. 0x40048900 >> (size=128) on 1 -th hit >> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:369]: >> qm_malloc(0x4000e008, 128) called from <core>: cfg.lex: addstr(1438) >> Jul 22 21:32:22 hostname kamailio: : <core> [mem/q_malloc.c:413]: >> qm_malloc(0x4000e008, 128) returns address 0x400489c8 frag. 0x400489b0 >> (size=128) on 1 -th hit >> > addstr() is a function used only for parsing configuration file, as long > as you can still see them, the configuration file parsing was not finish. > addstr() is not a source of leaks because it is not used at runtime. > > If you have large config file, then you can get close to the limits of the > private memory, which is set to 4MB. You can increase its value using -M > parameter (e.g., start kamailio with -M 8 to set it to use 8MB of memory). > > Over the time, the private memory can get used due to fragmentation, you > can set the mem_join parameter in config file to avoid it (works when > compiled with MEMDBG=1). > > To monitor usage of internal pkg memory, then you can use sercmd with > pkg.stats command: > > http://kamailio.org/docs/**modules/3.3.x/modules_k/kex.**html#idp16972640<http://kamailio.org/docs/modules/3.3.x/modules_k/kex.html#idp16972640> > > Shared memory stats are printed by 'kamctl fifo get_statistics shmem:' > > When you see significant increase of the memory usage, then you can > restart to get the summaries. > > You should run these commands after start, just to see the initial usage > of memory. > > Cheers, > Daniel > > -- > Daniel-Constantin Mierla - http://www.asipto.com > http://twitter.com/#!/miconda - > http://www.linkedin.com/in/**miconda<http://www.linkedin.com/in/miconda> > > > ______________________________**_________________ > SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list > sr-users@lists.sip-router.org > http://lists.sip-router.org/**cgi-bin/mailman/listinfo/sr-**users<http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users> > -- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
