Re: [SR-Users] Memory leak in 3.3.4

2013-10-16 Thread David Cunningham
Hi Daniel, Configuration sent to you privately. Thanks. On 16 October 2013 20:28, Daniel-Constantin Mierla wrote: > Hello, > > indeed, it looks like a system memory leak, for what so ever reason the > discussion was misled to pkg. > > Can you send me the list of modules you load (loadmodule l

Re: [SR-Users] Memory leak in 3.3.4

2013-10-16 Thread Daniel-Constantin Mierla
Hello, indeed, it looks like a system memory leak, for what so ever reason the discussion was misled to pkg. Can you send me the list of modules you load (loadmodule lines) as well as the functions from your perl script that are related to kamailio internal functions or variables? You can se

Re: [SR-Users] Memory leak in 3.3.4

2013-10-14 Thread David Cunningham
Hi Daniel, Thanks for the reply again. Looking at the email history, I'm not sure how we decided it was definitely a pkg memory problem. What we see is the output of "ps aux" as follows: root@sip0-test:~# ps aux | egrep -i "kamailio|mem" USER PID %CPU %MEMVSZ RSS TTY STAT START

Re: [SR-Users] Memory leak in 3.3.4

2013-10-11 Thread Daniel-Constantin Mierla
Hi David, On 10/10/13 11:36 PM, David Cunningham wrote: Hi Daniel, Thanks for the reply. Perhaps what we're seeing is normal, and the memory use is meant to increase as time progresses. Would you expect to see an ongoing memory use increase, or when should it stop increasing? private memor

Re: [SR-Users] Memory leak in 3.3.4

2013-10-10 Thread David Cunningham
Hi Daniel, I was looking to give you the information you requested in your previous reply for PID 6824. Hopefully this is it: # sercmd pkg.stats pid 6824 { entry: 1 pid: 6824 rank: 1 used: 209836 free: 3704080 real_used: 490224 } On 10 October 2013 21:41, Daniel-Consta

Re: [SR-Users] Memory leak in 3.3.4

2013-10-10 Thread Daniel-Constantin Mierla
Hello, getting pkg stats for a process: sercmd pkg.stats pid 1234 replace 1234 with the real PID value. There are options to use rank or index, see: - http://kamailio.org/docs/modules/stable/modules/kex.html#idp213872 Is it what you were looking for or did I misunderstood? Cheers, Daniel

Re: [SR-Users] Memory leak in 3.3.4

2013-10-09 Thread David Cunningham
Hi Daniel, How do I get that information please? The information above includes what I got from the pkgstats module. Is it by restarting Kamailio? On 4 October 2013 16:42, Daniel-Constantin Mierla wrote: > Hello, > > none of the chunks there are indication of a leak. Can you give the output

Re: [SR-Users] Memory leak in 3.3.4

2013-10-03 Thread Daniel-Constantin Mierla
Hello, none of the chunks there are indication of a leak. Can you give the output of pkg.stats for pid 6824? Which of the processes has the highest value of used pkg memory? Cheers, Daniel On 10/4/13 4:25 AM, David Cunningham wrote: Hi Daniel, Here are the 'kamctl ps' and pkg status logs.

Re: [SR-Users] Memory leak in 3.3.4

2013-10-03 Thread David Cunningham
Hi Daniel, Here are the 'kamctl ps' and pkg status logs. During the 24 hours it ran for lots of REGISTER packets came through and memory use definitely increased. Thank you. Process:: ID=0 PID=6822 Type=attendant Process:: ID=1 PID=6824 Type=udp receiver child=0 sock=67.58.182.98:5060 Process:

Re: [SR-Users] Memory leak in 3.3.4

2013-10-02 Thread David Cunningham
Thanks Daniel. We will provide feedback when we have let it run for a while. On 2 October 2013 22:51, Daniel-Constantin Mierla wrote: > Hello, > > yes, you should run it for a while, with traffic, otherwise the leak > cannot be discovered. > > You can watch the processes with rank 1,2,3 and se

Re: [SR-Users] Memory leak in 3.3.4

2013-10-02 Thread Daniel-Constantin Mierla
Hello, yes, you should run it for a while, with traffic, otherwise the leak cannot be discovered. You can watch the processes with rank 1,2,3 and see when the free memory is decreasing. Then get the pkg status logs. Get also the list of running processes with 'kamctl ps' so I know at what pi

Re: [SR-Users] Memory leak in 3.3.4

2013-10-02 Thread Daniel-Constantin Mierla
Hello, ahh, right, forgot that it was disabled because it was not safe to use at runtime for worker processes. You have to use: kamcmd cfg.set_now_int core mem_dump_pkg then wait for a bit so that process is receiving a SIP message. Or, if you restart, the stats are printed in syslog for al

Re: [SR-Users] Memory leak in 3.3.4

2013-10-01 Thread David Cunningham
Hi Daniel, When I send SIGUSR1 to a worker process nothing is logged. Any ideas there? The following is the output of "sercmd pkg.stats". { entry: 0 pid: 14570 rank: 0 used: 191100 free: 3731504 real_used: 462800 } { entry: 1 pid: 14571 rank: 1 used: 202656

Re: [SR-Users] Memory leak in 3.3.4

2013-10-01 Thread Daniel-Constantin Mierla
Hello, send SIGUSR1 to one of the worker processes (e.g., UDP listener). The main process of kamailio doesn't process SIP messages, so it is not exposed to leaks. You can used 'kamcmd pkg.stats" to see statistics about usage of pkg per each process (older versions have sercmd). 'kamctl ps

Re: [SR-Users] Memory leak in 3.3.4

2013-10-01 Thread David Cunningham
Hi Daniel, What I copied and pasted is all that's listed when I do a kill -SIGUSR1 on the Kamailio master process. How do I get fuller logs? Thanks for your help. On 1 October 2013 21:57, Daniel-Constantin Mierla wrote: > Hello, > > you gave only the first part of memory log output. The chun

Re: [SR-Users] Memory leak in 3.3.4

2013-10-01 Thread Daniel-Constantin Mierla
Hello, you gave only the first part of memory log output. The chunks listed there are from startup, thus not part of a leak, because they are not allocated again at runtime. Can you give the full logs with messages from qm_status? The leaks are typically listed towards the end of those messa

Re: [SR-Users] Memory leak in 3.3.4

2013-07-31 Thread David Cunningham
Hi Daniel, Thank you, we will try that. On 31 July 2013 20:54, Daniel-Constantin Mierla wrote: > Hello, > > revising the patch I noticed I was moving the initialization of the > variable after pushing it to perl environment (from the perl docs, the > variable should have been initialized afte

Re: [SR-Users] Memory leak in 3.3.4

2013-07-30 Thread David Cunningham
Hi Daniel, We tried that patch, but Kamailio logged lots of errors like the following. The undefined value in question is $m, which should be the SIP message. Would you have any advice? Thanks. Jul 31 02:13:57 hostname /sbin/kamailio[21087]: ERROR: perl [openserxs.xs:1022]: perl error: Can't call

Re: [SR-Users] Memory leak in 3.3.4

2013-07-25 Thread David Cunningham
Hi Daniel, I'll suggest that to the customer. Thank you! On 25 July 2013 15:45, Daniel-Constantin Mierla wrote: > Hello, > > can you try the attached patch? It's the same patch, just for two > versions, one is for 3.3.x and the other for devel version > > It initializes the SIP message variab

Re: [SR-Users] Memory leak in 3.3.4

2013-07-24 Thread Daniel-Constantin Mierla
Hello, can you try the attached patch? It's the same patch, just for two versions, one is for 3.3.x and the other for devel version It initializes the SIP message variable that is passed to perl after creating the temporary environment, so it is actually destroyed by the perl embedded interp

Re: [SR-Users] Memory leak in 3.3.4

2013-07-24 Thread David Cunningham
Hi Daniel, The system is running Perl 5.8.8 on Red Hat Enterprise Linux Server release 5.4. If I remember right programs running under Valgrind can have issues, so I'm not sure if the customer will want to do that. Ideally we'd do it on a test system, but I'm not sure if we have any RHEL available

Re: [SR-Users] Memory leak in 3.3.4

2013-07-24 Thread Daniel-Constantin Mierla
Hello, I would say that perl_exec() is the one with the highest chances to be the reason for the leak. Next is line would be db_mysql module, if liked with some custom mysql client library, although even in this case will be unlikely. Back to perl, the module itself does not call any malloc,

Re: [SR-Users] Memory leak in 3.3.4

2013-07-24 Thread David Cunningham
Hello, We don't do any kamctl commands at all. We do have various modules loaded, as follows. The primary functions we use Kamailio for are phone registrations through usrloc, and routing calls to Asterisk through logic contained in Perl via perl_exec(). Thanks for all your advice so far! loadmod

Re: [SR-Users] Memory leak in 3.3.4

2013-07-23 Thread Daniel-Constantin Mierla
Hello, On 7/24/13 4:24 AM, David Cunningham wrote: Hello, Thank you very much for the email. In reply: 1. The system ran out of memory. Linux's oom-killer killed Kamailio. then all the instructions I gave are useless, they are for debugging kamailio's internal memory manager, which handles pk

Re: [SR-Users] Memory leak in 3.3.4

2013-07-23 Thread David Cunningham
Hello, Thank you very much for the email. In reply: 1. The system ran out of memory. Linux's oom-killer killed Kamailio. 2. You're right, DEBUG_MEMORY is a local configuration setting. If defined it sets memdbg to -2, and memlog to -2. The debug setting is -1. 3. We'll try setting mem_summary=1

Re: [SR-Users] Memory leak in 3.3.4

2013-07-22 Thread Daniel-Constantin Mierla
Hello, first, to clarify, is the system memory or kamailio's pkg/shm memory running out? If the operating system runs out of memory, then should be a leak in a library, because kamailio modules uses only from a pre-allocated chunk, not going over it. On 7/23/13 7:33 AM, David Cunningham wrot

[SR-Users] Memory leak in 3.3.4

2013-07-22 Thread David Cunningham
Hello, We're running a Kamailio 3.3.4 system, and Kamailio is slowly using more and more memory. Over a couple of weeks it will run out of system memory. We tried to enable memory debugging doing the following, but it resulted in Kamailio not responding to any SIP packets. Would anyone have advic