Hi Daniel, Thanks for the reply again. Looking at the email history, I'm not sure how we decided it was definitely a pkg memory problem. What we see is the output of "ps aux" as follows:
root@sip0-test:~# ps aux | egrep -i "kamailio|mem" USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 6794 0.0 0.0 22480 1868 ? Ss Oct02 0:12 /opt/testuser/current/sbin/testuser_safe_kamailio testuser 6822 0.0 0.2 556528 4580 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6824 0.3 8.7 825552 180244 ? S Oct02 56:40 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6825 0.3 8.7 825536 180776 ? S Oct02 56:20 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6826 0.3 8.7 825912 180296 ? S Oct02 55:54 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6827 0.3 8.7 825744 180580 ? S Oct02 56:19 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6828 0.3 8.7 825536 180092 ? S Oct02 56:25 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6829 0.3 8.7 825536 180632 ? S Oct02 56:21 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6830 0.3 8.7 825472 180968 ? S Oct02 56:37 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6831 0.3 8.7 825276 180272 ? S Oct02 56:41 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6832 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6833 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6834 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6835 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6836 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6837 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6838 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6839 0.0 0.0 556528 1324 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6840 0.0 0.0 556528 1776 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6841 0.0 0.0 556528 1780 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6842 0.0 0.0 556528 1780 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6843 0.0 0.0 556528 1328 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6844 0.0 0.0 556528 1780 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6845 0.0 0.0 556528 1328 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6846 0.0 0.0 556528 1328 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6847 0.0 0.0 556528 1328 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6848 0.0 0.0 556528 1676 ? S Oct02 0:02 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6849 0.0 0.1 556528 3568 ? S Oct02 5:28 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6850 0.0 0.0 556612 1600 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6851 0.0 0.0 556532 1188 ? S Oct02 0:00 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid testuser 6852 0.0 0.0 556528 1360 ? S Oct02 0:02 /sbin/kamailio -m 512 -P /var/run/testuser/kamailio.pid You'll see for example that process 6824 is using 8.7% of memory, which is much more than it was using 5 days ago. Yet if I run the same sercmd again I get (exactly!) the same numbers: root@sip0-test:~# sercmd pkg.stats pid 6824 { entry: 1 pid: 6824 rank: 1 used: 209836 free: 3704080 real_used: 490224 } Any ideas? On 12 October 2013 00:23, Daniel-Constantin Mierla <mico...@gmail.com>wrote: > Hi David, > > > On 10/10/13 11:36 PM, David Cunningham wrote: > >> Hi Daniel, >> >> Thanks for the reply. Perhaps what we're seeing is normal, and the memory >> use is meant to increase as time progresses. Would you expect to see an >> ongoing memory use increase, or when should it stop increasing? >> >> >> private memory (pkg) should stay rather constant. It increases when > there is a sip message processed, but once is sent out, it should come back > around the average. > > There are couple of functions that can fill the private memory and keep it > up, such as doing an sql_query() that returns a big data and the result is > not freed (sql_result_free()). It is not actually a leak as the next > sql_query() will free previous result, but in case you have such query for > some corner case that is not executed frequently, then the memory can stay > filled in. Another example will be storing very large value in a $var(...) > (e.g., $var(x) ). > > This is private memory, per process, which is meant for temporary > operations. Shared memory (shm) can increase over the time, being the place > where the dynamic data required at runtime is stored (e.g., location > records, hash tables, transactions) - so as you get more traffic or more > phones using kamailio, more shm is used. But your problem was reported for > pkg. > > Anyhow, keep an eye on the pkg.stats and if you see constant increase > which is substantial, then get a mem log dump. > > Cheers, > Daniel > > > -- > Daniel-Constantin Mierla - http://www.asipto.com > http://twitter.com/#!/miconda - > http://www.linkedin.com/in/**miconda<http://www.linkedin.com/in/miconda> > Kamailio Advanced Trainings - Berlin, Nov 25-28; Miami, Nov 18-20, 2013 > - more details about Kamailio trainings at http://www.asipto.com - > > -- David Cunningham, Voisonics http://voisonics.com/ USA: +1 213 221 1092 UK: +44 (0) 20 3298 1642 Australia: +61 (0) 2 8063 9019
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users