Thanks for response Daniel, Can you point me to the location where the core dump should be generated? Afaik kamailio was compiled in /usr/local, what is the workdir for that setup? I can't figure it out from systemd config file. Also, is it really necessary to run as root? What if I set enough permissions on workdir? > 16 нояб. 2016 г., в 11:32, Daniel-Constantin Mierla <mico...@gmail.com> > написал(а): > > Hello, > > the plan is to freeze development of v5.0 before the Chirstmas of the > first week of January, then it will be a 1-1.5 months of testing, > followed by the release. So expect like 2-3 months till the stable > release of 5.0. > > As for the issue, it can be a buffer overflow somewhere, not related to > mysql module code, but triggered by use of it. > > It is important to get the core file for such case, before starting > kamailio do 'ulimit -c unlimited'. You may need to run kamailio as root > to be able to write the core file. Enable also the option for one core > file per process, typically is: > > echo "1" > /proc/sys/kernel/core_uses_pid > > Once you get core files, extract the output of 'bt full' with gdb and > send it over here. > > Cheers, > Daniel > > > On 15/11/16 22:35, Alexandru Covalschi wrote: >> Hello list, >> >> We’re using dev version of Kamailio: >> version: kamailio 5.0.0-dev4 (x86_64/linux) ff63e5 >> flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, >> DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, SHM_MMAP, PKG_MALLOC, >> Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, >> FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, >> USE_DST_BLACKLIST, HAVE_RESOLV_RES >> ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, >> MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB >> poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. >> id: ff63e5 >> compiled on 15:46:49 May 31 2016 with gcc 4.9.2 >> >> Sometimes we encounter such issue: >> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: <core> >> [db_row.c:114]: db_allocate_row(): no private memory left >> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql >> [km_row.c:57]: db_mysql_convert_row(): could not allocate row >> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql >> [km_res.c:188]: db_mysql_convert_rows(): error while converting row #16 >> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql >> [km_res.c:217]: db_mysql_convert_result(): error while converting rows >> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql >> [km_dbase.c:261]: db_mysql_store_result(): error while converting result >> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: <core> >> [db_query.c:139]: db_do_query_internal(): error while storing result >> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: permissions >> [trusted.c:91]: reload_trusted_table(): failed to query database >> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: <core> >> [db_row.c:114]: db_allocate_row(): no private memory left >> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql >> [km_row.c:57]: db_mysql_convert_row(): could not allocate row >> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: db_mysql >> [km_dbase.c:444]: db_mysql_fetch_result(): error while converting row #15 >> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: ERROR: htable >> [ht_db.c:234]: ht_db_load_table(): Error while fetching result >> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12276]: : <core> >> [mem/q_malloc.c:468]: qm_free(): BUG: qm_free: freeing already freed pointer >> (0x7f5ebda8ae18), called from db_mysql: km_dbase.c: >> db_mysql_free_result(305), first free db_mysql: km_dbase.c: >> db_mysql_free_result(305) - aborting >> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12281]: CRITICAL: <core> >> [pass_fd.c:275]: receive_fd(): EOF on 16 >> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12268]: ALERT: <core> >> [main.c:739]: handle_sigs(): child process 12276 exited by a signal 6 >> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12268]: ALERT: <core> >> [main.c:742]: handle_sigs(): core was not generated >> Nov 9 23:30:01 sbc01 /usr/local/sbin/kamailio[12268]: INFO: <core> >> [main.c:754]: handle_sigs(): terminating due to SIGCHLD >> >> The thing is we heavily use mysql module, but only to update the in-memory >> tables by kamcmd. Each N minutes a special script updates the >> trusted,address and htable executing kamcmd. Kamailio (and kamcmd as well) >> talks only with localhost mysql server. >> What I saw when encountered that issue on a live machine is that issue >> happens only with one of child processes, any other are ok. >> Interesting thing is that happens at the same time with machines on the same >> «set», I mean that issue happened simultaneously with two our test machines >> which actually didn’t have any load on them. >> The common thing between those machines is that they are in same subnet and >> local mysql databases are filled by scripts which query same external db. >> I can’t confirm if there were or there weren’t any networking issues at that >> time with those machines, but as soon as kamcmd queries localhost that >> shouldn’t be the source of the issue. >> >> So my questions are: >> 1. Has anyone encountered such thing? >> 2. Maybe the issue is already localized so it has sense to update? We >> actually use that on production (pls don’t throw too much rocks at me), so >> maintenance should be properly planned and I must be sure update won’t break >> anything. >> 3. If update is proposed - how to do it? I mean - follow the guide >> https://www.kamailio.org/wiki/install/devel/git or there are some other >> tips? I suppose in ideal world I don’t even stop the binary, only restart >> after make all && make install are done, as everything is in-memory. Am I >> correct? >> 4. When can we expect stable 5.0 version? (at least tell if it’s >> months/years) >> >> Thanks in advance! >> _______________________________________________ >> SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list >> sr-users@lists.sip-router.org >> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users > > -- > Daniel-Constantin Mierla > http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda > Kamailio Advanced Training, Berlin, Nov 28-30, 2016 - http://www.asipto.com > > > _______________________________________________ > SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list > sr-users@lists.sip-router.org > http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
