Hello, are you using async transactions (e.g., t_suspend() or tsilo module)?
Cheers, Daniel On 15/01/16 10:51, Igor Potjevlesch wrote: > > I also seen this in /var/log/messages during the crash: > > > > Jan 15 10:37:41 tanus /usr/local/sbin/kamailio[24021]: : <core> > [mem/q_malloc.c:149]: qm_debug_frag(): BUG: qm_*: fragm. > 0x7f9abd767300 (address 0x7f9abd767330) end > overwritten(d33762d69737465, 746e65746e6f430a)! > > Jan 15 10:37:41 tanus kernel: kamailio[24021] general protection > ip:62245e sp:7fff39bc7900 error:0 in kamailio[400000+3c8000] > > Jan 15 10:37:42 tanus abrtd: Directory > 'ccpp-2016-01-15-10:37:41-24021' creation detected > > Jan 15 10:37:42 tanus abrt[23992]: Saved core dump of pid 24021 > (/usr/local/sbin/kamailio) to > /var/spool/abrt/ccpp-2016-01-15-10:37:41-24021 (339316736 bytes) > > Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[23990]: ALERT: <core> > [main.c:784]: handle_sigs(): child process 24021 exited by a signal 11 > > Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[23990]: ALERT: <core> > [main.c:787]: handle_sigs(): core was generated > > Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[23990]: INFO: <core> > [main.c:799]: handle_sigs(): terminating due to SIGCHLD > > Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24019]: INFO: <core> > [main.c:850]: sig_usr(): signal 15 received > > Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24013]: INFO: <core> > [main.c:850]: sig_usr(): signal 15 received > > Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24017]: INFO: <core> > [main.c:850]: sig_usr(): signal 15 received > > Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24026]: INFO: <core> > [main.c:850]: sig_usr(): signal 15 received > > Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24023]: INFO: <core> > [main.c:850]: sig_usr(): signal 15 received > > Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24030]: INFO: <core> > [main.c:850]: sig_usr(): signal 15 received > > Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24005]: INFO: <core> > [main.c:850]: sig_usr(): signal 15 received > > Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24009]: INFO: <core> > [main.c:850]: sig_usr(): signal 15 received > > Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24011]: INFO: <core> > [main.c:850]: sig_usr(): signal 15 received > > Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24007]: INFO: <core> > [main.c:850]: sig_usr(): signal 15 received > > Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24015]: INFO: <core> > [main.c:850]: sig_usr(): signal 15 received > > Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[24003]: INFO: <core> > [main.c:850]: sig_usr(): signal 15 received > > Jan 15 10:37:42 tanus /usr/local/sbin/kamailio[23990]: : <core> > [mem/q_malloc.c:159]: qm_debug_frag(): BUG: qm_*: prev. fragm. tail > overwritten(732d6369676f6c61, > a0d2d2d312d77)[0x7f9abd767970:0x7f9abd7679a0]! > > Jan 15 10:37:42 tanus kernel: kamailio[23990] general protection > ip:62245e sp:7fff39bc7690 error:0 in kamailio[400000+3c8000] > > Jan 15 10:37:42 tanus abrt[23994]: Not saving repeating crash in > '/usr/local/sbin/kamailio' > > Jan 15 10:37:42 tanus abrtd: Executable '/usr/local/sbin/kamailio' > doesn't belong to any package and ProcessUnpackaged is set to 'no' > > Jan 15 10:37:42 tanus abrtd: 'post-create' on > '/var/spool/abrt/ccpp-2016-01-15-10:37:41-24021' exited with 1 > > Jan 15 10:37:42 tanus abrtd: Deleting problem directory > '/var/spool/abrt/ccpp-2016-01-15-10:37:41-24021' > > Jan 15 10:37:43 tanus abrt[23994]: Saved core dump of pid 23990 to > core.23990 (339316736 bytes) > > Jan 15 10:37:47 tanus kamailio: INFO: <core> [sctp_core.c:70]: > sctp_core_check_support(): SCTP API not enabled - if you want to use > it, load sctp module > > Jan 15 10:37:47 tanus kamailio: WARNING: <core> [daemonize.c:360]: > daemonize(): pid file contains old pid, replacing pid > > Jan 15 10:37:47 tanus /usr/local/sbin/kamailio[24042]: INFO: rr > [../outbound/api.h:54]: ob_load_api(): unable to import bind_ob - > maybe module is not loaded > > Jan 15 10:37:47 tanus /usr/local/sbin/kamailio[24042]: INFO: rr > [rr_mod.c:160]: mod_init(): outbound module not available > > Jan 15 10:37:47 tanus /usr/local/sbin/kamailio[24042]: INFO: usrloc > [hslot.c:53]: ul_init_locks(): locks array size 1024 > > > > Regards, > > > > Igor. > > > > *De :*Igor Potjevlesch [mailto:igor.potjevle...@gmail.com] > *Envoyé :* vendredi 15 janvier 2016 10:47 > *À :* mico...@gmail.com; 'Kamailio (SER) - Users Mailing List' > <sr-users@lists.sip-router.org> > *Objet :* RE: [SR-Users] Kamailio 4.2.6 crash > > > > Hello Daniel, > > > > I move to 4.2.7. This morning a new crash occurred. I got two coredump: > > > > Core was generated by `/usr/local/sbin/kamailio -P > /var/run/kamailio.pid -m 256 -M 64'. > > Program terminated with signal 11, Segmentation fault. > > #0 0x000000000062245e in qm_status (qm=0x7f9abd447000) at > mem/q_malloc.c:788 > > 788 f!=&(qm->free_hash[h].head); > f=f->u.nxt_free, i++, j++){ > > > > (gdb) bt full > > #0 0x000000000062245e in qm_status (qm=0x7f9abd447000) at > mem/q_malloc.c:788 > > f = 0x30a012010010a0d > > i = 57 > > j = 4 > > h = 4 > > unused = 0 > > memlog = 5 > > mem_summary = 3 > > __FUNCTION__ = "qm_status" > > #1 0x000000000061a795 in qm_debug_frag (qm=0x7f9abd447000, > f=0x7f9abd767970) at mem/q_malloc.c:160 > > __FUNCTION__ = "qm_debug_frag" > > #2 0x000000000061ca58 in qm_free (qm=0x7f9abd447000, > p=0x7f9abd7679a0, file=0x7f9ace7e2662 "dialog: dlg_hash.c", > func=0x7f9ace7e50e2 "destroy_dlg", line=380) at mem/q_malloc.c:468 > > f = 0x7f9abd767970 > > size = 176 > > next = 0x400 > > prev = 0x7fff39bc7910 > > __FUNCTION__ = "qm_free" > > #3 0x00007f9ace7a64ef in destroy_dlg (dlg=0x7f9abd7660b8) at > dlg_hash.c:380 > > ret = 0 > > var = 0x7f9ad54374e8 > > __FUNCTION__ = "destroy_dlg" > > #4 0x00007f9ace7a67df in destroy_dlg_table () at dlg_hash.c:419 > > dlg = 0x0 > > l_dlg = 0x7f9abd7660b8 > > i = 665 > > __FUNCTION__ = "destroy_dlg_table" > > #5 0x00007f9ace771263 in mod_destroy () at dialog.c:783 > > No locals. > > #6 0x00000000005929ee in destroy_modules () at sr_module.c:811 > > t = 0x7f9ad52b8d00 > > foo = 0x7f9ad52b8a30 > > __FUNCTION__ = "destroy_modules" > > #7 0x000000000049c917 in cleanup (show_status=1) at main.c:569 > > memlog = 0 > > __FUNCTION__ = "cleanup" > > #8 0x000000000049dee4 in shutdown_children (sig=15, show_status=1) at > main.c:711 > > __FUNCTION__ = "shutdown_children" > > #9 0x00000000004a04ba in handle_sigs () at main.c:802 > > chld = 0 > > chld_status = 139 > > memlog = -1119369840 > > __FUNCTION__ = "handle_sigs" > > #10 0x00000000004a82eb in main_loop () at main.c:1757 > > i = 8 > > pid = 24021 > > si = 0x0 > > si_desc = "udp receiver child=7 > sock=91.213.145.60:5060\000\177\000\000`~\274\071\377\177\000\000\033{N\000\000\000\000\000\260~\274\071\377\177\000\000\004\000\000\000\000\000\000\000`TA\000\000\000\000\000(\305G\275\232\177", > '\000' <repeats 14 times>, > "\001\000\000\000\260~\274\071\377\177\000\000\276{N\000\000\000\000" > > nrprocs = 8 > > __FUNCTION__ = "main_loop" > > #11 0x00000000004acfab in main (argc=7, argv=0x7fff39bc8128) at > main.c:2581 > > cfg_stream = 0x2392010 > > c = -1 > > r = 0 > > tmp = 0x7fff39bc8f70 "" > > tmp_len = 32767 > > port = 968654846 > > proto = 0 > > options = 0x7033b8 > ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:" > > ret = -1 > > seed = 1451157380 > > rfd = 4 > > debug_save = 0 > > debug_flag = 0 > > dont_fork_cnt = 0 > > n_lst = 0x40d134 > > p = 0xc2 <Address 0xc2 out of bounds> > > __FUNCTION__ = "main" > > > > Second one: > > > > Core was generated by `/usr/local/sbin/kamailio -P > /var/run/kamailio.pid -m 256 -M 64'. > > Program terminated with signal 11, Segmentation fault. > > #0 0x000000000062245e in qm_status (qm=0x7f9abd447000) at > mem/q_malloc.c:788 > > 788 f!=&(qm->free_hash[h].head); > f=f->u.nxt_free, i++, j++){ > > > > (gdb) bt full > > #0 0x000000000062245e in qm_status (qm=0x7f9abd447000) at > mem/q_malloc.c:788 > > f = 0x30a012010010a0d > > i = 16 > > j = 4 > > h = 4 > > unused = 0 > > memlog = 5 > > mem_summary = 3 > > __FUNCTION__ = "qm_status" > > #1 0x000000000061a420 in qm_debug_frag (qm=0x7f9abd447000, > f=0x7f9abd767300) at mem/q_malloc.c:150 > > __FUNCTION__ = "qm_debug_frag" > > #2 0x000000000061ca58 in qm_free (qm=0x7f9abd447000, > p=0x7f9abd767330, file=0x7f9ad3d2f34d "tm: h_table.c", > func=0x7f9ad3d2f628 "free_cell", line=186) at mem/q_malloc.c:468 > > f = 0x7f9abd767300 > > size = 40 > > next = 0x400 > > prev = 0x7fff39bc7b80 > > __FUNCTION__ = "qm_free" > > #3 0x00007f9ad3c70c9d in free_cell (dead_cell=0x7f9abd79b5c0) at > h_table.c:186 > > b = 0x7f9abd767330 "INVITE sip:00447798156873@goren > SIP/2.0\r\nRecord-Route: <sip:A.B.C.D;lr;did=4b7.60a>\r\nCSeq: 1 > INVITE\r\nCall-ID: 729d-7e9-015201693735-DSQ-1-A.B.C_leg2\r\nFrom: > <sip:0123456789@D.C.B.A>;"... > > i = 0 > > rpl = 0x0 > > tt = 0x7f9abd5d8bf8 > > foo = 0x7fff39bc7c50 > > cbs = 0x0 > > cbs_tmp = 0x7f9abd7600a0 > > __FUNCTION__ = "free_cell" > > #4 0x00007f9ad3cb5a1c in wait_handler (ti=1300688687, > wait_tl=0x7f9abd79b640, data=0x7f9abd79b5c0) at timer.c:675 > > p_cell = 0x7f9abd79b5c0 > > ret = 1 > > #5 0x00000000005fd30f in timer_list_expire (t=1300688687, > h=0x7f9abd4c0908, slow_l=0x7f9abd4c36d8, slow_mark=17084) at timer.c:888 > > tl = 0x7f9abd79b640 > > ret = 1300688687 > > #6 0x00000000005fd757 in timer_handler () at timer.c:953 > > saved_ticks = 1300688687 > > run_slow_timer = 0 > > i = 700 > > __FUNCTION__ = "timer_handler" > > #7 0x00000000005fdbc5 in timer_main () at timer.c:992 > > No locals. > > #8 0x00000000004a77e6 in main_loop () at main.c:1700 > > i = 8 > > pid = 0 > > si = 0x0 > > si_desc = "udp receiver child=7 > sock=91.213.145.60:5060\000\177\000\000`~\274\071\377\177\000\000\033{N\000\000\000\000\000\260~\274\071\377\177\000\000\004\000\000\000\000\000\000\000`TA\000\000\000\000\000(\305G\275\232\177", > '\000' <repeats 14 times>, > "\001\000\000\000\260~\274\071\377\177\000\000\276{N\000\000\000\000" > > nrprocs = 8 > > __FUNCTION__ = "main_loop" > > #9 0x00000000004acfab in main (argc=7, argv=0x7fff39bc8128) at > main.c:2581 > > cfg_stream = 0x2392010 > > c = -1 > > r = 0 > > tmp = 0x7fff39bc8f70 "" > > tmp_len = 32767 > > port = 968654846 > > proto = 0 > > options = 0x7033b8 > ":f:cm:M:dVIhEeb:l:L:n:vKrRDTN:W:w:t:u:g:P:G:SQ:O:a:A:" > > ret = -1 > > seed = 1451157380 > > rfd = 4 > > debug_save = 0 > > debug_flag = 0 > > dont_fork_cnt = 0 > > n_lst = 0x40d134 > > p = 0xc2 <Address 0xc2 out of bounds> > > __FUNCTION__ = "main" > > > > Regards, > > > > Igor. > > > > *De :*Igor Potjevlesch [mailto:igor.potjevle...@gmail.com] > *Envoyé :* lundi 4 janvier 2016 10:55 > *À :* mico...@gmail.com <mailto:mico...@gmail.com>; 'Kamailio (SER) - > Users Mailing List' <sr-users@lists.sip-router.org > <mailto:sr-users@lists.sip-router.org>> > *Objet :* RE: [SR-Users] Kamailio 4.2.6 crash > > > > Hi Daniel, > > > > I wish you an happy new year! I will schedule this update in the next > few days. > > > > Thank you for your support. > > > > Regards, > > > > Igor. > > > > *De :*Daniel-Constantin Mierla [mailto:mico...@gmail.com] > *Envoyé :* vendredi 18 décembre 2015 11:28 > *À :* Igor Potjevlesch <igor.potjevle...@gmail.com > <mailto:igor.potjevle...@gmail.com>>; 'Kamailio (SER) - Users Mailing > List' <sr-users@lists.sip-router.org > <mailto:sr-users@lists.sip-router.org>> > *Objet :* Re: [SR-Users] Kamailio 4.2.6 crash > > > > Hello, > > can you upgrade to 4.2.7, it has some fixes that may be related to > this issue -- there are no changes to config/database that need to be > done. > > Cheers, > Daniel > > On 16/12/15 15:06, Igor Potjevlesch wrote: > > Hello, > > > > Here is the output: > > > > kamailio -v > > version: kamailio 4.2.6 (x86_64/linux) db77ac > > flags: STATS: Off, USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, > USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MEM, > SHM_MMAP, PKG_MALLOC, F_MALLOC, DBG_F_MALLOC, USE_FUTEX, > FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, > USE_NAPTR, USE_DST_BLACKLIST, HAVE_RESOLV_RES > > ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN > 16, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 64MB > > poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. > > id: db77ac > > compiled on 12:19:48 Oct 2 2015 with gcc 4.4.7 > > > > Regards, > > > > Igor. > > > > *De :*Daniel-Constantin Mierla [mailto:mico...@gmail.com] > *Envoyé :* lundi 14 décembre 2015 16:16 > *À :* Igor Potjevlesch <igor.potjevle...@gmail.com> > <mailto:igor.potjevle...@gmail.com>; 'Kamailio (SER) - Users > Mailing List' <sr-users@lists.sip-router.org> > <mailto:sr-users@lists.sip-router.org> > *Objet :* Re: [SR-Users] Kamailio 4.2.6 crash > > > > Hello, > > yes, it is the output I wanted. > > What is the exact version are you running? It is the output of > 'kamailio -v'. > > Cheers, > Daniel > > On 14/12/15 15:53, Igor Potjevlesch wrote: > > Hello Daniel, > > > > Is this the expected output?: > > > > (gdb) frame 0 > > #0 0x0000000000619694 in fm_extract_free (qm=0x7f0114c07000, > frag=0x7f0114f0e4b0) at mem/f_malloc.c:206 > > 206 *pf=frag->u.nxt_free; > > (gdb) list > > 201 int hash; > > 202 > > 203 pf = frag->prv_free; > > 204 hash = GET_HASH(frag->size); > > 205 > > 206 *pf=frag->u.nxt_free; > > 207 > > 208 if(frag->u.nxt_free) > frag->u.nxt_free->prv_free = pf; > > 209 > > 210 qm->ffrags--; > > (gdb) > > > > Regards, > > > > Igor. > > > > *De :*sr-users [mailto:sr-users-boun...@lists.sip-router.org] > *De la part de* Daniel-Constantin Mierla > *Envoyé :* lundi 14 décembre 2015 13:15 > *À :* Kamailio (SER) - Users Mailing List > <sr-users@lists.sip-router.org> > <mailto:sr-users@lists.sip-router.org> > *Objet :* Re: [SR-Users] Kamailio 4.2.6 crash > > > > Hello, > > can you provide the list output in gdb for frame 0: > > frame 0 > list > > Cheers, > Daniel > > > > -- > > Daniel-Constantin Mierla > > http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - > http://www.linkedin.com/in/miconda > > Book: SIP Routing With Kamailio - http://www.asipto.com > > http://miconda.eu > > > > -- > Daniel-Constantin Mierla > http://twitter.com/#!/miconda <http://twitter.com/#%21/miconda> - > http://www.linkedin.com/in/miconda > Book: SIP Routing With Kamailio - http://www.asipto.com > http://miconda.eu -- Daniel-Constantin Mierla http://twitter.com/#!/miconda - http://www.linkedin.com/in/miconda Book: SIP Routing With Kamailio - http://www.asipto.com http://miconda.eu
_______________________________________________ SIP Express Router (SER) and Kamailio (OpenSER) - sr-users mailing list sr-users@lists.sip-router.org http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-users
