Package: asterisk
Version: 1:13.14.1~dfsg-2+deb9u3
Severity: important
Tags: upstream
Dear Maintainer,
I'm using Asterisk with its PJSIP backend. Every few hours Asterisk segfaults
in PJSIP library code. According to backtraces of coredumps the segfaults
seem to be related to SIP registration handling. I cannot say where the root
cause is, so I'm reporting this against asterisk and not the PJSIP library.
To work around this problem I'm currently using a self-built version of
upstream Asterisk (built-in PJSIP). From this experience I can say, that
upstream version 13.15.0 does NOT have the described problem (not a single
segfault over months). However I would really like to use standard Debian
stable packages, without self-built stuff.
Details:
Over the course of roughly 24h hours I recently got 13 segfaults. 6 of these
segfaults occured in a function called tx_data_destroy() in libpjsip:
#0 tx_data_destroy (tdata=<optimized out>) at ../src/pjsip/sip_transport.c:485
485 pjsip_endpt_release_pool( tdata->mgr->endpt, tdata->pool );
(gdb) bt
#0 tx_data_destroy (tdata=<optimized out>) at ../src/pjsip/sip_transport.c:485
#1 0x00007f686cb59cc8 in pjsip_tx_data_dec_ref (tdata=0x7f6814005748) at
../src/pjsip/sip_transport.c:501
#2 0x00007f67b22b5740 in registration_response_destroy (obj=0x7f685c000dc0) at
res_pjsip_outbound_registration.c:741
#3 0x000055ac1cbe7f39 in internal_ao2_ref
(user_data=user_data@entry=0x7f685c000dc0, delta=delta@entry=-1,
file=file@entry=0x55ac1cd4e066 "astobj2.c", line=line@entry=518,
func=func@entry=0x55ac1cd4e158 <__FUNCTION__.9326> "__ao2_ref") at
astobj2.c:451
#4 0x000055ac1cbe8528 in __ao2_ref (user_data=user_data@entry=0x7f685c000dc0,
delta=delta@entry=-1) at astobj2.c:518
#5 0x00007f67b22b6ffa in handle_registration_response (data=0x7f685c000dc0) at
res_pjsip_outbound_registration.c:825
#6 0x000055ac1cd290e8 in ast_taskprocessor_execute
(tps=tps@entry=0x55ac1e968ff0) at taskprocessor.c:965
#7 0x000055ac1cd310a0 in execute_tasks (data=0x55ac1e968ff0) at
threadpool.c:1322
#8 0x000055ac1cd290e8 in ast_taskprocessor_execute (tps=0x55ac1e39b2c0) at
taskprocessor.c:965
#9 0x000055ac1cd30a74 in threadpool_execute (pool=0x55ac1e39ae80) at
threadpool.c:351
#10 worker_active (worker=0x7f67e0001a30) at threadpool.c:1105
#11 worker_start (arg=arg@entry=0x7f67e0001a30) at threadpool.c:1024
#12 0x000055ac1cd3908c in dummy_start (data=<optimized out>) at utils.c:1235
#13 0x00007f687358a494 in start_thread (arg=0x7f686e2ae700) at
pthread_create.c:333
#14 0x00007f6872194acf in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
(gdb) list
480 pj_lock_release(tdata->mgr->lock);
481 #endif
482
483 pj_atomic_destroy( tdata->ref_cnt );
484 pj_lock_destroy( tdata->lock );
485 pjsip_endpt_release_pool( tdata->mgr->endpt, tdata->pool );
486 }
(gdb) disassemble
Dump of assembler code for function tx_data_destroy:
0x00007f686cb59c20 <+0>: push %rbx
0x00007f686cb59c21 <+1>: mov %rdi,%rbx
0x00007f686cb59c24 <+4>: callq 0x7f686cb482d0 <pj_log_get_level@plt>
0x00007f686cb59c29 <+9>: cmp $0x4,%eax
0x00007f686cb59c2c <+12>: jle 0x7f686cb59c4b <tx_data_destroy+43>
0x00007f686cb59c2e <+14>: mov %rbx,%rdi
0x00007f686cb59c31 <+17>: callq 0x7f686cb48c70
<pjsip_tx_data_get_info@plt>
0x00007f686cb59c36 <+22>: lea 0x18(%rbx),%rdi
0x00007f686cb59c3a <+26>: lea 0x16701(%rip),%rsi #
0x7f686cb70342
0x00007f686cb59c41 <+33>: mov %rax,%rdx
0x00007f686cb59c44 <+36>: xor %eax,%eax
0x00007f686cb59c46 <+38>: callq 0x7f686cb48100 <pj_log_5@plt>
0x00007f686cb59c4b <+43>: lea 0x3a8(%rbx),%rdi
0x00007f686cb59c52 <+50>: callq 0x7f686cb48b10
<pjsip_tpselector_dec_ref@plt>
0x00007f686cb59c57 <+55>: mov 0x1b0(%rbx),%rdi
0x00007f686cb59c5e <+62>: callq 0x7f686cb48400 <pj_atomic_destroy@plt>
0x00007f686cb59c63 <+67>: mov 0x180(%rbx),%rdi
0x00007f686cb59c6a <+74>: callq 0x7f686cb48870 <pj_lock_destroy@plt>
0x00007f686cb59c6f <+79>: mov 0x50(%rbx),%rax
0x00007f686cb59c73 <+83>: mov 0x10(%rbx),%rsi
0x00007f686cb59c77 <+87>: pop %rbx
=> 0x00007f686cb59c78 <+88>: mov 0x10(%rax),%rdi
0x00007f686cb59c7c <+92>: jmpq 0x7f686cb48be0
<pjsip_endpt_release_pool@plt>
End of assembler dump.
(gdb) up
#1 0x00007f686cb59cc8 in pjsip_tx_data_dec_ref (tdata=0x7f6814005748) at
../src/pjsip/sip_transport.c:501
501 tx_data_destroy(tdata);
(gdb) print tdata
$1 = (pjsip_tx_data *) 0x7f6814005748
(gdb) print tdata->pool
$2 = (pj_pool_t *) 0x7f6814005645
(gdb) print tdata->mgr
$3 = (pjsip_tpmgr *) 0x554b43415250
(gdb) print tdata->mgr->endpt
Cannot access memory at address 0x554b43415260
It seems like the endpoint struct is gone? But why? Broken pointer? Already
free'ed?
Here are the other types of segfaults, which I haven't had a closer look at yet:
2 segfaults occured in function pj_atomic_inc_and_get() in libpj:
(gdb) bt
#0 0x00007fce2dcd4999 in pj_atomic_inc_and_get () from
/usr/lib/x86_64-linux-gnu/libpj.so.2
#1 0x00007fcdb878e5a3 in sip_outbound_registration_response_cb
(param=0x7fce7467c6e0) at res_pjsip_outbound_registration.c:956
#2 0x00007fce2f250358 in ?? () from /usr/lib/x86_64-linux-gnu/libpjsip-ua.so.2
#3 0x00007fce2f251a2f in ?? () from /usr/lib/x86_64-linux-gnu/libpjsip-ua.so.2
#4 0x00007fce2ee0bb11 in tsx_set_state (tsx=tsx@entry=0x7fce34005988,
state=state@entry=PJSIP_TSX_STATE_COMPLETED,
event_src_type=event_src_type@entry=PJSIP_EVENT_RX_MSG,
event_src=0x7fce180098e8, flag=flag@entry=0) at
../src/pjsip/sip_transaction.c:1234
#5 0x00007fce2ee0d550 in tsx_on_state_proceeding_uac (tsx=0x7fce34005988,
event=0x7fce7467ca80) at ../src/pjsip/sip_transaction.c:2958
#6 0x00007fce2ee0d76e in tsx_on_state_calling (tsx=0x7fce34005988,
event=0x7fce7467ca80) at ../src/pjsip/sip_transaction.c:2541
#7 0x00007fce2ee0eaef in pjsip_tsx_recv_msg (tsx=tsx@entry=0x7fce34005988,
rdata=rdata@entry=0x7fce180098e8) at ../src/pjsip/sip_transaction.c:1788
#8 0x00007fce2ee0ebb5 in mod_tsx_layer_on_rx_response (rdata=0x7fce180098e8)
at ../src/pjsip/sip_transaction.c:876
#9 0x00007fce2edf93e6 in pjsip_endpt_process_rx_data (endpt=<optimized out>,
rdata=rdata@entry=0x7fce180098e8, p=p@entry=0x7fcded3a8be0 <param>,
p_handled=p_handled@entry=0x7fce7467cb94) at ../src/pjsip/sip_endpoint.c:895
#10 0x00007fcded1851bc in distribute (data=0x7fce180098e8) at
res_pjsip/pjsip_distributor.c:769
#11 0x000055ef7b8b30e8 in ast_taskprocessor_execute
(tps=tps@entry=0x55ef7c9b0e40) at taskprocessor.c:965
#12 0x000055ef7b8bb0a0 in execute_tasks (data=0x55ef7c9b0e40) at
threadpool.c:1322
#13 0x000055ef7b8b30e8 in ast_taskprocessor_execute (tps=0x55ef7c101420) at
taskprocessor.c:965
#14 0x000055ef7b8baa74 in threadpool_execute (pool=0x55ef7c0ffde0) at
threadpool.c:351
#15 worker_active (worker=0x7fcddc00b9e0) at threadpool.c:1105
#16 worker_start (arg=arg@entry=0x7fcddc00b9e0) at threadpool.c:1024
#17 0x000055ef7b8c308c in dummy_start (data=<optimized out>) at utils.c:1235
#18 0x00007fce79959494 in start_thread (arg=0x7fce7467d700) at
pthread_create.c:333
#19 0x00007fce78563acf in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
2 segfaults occured in function pj_pool_alloc_from_block() in libpj:
(gdb) bt
#0 0x00007fdde6efcea4 in pj_pool_alloc_from_block () from
/usr/lib/x86_64-linux-gnu/libpj.so.2
#1 0x00007fdde6efd105 in pj_pool_alloc () from
/usr/lib/x86_64-linux-gnu/libpj.so.2
#2 0x00007fdde6efd145 in pj_pool_calloc () from
/usr/lib/x86_64-linux-gnu/libpj.so.2
#3 0x00007fddfc0550d3 in pj_pool_zalloc (size=288,
pool=pool@entry=0x7fdd6400ca18) at ../../pjlib/include/pj/pool.h:476
#4 pjsip_authorization_hdr_create (pool=pool@entry=0x7fdd6400ca18) at
../src/pjsip/sip_auth_msg.c:51
#5 0x00007fddfc0532b0 in auth_respond (req_pool=req_pool@entry=0x7fdd6400ca18,
hdr=hdr@entry=0x7fddb8008c00, uri=uri@entry=0x7fdd6400ced0,
cred_info=0x7fdd6400c998, method=0x7fdd6400d408, sess_pool=<optimized out>,
cached_auth=0x7fdd6400ca48, p_h_auth=0x7fdd3ac205e8) at
../src/pjsip/sip_auth_client.c:694
#6 0x00007fddfc05425c in process_auth (h_auth=0x7fdd3ac205e8,
cached_auth=0x7fdd6400ca48, sess=0x7fdd3ac20650, tdata=0x7fdd6400c998,
uri=0x7fdd6400ced0, hchal=0x7fddb8008c00, req_pool=0x7fdd6400ca18) at
../src/pjsip/sip_auth_client.c:1115
#7 pjsip_auth_clt_reinit_req (sess=sess@entry=0x7fdd3ac20650,
rdata=rdata@entry=0x7fddb8007218, old_request=old_request@entry=0x7fdd6400c998,
new_request=new_request@entry=0x7fdd3ac20728) at
../src/pjsip/sip_auth_client.c:1200
#8 0x00007fdd41aa6009 in digest_create_request_with_auth_from_old
(auths=<optimized out>, challenge=0x7fddb8007218, old_request=0x7fdd6400c998,
new_request=0x7fdd3ac20728) at res_pjsip_outbound_authenticator_digest.c:126
#9 0x00007fdd4189f162 in handle_registration_response (data=0x7fddb8003640) at
res_pjsip_outbound_registration.c:811
#10 0x00005651df0160e8 in ast_taskprocessor_execute
(tps=tps@entry=0x5651dfdd1690) at taskprocessor.c:965
#11 0x00005651df01e0a0 in execute_tasks (data=0x5651dfdd1690) at
threadpool.c:1322
#12 0x00005651df0160e8 in ast_taskprocessor_execute (tps=0x5651e02389b0) at
taskprocessor.c:965
#13 0x00005651df01da74 in threadpool_execute (pool=0x5651e0237f30) at
threadpool.c:351
#14 worker_active (worker=0x7fdd6c000a80) at threadpool.c:1105
#15 worker_start (arg=arg@entry=0x7fdd6c000a80) at threadpool.c:1024
#16 0x00005651df02608c in dummy_start (data=<optimized out>) at utils.c:1235
#17 0x00007fde02a7a494 in start_thread (arg=0x7fdd3ac21700) at
pthread_create.c:333
#18 0x00007fde01684acf in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
2 segfaults occured in function process_auth() in libpjsip:
(gdb) bt
#0 0x00007f7858a28fab in process_auth (h_auth=0x7f77d83435e8,
cached_auth=0x7f7804004698, sess=0x7f77d8343650, tdata=0x7f78040045e8,
uri=0x7f7804004b20, hchal=0x7f78200086e0, req_pool=0x7f7804004668) at
../src/pjsip/sip_auth_client.c:1025
#1 pjsip_auth_clt_reinit_req (sess=sess@entry=0x7f77d8343650,
rdata=rdata@entry=0x7f7820006cf8, old_request=old_request@entry=0x7f78040045e8,
new_request=new_request@entry=0x7f77d8343728) at
../src/pjsip/sip_auth_client.c:1200
#2 0x00007f779e4be009 in digest_create_request_with_auth_from_old
(auths=<optimized out>, challenge=0x7f7820006cf8, old_request=0x7f78040045e8,
new_request=0x7f77d8343728) at res_pjsip_outbound_authenticator_digest.c:126
#3 0x00007f779e2b7162 in handle_registration_response (data=0x7f7820001c80) at
res_pjsip_outbound_registration.c:811
#4 0x00005604f0d320e8 in ast_taskprocessor_execute
(tps=tps@entry=0x5604f28b3390) at taskprocessor.c:965
#5 0x00005604f0d3a0a0 in execute_tasks (data=0x5604f28b3390) at
threadpool.c:1322
#6 0x00005604f0d320e8 in ast_taskprocessor_execute (tps=0x5604f29697c0) at
taskprocessor.c:965
#7 0x00005604f0d39a74 in threadpool_execute (pool=0x5604f2967e00) at
threadpool.c:351
#8 worker_active (worker=0x7f77cc0055a0) at threadpool.c:1105
#9 worker_start (arg=arg@entry=0x7f77cc0055a0) at threadpool.c:1024
#10 0x00005604f0d4208c in dummy_start (data=<optimized out>) at utils.c:1235
#11 0x00007f785f44f494 in start_thread (arg=0x7f77d8344700) at
pthread_create.c:333
#12 0x00007f785e059acf in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
I'm not familier and have never hacked on Asterisk/PJSIP code yet, so
I'm happy to be guided by someone who knows the code and where to
look.
-- System Information:
Debian Release: 9.5
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 4.9.110 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8),
LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Init: sysvinit (via /sbin/init)
Versions of packages asterisk depends on:
ii adduser 3.115
ii asterisk-config 1:13.14.1~dfsg-2+deb9u3
ii asterisk-core-sounds-en 1.4.27-1
ii asterisk-modules 1:13.14.1~dfsg-2+deb9u3
ii init-system-helpers 1.48
ii libbsd0 0.8.3-1
ii libc6 2.24-11+deb9u3
ii libcap2 1:2.25-1
ii libedit2 3.1-20160903-3
ii libgcc1 1:6.3.0-18+deb9u1
ii libjansson4 2.9-1
ii libncurses5 6.0+20161126-1+deb9u2
ii libpopt0 1.16-10+b2
ii libsqlite3-0 3.16.2-5+deb9u1
ii libssl1.1 1.1.0f-3+deb9u2
ii libstdc++6 6.3.0-18+deb9u1
ii libsystemd0 232-23
ii libtinfo5 6.0+20161126-1+deb9u2
ii liburiparser1 0.8.4-1
ii libuuid1 2.29.2-1+deb9u1
ii libxml2 2.9.4+dfsg1-2.2+deb9u2
ii libxslt1.1 1.1.29-2.1
ii lsb-base 9.20161125
Versions of packages asterisk recommends:
ii asterisk-moh-opsound-gsm 2.03-1
ii asterisk-voicemail-imapstorage [asterisk-voicemail 1:13.14.1~dfsg-2+deb9u3
ii sox 14.4.1-5+b2
Versions of packages asterisk suggests:
pn asterisk-dahdi <none>
pn asterisk-dev <none>
pn asterisk-doc <none>
pn asterisk-ooh323 <none>
pn asterisk-opus <none>
pn asterisk-vpb <none>
-- no debconf information
