alexbakker created an issue (kamailio/kamailio#4231)
### Description
Performing an HTTPS request using the ``http_client`` module results in a
shared memory leak, seemingly related to TLS.
### Troubleshooting
#### Reproduction
Our use case for ``http_client`` is to update an htable with the latest banned
IP's from APIBAN as described here:
https://github.com/apiban/apiban?tab=readme-ov-file#integration-into-kamailio.
Normally, we refresh the list of banned IP's once every 15 minutes, so the
shared memory usage grows fairly slowly.
The issue can be reproduced using the Kamailio configuration file below. To
demonstrate the issue and make the shared memory usage grow faster, we set up a
timer that calls ``http_client_query`` in a tight loop. To prevent unnecessary
load on the APIBAN infrastructure, we request an example file from my personal
server instead.
```
#!KAMAILIO
debug=2
log_stderror=yes
fork=yes
enable_tls=1
loadmodule "tls.so"
loadmodule "cfg_rpc.so"
loadmodule "pv.so"
loadmodule "xlog.so"
loadmodule "ctl.so"
loadmodule "rtimer.so"
loadmodule "http_client.so"
modparam("tls", "certificate", "")
modparam("tls", "private_key", "")
modparam("rtimer", "timer", "name=apiban;interval=100u;mode=1;")
modparam("rtimer", "exec", "timer=apiban;route=APIBAN;")
route[APIBAN] {
xinfo("running apiban refresh\n");
http_client_query("https://alexbakker.me/u/7bvjn9jfas.txt";, "$var(banned)");
}
```
Start Kamailio:
```
kamailio -f kamailio.cfg -DD
```
And then watch the shared memory usage grow rapidly:
```
watch -n1 -- kamcmd core.shmmem
```
#### Log Messages
Eventually, Kamailio will run out of shared memory and print the following
messages to the log:
```
21(184184) INFO: <script>: running apiban refresh
21(184184) ERROR: http_client [functions.c:471]: curL_request_url(): failed to
perform curl (56) (url: https://alexbakker.me/u/7bvjn9jfas.txt)
21(184184) INFO: <script>: running apiban refresh
21(184184) ERROR: http_client [functions.c:471]: curL_request_url(): failed to
perform curl (56) (url: https://alexbakker.me/u/7bvjn9jfas.txt)
21(184184) INFO: <script>: running apiban refresh
21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS
error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt)
21(184184) INFO: <script>: running apiban refresh
21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS
error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt)
21(184184) INFO: <script>: running apiban refresh
21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS
error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt)
21(184184) INFO: <script>: running apiban refresh
21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS
error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt)
21(184184) INFO: <script>: running apiban refresh
21(184184) ERROR: <core> [core/mem/q_malloc.c:758]: qm_realloc():
qm_realloc(0x7585c73c4000, 1024) called from tls: tls_init.c: ser_realloc(372),
module: tls; qm_malloc() failed!
21(184184) INFO: <script>: running apiban refresh
21(184184) INFO: <script>: running apiban refresh
21(184184) ERROR: <core> [core/mem/q_malloc.c:758]: qm_realloc():
qm_realloc(0x7585c73c4000, 1024) called from tls: tls_init.c: ser_realloc(372),
module: tls; qm_malloc() failed!
21(184184) WARNING: http_client [functions.c:453]: curL_request_url(): TLS
error in curl connection (url: https://alexbakker.me/u/7bvjn9jfas.txt)
21(184184) INFO: <script>: running apiban refresh
21(184184) INFO: <script>: running apiban refresh
21(184184) WARNING: http_client [functions.c:463]: curL_request_url(): TLS CA
certificate read error (url: https://alexbakker.me/u/7bvjn9jfas.txt)
```
``mem_dump_shm`` reports a large list of TLS-related allocations. Just sharing
the last few lines here, as they're seemingly all from the same location.
```
20(204914) ALERT: qm_status: qm_status(): 28500. N address=0x705ab3178710
frag=0x705ab31786d0 size=112 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls:
tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28501. N address=0x705ab31787f0
frag=0x705ab31787b0 size=48 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls:
tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28502. N address=0x705ab3178890
frag=0x705ab3178850 size=32 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls:
tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28503. N address=0x705ab3178920
frag=0x705ab31788e0 size=32 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls:
tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28504. N address=0x705ab31789b0
frag=0x705ab3178970 size=32 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls:
tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28505. N address=0x705ab3178a40
frag=0x705ab3178a00 size=48 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls:
tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28506. N address=0x705ab3178ae0
frag=0x705ab3178aa0 size=48 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls:
tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28507. N address=0x705ab3178b80
frag=0x705ab3178b40 size=144 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls:
tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28509. N address=0x705ab3178d00
frag=0x705ab3178cc0 size=32 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls:
tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28510. N address=0x705ab3178d90
frag=0x705ab3178d50 size=32 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls:
tls_init.c: ser_realloc(372)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28512. N address=0x705ab3178ea0
frag=0x705ab3178e60 size=16 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls:
tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28513. N address=0x705ab3178f20
frag=0x705ab3178ee0 size=32 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls:
tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28515. N address=0x705ab317a3c0
frag=0x705ab317a380 size=112 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls:
tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28516. N address=0x705ab317a4a0
frag=0x705ab317a460 size=512 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls:
tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28517. N address=0x705ab317a710
frag=0x705ab317a6d0 size=1504 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls:
tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
20(204914) ALERT: qm_status: qm_status(): 28518. N address=0x705ab317ad60
frag=0x705ab317ad20 size=1040 used=1
20(204914) ALERT: qm_status: qm_status(): alloc'd from tls:
tls_init.c: ser_malloc(364)
20(204914) ALERT: qm_status: qm_status(): start check=f0f0f0f0, end
check= c0c0c0c0, abcdefed
```
### Possible Solutions
I'm not aware of a workaround for this issue. For our specific use case, I'll
move the APIBAN polling out of Kamailio to a separate process for now.
### Additional Information
I've verified that this issue is reproducible on ``5.5.7``, ``5.6.6``,
``5.7.6``, ``5.8.6`` and ``6.0.1``.
* **Kamailio Version** - output of `kamailio -v`
```
version: kamailio 6.0.1 (x86_64/linux) fce50d
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE,
USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC,
F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT,
USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES,
TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE
262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: fce50d
compiled on 06:42:29 May 6 2025 with gcc 12.2.0
```
* **Operating System**:
```
$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description: Debian GNU/Linux 12 (bookworm)
Release: 12
Codename: bookworm
$ uname -a
Linux ip-10-146-128-121 6.1.0-32-cloud-arm64 #1 SMP Debian 6.1.129-1
(2025-03-06) aarch64 GNU/Linux
```
--
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/4231
You are receiving this because you are subscribed to this thread.
Message ID: <kamailio/kamailio/issues/4...@github.com>_______________________________________________
Kamailio - Development Mailing List -- sr-dev@lists.kamailio.org
To unsubscribe send an email to sr-dev-le...@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the
sender!
