On 2023-01-24 06:58:57, Willy Tarreau wrote:
> Hi Marc,
Hi Willy,
> See the difference ? There seems to be an insane FD locking cost on this
> system that simply wastes 40% of the CPU there. So I suspect that in your
> first tests you were stressing the locking while in the last ones you
> were stressing the SSL stack.
>
> Stupid question but I prefer to ask in order to be certain, are all of
> these 32 threads located on the same physical CPU ? I just want to be
> sure that locks (kernel or user) are not traveling between multiple CPU
> sockets, as that ruins scalability.
Very good observation. This is a 2 socket system, 20 cores per socket,
and since there is no affinity in OpenBSD unfortunately the 32 threads
are not on the same physical CPU.
cpu0 reports as core 0 package 0, cpu1 core 0 package 1, and so on that
the odd cpus are socket 1 and evens are socket 2. Watching threads in
top with a 1 sec interval I see them bouncing around a lot between
sockets.
> It's great already to be able to rule out that one. Another useful
> test you could run is to place a reject rule at the connection level
> in your frontend (it will happen before SSL tries to process traffic):
>
> tcp-request connection reject
>
> (don't do that in production of course). Once this is installed you
> can compare again the CPU usage with no-thread, and 32-thread without
> shards and 32 threads with 8 shards.
This is HTTPS with the reject rule:
- no nbthread, no shards:
-- 40 CPUs: 0.3% user, 0.0% nice, 0.7% sys, 0.1% spin, 0.1% intr, 98.9%
idle
-- haproxy peaked at 20% CPU
- nbthread 32, no shards:
-- 40 CPUs: 0.5% user, 0.0% nice, 19.9% sys, 4.6% spin, 0.1% intr, 75.0%
idle
-- haproxy peaked at 863% CPU
- nbthread 32, 8 shards:
-- 40 CPUs: 0.4% user, 0.0% nice, 1.9% sys, 0.1% spin, 0.1% intr, 97.4%
idle
-- haproxy peaked at 62% CPU
-- this only used 5 threads according to top FYI
Since the SSL stack has been mentioned a couple times I retested with
HTTP instead of HTTPS and the results are interesting:
- HTTP, no nbthread, no shards
-- 40 CPUs: 0.6% user, 0.0% nice, 2.0% sys, 0.2% spin, 0.2% intr, 97.0%
idle
-- haproxy peaked at 54% CPU
-- current conns = 475; current pipes = 0/0; conn rate = 0/sec; bit rate
403.512 Mbps, Running tasks: 0/922; idle = 28 %
-- First 3 seconds 7001, 14474, 22000 responses. 7-8k/sec throughout
test. 100% success 0 fail
- HTTP, nbthread 32, no shards
-- 40 CPUs: 0.7% user, 0.0% nice, 29.6% sys, 0.2% spin, 0.5% intr, 69.1%
idle
-- haproxy peaked at 800% CPU
-- current conns = 501; current pipes = 0/0; conn rate = 1/sec; bit rate
410.711 Mbps, Running tasks: 28/967; idle = 64 %
-- First 3 seconds 7000, 14500, 22002 responses. 7-8k/sec during test. 100%
success 0 fail
- HTTP, nbthread 32, 8 shards
-- 40 CPUs: 0.7% user, 0.0% nice, 5.0% sys, 0.2% spin, 0.2% intr, 94.0%
idle
-- haproxy peaked at 147% CPUs
-- current conns = 500; current pipes = 0/0; conn rate = 1/sec; bit rate
401.946 Mbps, Running tasks: 0/746; idle = 93 %
-- First 3 seconds 7000, 14500, 22000 responses. 7-8k/sec during test. 100%
success 0 fail
Here we still see sys% growing with more threads but even with the waste,
with HTTP we get a reliable 7-8k responses/sec and 100% success rate
instead of 400-500/sec for a few second burst and then a total stall /
nearly 0% success rate with HTTPS. I expect SSL to have some cost but
not quite this huge, and the "stall" of traffic/health checks under
heavy HTTPS load is a bit puzzling.
Do you think it would be worth trying to install OpenSSL 3.0.7 from
ports and manually build haproxy against that to compare with the
current LibreSSL 3.6.0? Or is the bottleneck likely somewhere else?
> Ah, great! Do you have any info on the signal that was received there ?
> If you still have the core, issuing "info registers", then "disassemble
> conn_backend_get" and pressing enter till the end of the function could
> be useful to try to locate what's happening there.
It was signal 11 and yes I do still have the core, attached!
GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-unknown-openbsd7.2"...
Core was generated by `haproxy'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /usr/lib/libpthread.so.26.2...done.
Loaded symbols for /usr/lib/libpthread.so.26.2
Loaded symbols for /usr/local/sbin/haproxy
Reading symbols from /usr/lib/libz.so.7.0...done.
Loaded symbols for /usr/lib/libz.so.7.0
Symbols already loaded for /usr/lib/libpthread.so.26.2
Reading symbols from /usr/lib/libssl.so.53.0...done.
Loaded symbols for /usr/lib/libssl.so.53.0
Reading symbols from /usr/lib/libcrypto.so.50.0...done.
Loaded symbols for /usr/lib/libcrypto.so.50.0
Reading symbols from /usr/local/lib/libpcre2-8.so.0.6...done.
Loaded symbols for /usr/local/lib/libpcre2-8.so.0.6
Reading symbols from /usr/local/lib/libpcre2-posix.so.1.0...done.
Loaded symbols for /usr/local/lib/libpcre2-posix.so.1.0
Reading symbols from /usr/lib/libc.so.96.2...done.
Loaded symbols for /usr/lib/libc.so.96.2
Reading symbols from /usr/libexec/ld.so...Error while reading shared library
symbols:
Dwarf Error: wrong version in compilation unit header (is 4, should be 2) [in
module /usr/libexec/ld.so]
#0 conn_backend_get (s=0xafab7609000, srv=0xafac107d000, is_safe=0,
hash=25106765055229623) at src/backend.c:1230
1230 src/backend.c: No such file or directory.
in src/backend.c
(gdb) bt
#0 conn_backend_get (s=0xafab7609000, srv=0xafac107d000, is_safe=0,
hash=25106765055229623) at src/backend.c:1230
#1 0x00000af892c7442e in back_try_conn_req (s=0xafab7609000) at endian.h:37
#2 0x00000af892bfa355 in process_stream (t=0xafafd7ed800,
context=0xafab7609000, state=Unhandled dwarf expression opcode 0xa3
) at src/stream.c:2302
#3 0x00000af892ceaf28 in run_tasks_from_lists (budgets=0xafb15d0d250) at
src/task.c:682
#4 0x00000af892cebeb9 in process_runnable_tasks () at src/task.c:925
#5 0x00000af892cb2e03 in run_poll_loop () at src/haproxy.c:2818
#6 0x00000af892cb7641 in run_thread_poll_loop (data=Variable "data" is not
available.) at src/haproxy.c:2993
#7 0x00000afaa6bf3ab1 in _rthread_start (v=Unhandled dwarf expression opcode
0xa3) at /usr/src/lib/librthread/rthread.c:96
#8 0x00000afb06bbdaca in __tfork_thread () at
/usr/src/lib/libc/arch/amd64/sys/tfork_thread.S:84#9 0x00000afb06bbdaca in
__tfork_thread () at /usr/src/lib/libc/arch/amd64/sys/tfork_thread.S:84Previous
frame identical to this frame (corrupt stack?)
(gdb) bt full
#0 conn_backend_get (s=0xafab7609000, srv=0xafac107d000, is_safe=0,
hash=25106765055229623) at src/backend.c:1230 __pl_r = Variable "__pl_r"
is not available.
(gdb)
(gdb) info registers
rax 0x0 0
rbx 0xafab7609000 12071934660608
rcx 0xafad0cc51c0 12072361152960
rdx 0xaf892d7a2c0 12062731772608
rsi 0x59327844937eb7 25106765055229623
rdi 0xafb692210a0 12074916909216
rbp 0xafb15d0cff0 0xafb15d0cff0
rsp 0xafb15d0cfa0 0xafb15d0cfa0
r8 0x165667b19e3779f9 1609587929392839161
r9 0x8 8
r10 0x85ebca77c2b2ae63 -8796714831421723037
r11 0x3e1ef055ec5ff740 4476279331481450304
r12 0xafad0cb9600 12072361104896
r13 0x2 2
r14 0xaf892d7a358 12062731772760
r15 0xafac107d000 12072096616448
rip 0xaf892c770c5 0xaf892c770c5 <conn_backend_get+1157>
eflags 0x10206 66054
cs 0x2b 43
ss 0x23 35
ds 0x23 35
es 0x23 35
fs 0x23 35
gs 0x23 35
Current language: auto; currently minimal
(gdb) disassemble
Dump of assembler code for function conn_backend_get:
0x00000af892c76c40 <conn_backend_get+0>: mov 900913(%rip),%r11
# 0xaf892d52b78 <__retguard_600>
0x00000af892c76c47 <conn_backend_get+7>: xor (%rsp),%r11
0x00000af892c76c4b <conn_backend_get+11>: push %rbp
0x00000af892c76c4c <conn_backend_get+12>: mov %rsp,%rbp
0x00000af892c76c4f <conn_backend_get+15>: push %r11
0x00000af892c76c51 <conn_backend_get+17>: push %r15
0x00000af892c76c53 <conn_backend_get+19>: push %r14
0x00000af892c76c55 <conn_backend_get+21>: push %r13
0x00000af892c76c57 <conn_backend_get+23>: push %r12
0x00000af892c76c59 <conn_backend_get+25>: push %rbx
0x00000af892c76c5a <conn_backend_get+26>: sub $0x20,%rsp
0x00000af892c76c5e <conn_backend_get+30>: mov
%rcx,0xffffffffffffffc0(%rbp)
0x00000af892c76c62 <conn_backend_get+34>: mov
%edx,0xffffffffffffffcc(%rbp)
0x00000af892c76c65 <conn_backend_get+37>: mov %rsi,%r15
0x00000af892c76c68 <conn_backend_get+40>: mov
%rdi,0xffffffffffffffb8(%rbp)
0x00000af892c76c6c <conn_backend_get+44>: mov 953093(%rip),%r12
# 0xaf892d5f778 <_DYNAMIC+472>
0x00000af892c76c73 <conn_backend_get+51>: mov %r12,%rdi
0x00000af892c76c76 <conn_backend_get+54>: callq 0xaf892d4dba0
<__emutls_get_address>
0x00000af892c76c7b <conn_backend_get+59>: mov (%rax),%r13d
0x00000af892c76c7e <conn_backend_get+62>: mov 953267(%rip),%r14
# 0xaf892d5f838 <_DYNAMIC+664>
0x00000af892c76c85 <conn_backend_get+69>: mov $0x100000004,%rbx
0x00000af892c76c8f <conn_backend_get+79>: mov %r13d,%eax
0x00000af892c76c92 <conn_backend_get+82>: mov %eax,%eax
0x00000af892c76c94 <conn_backend_get+84>: shl $0x6,%rax
0x00000af892c76c98 <conn_backend_get+88>: mov 0x18(%r14,%rax,1),%rax
0x00000af892c76c9d <conn_backend_get+93>: shr $0x20,%rax
0x00000af892c76ca1 <conn_backend_get+97>: jne 0xaf892c76cdc
<conn_backend_get+156>
0x00000af892c76ca3 <conn_backend_get+99>: mov %r12,%rdi
0x00000af892c76ca6 <conn_backend_get+102>: callq 0xaf892d4dba0
<__emutls_get_address>
0x00000af892c76cab <conn_backend_get+107>: mov (%rax),%eax
0x00000af892c76cad <conn_backend_get+109>: shl $0x6,%rax
0x00000af892c76cb1 <conn_backend_get+113>: mov %rbx,%rcx
0x00000af892c76cb4 <conn_backend_get+116>: lock xadd %rcx,0x18(%r14,%rax,1)
0x00000af892c76cbb <conn_backend_get+123>: shr $0x20,%rcx
0x00000af892c76cbf <conn_backend_get+127>: je 0xaf892c76cea
<conn_backend_get+170>
0x00000af892c76cc1 <conn_backend_get+129>: mov %r12,%rdi
0x00000af892c76cc4 <conn_backend_get+132>: callq 0xaf892d4dba0
<__emutls_get_address>
0x00000af892c76cc9 <conn_backend_get+137>: mov (%rax),%eax
0x00000af892c76ccb <conn_backend_get+139>: shl $0x6,%rax
0x00000af892c76ccf <conn_backend_get+143>: mov 953186(%rip),%r14
# 0xaf892d5f838 <_DYNAMIC+664>
0x00000af892c76cd6 <conn_backend_get+150>: lock sub %rbx,0x18(%r14,%rax,1)
0x00000af892c76cdc <conn_backend_get+156>: pause
0x00000af892c76cde <conn_backend_get+158>: mov %r12,%rdi
0x00000af892c76ce1 <conn_backend_get+161>: callq 0xaf892d4dba0
<__emutls_get_address>
0x00000af892c76ce6 <conn_backend_get+166>: mov (%rax),%eax
0x00000af892c76ce8 <conn_backend_get+168>: jmp 0xaf892c76c92
<conn_backend_get+82>
0x00000af892c76cea <conn_backend_get+170>: mov 952967(%rip),%rdi
# 0xaf892d5f778 <_DYNAMIC+472>
0x00000af892c76cf1 <conn_backend_get+177>: callq 0xaf892d4dba0
<__emutls_get_address>
0x00000af892c76cf6 <conn_backend_get+182>: mov 0x58(%r15),%rcx
0x00000af892c76cfa <conn_backend_get+186>: mov (%rax),%eax
0x00000af892c76cfc <conn_backend_get+188>: cmpl
$0x0,0xffffffffffffffcc(%rbp)
0x00000af892c76d00 <conn_backend_get+192>: je 0xaf892c76d25
<conn_backend_get+229>
0x00000af892c76d02 <conn_backend_get+194>: shl $0x6,%rax
0x00000af892c76d06 <conn_backend_get+198>: lea (%rcx,%rax,1),%rdi
0x00000af892c76d0a <conn_backend_get+202>: add $0x20,%rdi
0x00000af892c76d0e <conn_backend_get+206>: mov
0xffffffffffffffc0(%rbp),%rsi
0x00000af892c76d12 <conn_backend_get+210>: callq 0xaf892c24e00
<srv_lookup_conn>
---Type <return> to continue, or q <return> to quit---conn_backend_get
0x00000af892c76d17 <conn_backend_get+215>: test %rax,%rax
0x00000af892c76d1a <conn_backend_get+218>: mov
0xffffffffffffffb8(%rbp),%rbx
0x00000af892c76d1e <conn_backend_get+222>: je 0xaf892c76d55
<conn_backend_get+277>
0x00000af892c76d20 <conn_backend_get+224>: mov %rax,%r12
0x00000af892c76d23 <conn_backend_get+227>: jmp 0xaf892c76d46
<conn_backend_get+262>
0x00000af892c76d25 <conn_backend_get+229>: shl $0x6,%rax
0x00000af892c76d29 <conn_backend_get+233>: lea (%rcx,%rax,1),%rdi
0x00000af892c76d2d <conn_backend_get+237>: add $0x10,%rdi
0x00000af892c76d31 <conn_backend_get+241>: mov
0xffffffffffffffc0(%rbp),%rsi
0x00000af892c76d35 <conn_backend_get+245>: callq 0xaf892c24e00
<srv_lookup_conn>
0x00000af892c76d3a <conn_backend_get+250>: mov %rax,%r12
0x00000af892c76d3d <conn_backend_get+253>: test %rax,%rax
0x00000af892c76d40 <conn_backend_get+256>: mov
0xffffffffffffffb8(%rbp),%rbx
0x00000af892c76d44 <conn_backend_get+260>: je 0xaf892c76d5a
<conn_backend_get+282>
0x00000af892c76d46 <conn_backend_get+262>: mov 0xb0(%r12),%rdi
0x00000af892c76d4e <conn_backend_get+270>: callq 0xaf892c89cb0
<conn_delete_from_tree>
0x00000af892c76d53 <conn_backend_get+275>: jmp 0xaf892c76da6
<conn_backend_get+358>
0x00000af892c76d55 <conn_backend_get+277>: xor %r12d,%r12d
0x00000af892c76d58 <conn_backend_get+280>: jmp 0xaf892c76da6
<conn_backend_get+358>
0x00000af892c76d5a <conn_backend_get+282>: cmpl $0x0,0x150(%r15)
0x00000af892c76d62 <conn_backend_get+290>: je 0xaf892c76d9c
<conn_backend_get+348>
0x00000af892c76d64 <conn_backend_get+292>: mov 0x58(%r15),%r14
0x00000af892c76d68 <conn_backend_get+296>: mov 952841(%rip),%rdi
# 0xaf892d5f778 <_DYNAMIC+472>
0x00000af892c76d6f <conn_backend_get+303>: callq 0xaf892d4dba0
<__emutls_get_address>
0x00000af892c76d74 <conn_backend_get+308>: mov (%rax),%eax
0x00000af892c76d76 <conn_backend_get+310>: shl $0x6,%rax
0x00000af892c76d7a <conn_backend_get+314>: lea (%r14,%rax,1),%rdi
0x00000af892c76d7e <conn_backend_get+318>: add $0x20,%rdi
0x00000af892c76d82 <conn_backend_get+322>: mov
0xffffffffffffffc0(%rbp),%rsi
0x00000af892c76d86 <conn_backend_get+326>: callq 0xaf892c24e00
<srv_lookup_conn>
0x00000af892c76d8b <conn_backend_get+331>: test %rax,%rax
0x00000af892c76d8e <conn_backend_get+334>: je 0xaf892c76d9c
<conn_backend_get+348>
0x00000af892c76d90 <conn_backend_get+336>: mov %rax,%r12
0x00000af892c76d93 <conn_backend_get+339>: movl
$0x1,0xffffffffffffffcc(%rbp)
0x00000af892c76d9a <conn_backend_get+346>: jmp 0xaf892c76d46
<conn_backend_get+262>
0x00000af892c76d9c <conn_backend_get+348>: xor %r12d,%r12d
0x00000af892c76d9f <conn_backend_get+351>: movl
$0x0,0xffffffffffffffcc(%rbp)
0x00000af892c76da6 <conn_backend_get+358>: mov 952779(%rip),%rdi
# 0xaf892d5f778 <_DYNAMIC+472>
0x00000af892c76dad <conn_backend_get+365>: callq 0xaf892d4dba0
<__emutls_get_address>
0x00000af892c76db2 <conn_backend_get+370>: mov (%rax),%eax
0x00000af892c76db4 <conn_backend_get+372>: shl $0x6,%rax
0x00000af892c76db8 <conn_backend_get+376>: mov $0x100000004,%rcx
0x00000af892c76dc2 <conn_backend_get+386>: mov 952943(%rip),%rdx
# 0xaf892d5f838 <_DYNAMIC+664>
0x00000af892c76dc9 <conn_backend_get+393>: lock sub %rcx,0x18(%rdx,%rax,1)
0x00000af892c76dcf <conn_backend_get+399>: test %r12,%r12
0x00000af892c76dd2 <conn_backend_get+402>: je 0xaf892c76ea4
<conn_backend_get+612>
0x00000af892c76dd8 <conn_backend_get+408>: lea 0x1(%r13),%eax
0x00000af892c76ddc <conn_backend_get+412>: mov 952685(%rip),%rcx
# 0xaf892d5f750 <_DYNAMIC+432>
0x00000af892c76de3 <conn_backend_get+419>: xor %edx,%edx
0x00000af892c76de5 <conn_backend_get+421>: cmp 0xc(%rcx),%eax
0x00000af892c76de8 <conn_backend_get+424>: cmovne %eax,%edx
0x00000af892c76deb <conn_backend_get+427>: mov %edx,0x160(%r15)
0x00000af892c76df2 <conn_backend_get+434>: mov $0x1,%eax
0x00000af892c76df7 <conn_backend_get+439>: lock xadd %eax,0x154(%r15)
0x00000af892c76e00 <conn_backend_get+448>: add $0x1,%eax
0x00000af892c76e03 <conn_backend_get+451>: mov 0x158(%r15),%ecx
0x00000af892c76e0a <conn_backend_get+458>: cmp %eax,%ecx
0x00000af892c76e0c <conn_backend_get+460>: jae 0xaf892c76e15
<conn_backend_get+469>
---Type <return> to continue, or q <return> to quit---
0x00000af892c76e0e <conn_backend_get+462>: mov %eax,0x158(%r15)
0x00000af892c76e15 <conn_backend_get+469>: mov 0x15c(%r15),%ecx
0x00000af892c76e1c <conn_backend_get+476>: cmp %eax,%ecx
0x00000af892c76e1e <conn_backend_get+478>: jae 0xaf892c76e27
<conn_backend_get+487>
0x00000af892c76e20 <conn_backend_get+480>: mov %eax,0x15c(%r15)
0x00000af892c76e27 <conn_backend_get+487>: lock subl $0x1,0x148(%r15)
0x00000af892c76e30 <conn_backend_get+496>: mov 0x4(%r12),%eax
0x00000af892c76e35 <conn_backend_get+501>: and $0x1,%eax
0x00000af892c76e38 <conn_backend_get+504>: lock subl
$0x1,0x14c(%r15,%rax,4)
0x00000af892c76e42 <conn_backend_get+514>: mov 0x60(%r15),%rax
0x00000af892c76e46 <conn_backend_get+518>: movslq %r13d,%rcx
0x00000af892c76e49 <conn_backend_get+521>: lock subl $0x1,(%rax,%rcx,4)
0x00000af892c76e4e <conn_backend_get+526>: andb $0xfc,0x4(%r12)
0x00000af892c76e54 <conn_backend_get+532>: mov 0xe0(%rbx),%rax
0x00000af892c76e5b <conn_backend_get+539>: mov 0x8(%rax),%eax
0x00000af892c76e5e <conn_backend_get+542>: and $0xc,%eax
0x00000af892c76e61 <conn_backend_get+545>: cmp $0x4,%eax
0x00000af892c76e64 <conn_backend_get+548>: jne 0xaf892c76e74
<conn_backend_get+564>
0x00000af892c76e66 <conn_backend_get+550>: mov 0x18(%r12),%rax
0x00000af892c76e6b <conn_backend_get+555>: testb $0x2,0xa8(%rax)
0x00000af892c76e72 <conn_backend_get+562>: jne 0xaf892c76ee1
<conn_backend_get+673>
0x00000af892c76e74 <conn_backend_get+564>: mov 0x58(%r15),%rbx
0x00000af892c76e78 <conn_backend_get+568>: mov 952569(%rip),%rdi
# 0xaf892d5f778 <_DYNAMIC+472>
0x00000af892c76e7f <conn_backend_get+575>: callq 0xaf892d4dba0
<__emutls_get_address>
0x00000af892c76e84 <conn_backend_get+580>: mov (%rax),%eax
0x00000af892c76e86 <conn_backend_get+582>: shl $0x6,%rax
0x00000af892c76e8a <conn_backend_get+586>: lea (%rbx,%rax,1),%rdi
0x00000af892c76e8e <conn_backend_get+590>: add $0x30,%rdi
0x00000af892c76e92 <conn_backend_get+594>: mov 0xb0(%r12),%rsi
0x00000af892c76e9a <conn_backend_get+602>: callq 0xaf892d448a0
<eb64_insert>
0x00000af892c76e9f <conn_backend_get+607>: jmpq 0xaf892c771a6
<conn_backend_get+1382>
0x00000af892c76ea4 <conn_backend_get+612>: mov 952485(%rip),%rax
# 0xaf892d5f750 <_DYNAMIC+432>
0x00000af892c76eab <conn_backend_get+619>: testb $0x10,0xfa(%rax)
0x00000af892c76eb2 <conn_backend_get+626>: je 0xaf892c76ed9
<conn_backend_get+665>
0x00000af892c76eb4 <conn_backend_get+628>: mov 0x148(%r15),%eax
0x00000af892c76ebb <conn_backend_get+635>: cmp 0x6c(%r15),%eax
0x00000af892c76ebf <conn_backend_get+639>: jae 0xaf892c76f32
<conn_backend_get+754>
0x00000af892c76ec1 <conn_backend_get+641>: mov 953352(%rip),%rax
# 0xaf892d5fad0 <_DYNAMIC+1328>
0x00000af892c76ec8 <conn_backend_get+648>: mov (%rax),%eax
0x00000af892c76eca <conn_backend_get+650>: mov 952447(%rip),%rcx
# 0xaf892d5f750 <_DYNAMIC+432>
0x00000af892c76ed1 <conn_backend_get+657>: cmp 0x148(%rcx),%eax
0x00000af892c76ed7 <conn_backend_get+663>: jge 0xaf892c76f32
<conn_backend_get+754>
0x00000af892c76ed9 <conn_backend_get+665>: xor %r12d,%r12d
0x00000af892c76edc <conn_backend_get+668>: jmpq 0xaf892c771a6
<conn_backend_get+1382>
0x00000af892c76ee1 <conn_backend_get+673>: mov 0x18(%rbx),%rax
0x00000af892c76ee5 <conn_backend_get+677>: mov %rax,0x30(%r12)
0x00000af892c76eea <conn_backend_get+682>: mov 0x18(%rbx),%r15
0x00000af892c76eee <conn_backend_get+686>: mov %r12,%rbx
0x00000af892c76ef1 <conn_backend_get+689>: mov 0x38(%r12),%r12
0x00000af892c76ef6 <conn_backend_get+694>: test %r12,%r12
0x00000af892c76ef9 <conn_backend_get+697>: je 0xaf892c7710b
<conn_backend_get+1227>
0x00000af892c76eff <conn_backend_get+703>: cmpb $0x1,(%r12)
0x00000af892c76f04 <conn_backend_get+708>: jne 0xaf892c7710b
<conn_backend_get+1227>
0x00000af892c76f0a <conn_backend_get+714>: lea -1510910(%rip),%rsi
# 0xaf892b06113 <__emutls_t.poll_events+25939>
0x00000af892c76f11 <conn_backend_get+721>: xor %edi,%edi
0x00000af892c76f13 <conn_backend_get+723>: mov $0x3,%edx
0x00000af892c76f18 <conn_backend_get+728>: callq 0xaf892cddb00 <complain>
0x00000af892c76f1d <conn_backend_get+733>: callq 0xaf892cdd5f0
<ha_backtrace_to_stderr>
---Type <return> to continue, or q <return> to quit---
0x00000af892c76f22 <conn_backend_get+738>: movb $0x0,0x1
0x00000af892c76f2a <conn_backend_get+746>: jmp 0xaf892c76f30
<conn_backend_get+752>
0x00000af892c76f2c <conn_backend_get+748>: int3
0x00000af892c76f2d <conn_backend_get+749>: int3
0x00000af892c76f2e <conn_backend_get+750>: int3
0x00000af892c76f2f <conn_backend_get+751>: int3
0x00000af892c76f30 <conn_backend_get+752>: jmp 0xaf892c76f30
<conn_backend_get+752>
0x00000af892c76f32 <conn_backend_get+754>: mov 0x160(%r15),%ecx
0x00000af892c76f39 <conn_backend_get+761>: xor %r12d,%r12d
0x00000af892c76f3c <conn_backend_get+764>: mov 952333(%rip),%rax
# 0xaf892d5f750 <_DYNAMIC+432>
0x00000af892c76f43 <conn_backend_get+771>: cmp 0xc(%rax),%ecx
0x00000af892c76f46 <conn_backend_get+774>: cmovge %r12d,%ecx
0x00000af892c76f4a <conn_backend_get+778>: mov
%ecx,0xffffffffffffffb4(%rbp)
0x00000af892c76f4d <conn_backend_get+781>: mov %ecx,%r13d
0x00000af892c76f50 <conn_backend_get+784>: jmp 0xaf892c76f9b
<conn_backend_get+859>
0x00000af892c76f52 <conn_backend_get+786>: int3
0x00000af892c76f53 <conn_backend_get+787>: int3
0x00000af892c76f54 <conn_backend_get+788>: int3
0x00000af892c76f55 <conn_backend_get+789>: int3
0x00000af892c76f56 <conn_backend_get+790>: int3
0x00000af892c76f57 <conn_backend_get+791>: int3
0x00000af892c76f58 <conn_backend_get+792>: int3
0x00000af892c76f59 <conn_backend_get+793>: int3
0x00000af892c76f5a <conn_backend_get+794>: int3
0x00000af892c76f5b <conn_backend_get+795>: int3
0x00000af892c76f5c <conn_backend_get+796>: int3
0x00000af892c76f5d <conn_backend_get+797>: int3
0x00000af892c76f5e <conn_backend_get+798>: int3
0x00000af892c76f5f <conn_backend_get+799>: int3
0x00000af892c76f60 <conn_backend_get+800>: cmpl
$0x0,0xffffffffffffffcc(%rbp)
0x00000af892c76f64 <conn_backend_get+804>: mov $0x0,%r12d
0x00000af892c76f6a <conn_backend_get+810>: je 0xaf892c77069
<conn_backend_get+1065>
0x00000af892c76f70 <conn_backend_get+816>: mov $0x100000004,%rax
0x00000af892c76f7a <conn_backend_get+826>: lock sub %rax,(%rbx)
0x00000af892c76f7e <conn_backend_get+830>: add $0x1,%r13d
0x00000af892c76f82 <conn_backend_get+834>: mov 952263(%rip),%rax
# 0xaf892d5f750 <_DYNAMIC+432>
0x00000af892c76f89 <conn_backend_get+841>: cmp 0xc(%rax),%r13d
0x00000af892c76f8d <conn_backend_get+845>: cmove %r12d,%r13d
0x00000af892c76f91 <conn_backend_get+849>: cmp
0xffffffffffffffb4(%rbp),%r13d
0x00000af892c76f95 <conn_backend_get+853>: je 0xaf892c771a6
<conn_backend_get+1382>
0x00000af892c76f9b <conn_backend_get+859>: mov 0x60(%r15),%rbx
0x00000af892c76f9f <conn_backend_get+863>: movslq %r13d,%r14
0x00000af892c76fa2 <conn_backend_get+866>: mov 952271(%rip),%rdi
# 0xaf892d5f778 <_DYNAMIC+472>
0x00000af892c76fa9 <conn_backend_get+873>: callq 0xaf892d4dba0
<__emutls_get_address>
0x00000af892c76fae <conn_backend_get+878>: cmpl $0x0,(%rbx,%r14,4)
0x00000af892c76fb3 <conn_backend_get+883>: je 0xaf892c76f7e
<conn_backend_get+830>
0x00000af892c76fb5 <conn_backend_get+885>: cmp (%rax),%r13d
0x00000af892c76fb8 <conn_backend_get+888>: mov 952441(%rip),%rcx
# 0xaf892d5f838 <_DYNAMIC+664>
0x00000af892c76fbf <conn_backend_get+895>: je 0xaf892c76f7e
<conn_backend_get+830>
0x00000af892c76fc1 <conn_backend_get+897>: shl $0x6,%r14
0x00000af892c76fc5 <conn_backend_get+901>: mov 0x18(%rcx,%r14,1),%rax
0x00000af892c76fca <conn_backend_get+906>: shr $0x20,%rax
0x00000af892c76fce <conn_backend_get+910>: jne 0xaf892c76f7e
<conn_backend_get+830>
0x00000af892c76fd0 <conn_backend_get+912>: lea (%rcx,%r14,1),%rbx
0x00000af892c76fd4 <conn_backend_get+916>: xchg %rbx,%rax
0x00000af892c76fd7 <conn_backend_get+919>: add $0x18,%rax
0x00000af892c76fdb <conn_backend_get+923>: xchg %rbx,%rax
0x00000af892c76fde <conn_backend_get+926>: mov $0x100000004,%rax
---Type <return> to continue, or q <return> to quit---
0x00000af892c76fe8 <conn_backend_get+936>: lock xadd %rax,(%rbx)
0x00000af892c76fed <conn_backend_get+941>: shr $0x20,%rax
0x00000af892c76ff1 <conn_backend_get+945>: jne 0xaf892c770f8
<conn_backend_get+1208>
0x00000af892c76ff7 <conn_backend_get+951>: xor %eax,%eax
0x00000af892c76ff9 <conn_backend_get+953>: cmpl
$0x0,0xffffffffffffffcc(%rbp)
0x00000af892c76ffd <conn_backend_get+957>: setne %al
0x00000af892c77000 <conn_backend_get+960>: mov 0x58(%r15),%rcx
0x00000af892c77004 <conn_backend_get+964>: add %r14,%rcx
0x00000af892c77007 <conn_backend_get+967>: shl $0x4,%rax
0x00000af892c7700b <conn_backend_get+971>: lea (%rax,%rcx,1),%rdi
0x00000af892c7700f <conn_backend_get+975>: add $0x10,%rdi
0x00000af892c77013 <conn_backend_get+979>: mov
0xffffffffffffffc0(%rbp),%rsi
0x00000af892c77017 <conn_backend_get+983>: callq 0xaf892c24e00
<srv_lookup_conn>
0x00000af892c7701c <conn_backend_get+988>: test %rax,%rax
0x00000af892c7701f <conn_backend_get+991>: je 0xaf892c76f60
<conn_backend_get+800>
0x00000af892c77025 <conn_backend_get+997>: mov %rax,%r12
0x00000af892c77028 <conn_backend_get+1000>: jmp 0xaf892c77044
<conn_backend_get+1028>
0x00000af892c7702a <conn_backend_get+1002>: int3
0x00000af892c7702b <conn_backend_get+1003>: int3
0x00000af892c7702c <conn_backend_get+1004>: int3
0x00000af892c7702d <conn_backend_get+1005>: int3
0x00000af892c7702e <conn_backend_get+1006>: int3
0x00000af892c7702f <conn_backend_get+1007>: int3
0x00000af892c77030 <conn_backend_get+1008>: mov %r12,%rdi
0x00000af892c77033 <conn_backend_get+1011>: callq 0xaf892c24e40
<srv_lookup_conn_next>
0x00000af892c77038 <conn_backend_get+1016>: mov %rax,%r12
0x00000af892c7703b <conn_backend_get+1019>: test %rax,%rax
0x00000af892c7703e <conn_backend_get+1022>: je 0xaf892c76f60
<conn_backend_get+800>
0x00000af892c77044 <conn_backend_get+1028>: mov 0x18(%r12),%rax
0x00000af892c77049 <conn_backend_get+1033>: mov 0xa0(%rax),%r11
0x00000af892c77050 <conn_backend_get+1040>: test %r11,%r11
0x00000af892c77053 <conn_backend_get+1043>: je 0xaf892c77030
<conn_backend_get+1008>
0x00000af892c77055 <conn_backend_get+1045>: mov %r12,%rdi
0x00000af892c77058 <conn_backend_get+1048>: mov %r13d,%esi
0x00000af892c7705b <conn_backend_get+1051>: callq 0xaf892b83050
<__llvm_retpoline_r11>
0x00000af892c77060 <conn_backend_get+1056>: test %eax,%eax
0x00000af892c77062 <conn_backend_get+1058>: jne 0xaf892c77030
<conn_backend_get+1008>
0x00000af892c77064 <conn_backend_get+1060>: jmpq 0xaf892c771d0
<conn_backend_get+1424>
0x00000af892c77069 <conn_backend_get+1065>: movl
$0x0,0xffffffffffffffcc(%rbp)
0x00000af892c77070 <conn_backend_get+1072>: cmpl $0x0,0x150(%r15)
0x00000af892c77078 <conn_backend_get+1080>: je 0xaf892c76f70
<conn_backend_get+816>
0x00000af892c7707e <conn_backend_get+1086>: mov 0x58(%r15),%rax
0x00000af892c77082 <conn_backend_get+1090>: lea (%rax,%r14,1),%rdi
0x00000af892c77086 <conn_backend_get+1094>: add $0x20,%rdi
0x00000af892c7708a <conn_backend_get+1098>: mov
0xffffffffffffffc0(%rbp),%rsi
0x00000af892c7708e <conn_backend_get+1102>: callq 0xaf892c24e00
<srv_lookup_conn>
0x00000af892c77093 <conn_backend_get+1107>: test %rax,%rax
0x00000af892c77096 <conn_backend_get+1110>: je 0xaf892c770f0
<conn_backend_get+1200>
0x00000af892c77098 <conn_backend_get+1112>: mov %rax,%r12
0x00000af892c7709b <conn_backend_get+1115>: mov %rbx,%r14
0x00000af892c7709e <conn_backend_get+1118>: mov
0xffffffffffffffb8(%rbp),%rbx
0x00000af892c770a2 <conn_backend_get+1122>: jmp 0xaf892c770c0
<conn_backend_get+1152>
0x00000af892c770a4 <conn_backend_get+1124>: int3
0x00000af892c770a5 <conn_backend_get+1125>: int3
0x00000af892c770a6 <conn_backend_get+1126>: int3
0x00000af892c770a7 <conn_backend_get+1127>: int3
0x00000af892c770a8 <conn_backend_get+1128>: int3
0x00000af892c770a9 <conn_backend_get+1129>: int3
---Type <return> to continue, or q <return> to quit---
0x00000af892c770aa <conn_backend_get+1130>: int3
0x00000af892c770ab <conn_backend_get+1131>: int3
0x00000af892c770ac <conn_backend_get+1132>: int3
0x00000af892c770ad <conn_backend_get+1133>: int3
0x00000af892c770ae <conn_backend_get+1134>: int3
0x00000af892c770af <conn_backend_get+1135>: int3
0x00000af892c770b0 <conn_backend_get+1136>: mov %r12,%rdi
0x00000af892c770b3 <conn_backend_get+1139>: callq 0xaf892c24e40
<srv_lookup_conn_next>
0x00000af892c770b8 <conn_backend_get+1144>: mov %rax,%r12
0x00000af892c770bb <conn_backend_get+1147>: test %rax,%rax
0x00000af892c770be <conn_backend_get+1150>: je 0xaf892c770e5
<conn_backend_get+1189>
0x00000af892c770c0 <conn_backend_get+1152>: mov 0x18(%r12),%rax
0x00000af892c770c5 <conn_backend_get+1157>: mov 0xa0(%rax),%r11
0x00000af892c770cc <conn_backend_get+1164>: test %r11,%r11
0x00000af892c770cf <conn_backend_get+1167>: je 0xaf892c770b0
<conn_backend_get+1136>
0x00000af892c770d1 <conn_backend_get+1169>: mov %r12,%rdi
0x00000af892c770d4 <conn_backend_get+1172>: mov %r13d,%esi
0x00000af892c770d7 <conn_backend_get+1175>: callq 0xaf892b83050
<__llvm_retpoline_r11>
0x00000af892c770dc <conn_backend_get+1180>: test %eax,%eax
0x00000af892c770de <conn_backend_get+1182>: jne 0xaf892c770b0
<conn_backend_get+1136>
0x00000af892c770e0 <conn_backend_get+1184>: jmpq 0xaf892c771d7
<conn_backend_get+1431>
0x00000af892c770e5 <conn_backend_get+1189>: xor %r12d,%r12d
0x00000af892c770e8 <conn_backend_get+1192>: mov %r14,%rbx
0x00000af892c770eb <conn_backend_get+1195>: jmpq 0xaf892c76f70
<conn_backend_get+816>
0x00000af892c770f0 <conn_backend_get+1200>: xor %r12d,%r12d
0x00000af892c770f3 <conn_backend_get+1203>: jmpq 0xaf892c76f70
<conn_backend_get+816>
0x00000af892c770f8 <conn_backend_get+1208>: mov $0x100000004,%rax
0x00000af892c77102 <conn_backend_get+1218>: lock sub %rax,(%rbx)
0x00000af892c77106 <conn_backend_get+1222>: jmpq 0xaf892c76f7e
<conn_backend_get+830>
0x00000af892c7710b <conn_backend_get+1227>: mov %rbx,%rcx
0x00000af892c7710e <conn_backend_get+1230>: lea 0x58(%rbx),%r14
0x00000af892c77112 <conn_backend_get+1234>: cmp %r14,0x58(%rbx)
0x00000af892c77116 <conn_backend_get+1238>: jne 0xaf892c771a3
<conn_backend_get+1379>
0x00000af892c7711c <conn_backend_get+1244>: test %rax,%rax
0x00000af892c7711f <conn_backend_get+1247>: je 0xaf892c77126
<conn_backend_get+1254>
0x00000af892c77121 <conn_backend_get+1249>: cmp %r15,%rax
0x00000af892c77124 <conn_backend_get+1252>: jne 0xaf892c771a3
<conn_backend_get+1379>
0x00000af892c77126 <conn_backend_get+1254>: lea 0xa8(%r15),%r13
0x00000af892c7712d <conn_backend_get+1261>: mov 0xa8(%r15),%rax
0x00000af892c77134 <conn_backend_get+1268>: cmp %rax,%r13
0x00000af892c77137 <conn_backend_get+1271>: je 0xaf892c77152
<conn_backend_get+1298>
0x00000af892c77139 <conn_backend_get+1273>: jmp 0xaf892c77140
<conn_backend_get+1280>
0x00000af892c7713b <conn_backend_get+1275>: int3
0x00000af892c7713c <conn_backend_get+1276>: int3
0x00000af892c7713d <conn_backend_get+1277>: int3
0x00000af892c7713e <conn_backend_get+1278>: int3
0x00000af892c7713f <conn_backend_get+1279>: int3
0x00000af892c77140 <conn_backend_get+1280>: cmp
%r12,0xffffffffffffffe8(%rax)
0x00000af892c77144 <conn_backend_get+1284>: je 0xaf892c77216
<conn_backend_get+1494>
0x00000af892c7714a <conn_backend_get+1290>: mov (%rax),%rax
0x00000af892c7714d <conn_backend_get+1293>: cmp %rax,%r13
0x00000af892c77150 <conn_backend_get+1296>: jne 0xaf892c77140
<conn_backend_get+1280>
0x00000af892c77152 <conn_backend_get+1298>: mov 952295(%rip),%rax
# 0xaf892d5f940 <_DYNAMIC+928>
0x00000af892c77159 <conn_backend_get+1305>: mov (%rax),%rdi
0x00000af892c7715c <conn_backend_get+1308>: xor %esi,%esi
0x00000af892c7715e <conn_backend_get+1310>: callq 0xaf892d05020
<__pool_alloc>
0x00000af892c77163 <conn_backend_get+1315>: test %rax,%rax
0x00000af892c77166 <conn_backend_get+1318>: je 0xaf892c771a3
<conn_backend_get+1379>
---Type <return> to continue, or q <return> to quit---
0x00000af892c77168 <conn_backend_get+1320>: mov %r12,(%rax)
0x00000af892c7716b <conn_backend_get+1323>: lea 0x8(%rax),%rcx
0x00000af892c7716f <conn_backend_get+1327>: mov %rcx,0x10(%rax)
0x00000af892c77173 <conn_backend_get+1331>: mov %rcx,0x8(%rax)
0x00000af892c77177 <conn_backend_get+1335>: mov 0xb0(%r15),%rcx
0x00000af892c7717e <conn_backend_get+1342>: mov %rax,%rdx
0x00000af892c77181 <conn_backend_get+1345>: xchg %rdx,%rax
0x00000af892c77184 <conn_backend_get+1348>: add $0x18,%rax
0x00000af892c77188 <conn_backend_get+1352>: xchg %rdx,%rax
0x00000af892c7718b <conn_backend_get+1355>: mov %rcx,0x20(%rax)
0x00000af892c7718f <conn_backend_get+1359>: mov %rdx,0xb0(%r15)
0x00000af892c77196 <conn_backend_get+1366>: mov 0x20(%rax),%rcx
0x00000af892c7719a <conn_backend_get+1370>: mov %rdx,(%rcx)
0x00000af892c7719d <conn_backend_get+1373>: mov %r13,0x18(%rax)
0x00000af892c771a1 <conn_backend_get+1377>: jmp 0xaf892c7721a
<conn_backend_get+1498>
0x00000af892c771a3 <conn_backend_get+1379>: mov %rbx,%r12
0x00000af892c771a6 <conn_backend_get+1382>: mov %r12,%rax
0x00000af892c771a9 <conn_backend_get+1385>: add $0x20,%rsp
0x00000af892c771ad <conn_backend_get+1389>: pop %rbx
0x00000af892c771ae <conn_backend_get+1390>: pop %r12
0x00000af892c771b0 <conn_backend_get+1392>: pop %r13
0x00000af892c771b2 <conn_backend_get+1394>: pop %r14
0x00000af892c771b4 <conn_backend_get+1396>: pop %r15
0x00000af892c771b6 <conn_backend_get+1398>: pop %r11
0x00000af892c771b8 <conn_backend_get+1400>: pop %rbp
0x00000af892c771b9 <conn_backend_get+1401>: xor (%rsp),%r11
0x00000af892c771bd <conn_backend_get+1405>: cmp 899508(%rip),%r11
# 0xaf892d52b78 <__retguard_600>
0x00000af892c771c4 <conn_backend_get+1412>: je 0xaf892c771cf
<conn_backend_get+1423>
0x00000af892c771c6 <conn_backend_get+1414>: int3
0x00000af892c771c7 <conn_backend_get+1415>: int3
0x00000af892c771c8 <conn_backend_get+1416>: int3
0x00000af892c771c9 <conn_backend_get+1417>: int3
0x00000af892c771ca <conn_backend_get+1418>: int3
0x00000af892c771cb <conn_backend_get+1419>: int3
0x00000af892c771cc <conn_backend_get+1420>: int3
0x00000af892c771cd <conn_backend_get+1421>: int3
0x00000af892c771ce <conn_backend_get+1422>: int3
0x00000af892c771cf <conn_backend_get+1423>: retq
0x00000af892c771d0 <conn_backend_get+1424>: mov %rbx,%r14
0x00000af892c771d3 <conn_backend_get+1427>: mov
0xffffffffffffffb8(%rbp),%rbx
0x00000af892c771d7 <conn_backend_get+1431>: mov 0xb0(%r12),%rdi
0x00000af892c771df <conn_backend_get+1439>: callq 0xaf892c89cb0
<conn_delete_from_tree>
0x00000af892c771e4 <conn_backend_get+1444>: mov 951693(%rip),%rdi
# 0xaf892d5f778 <_DYNAMIC+472>
0x00000af892c771eb <conn_backend_get+1451>: callq 0xaf892d4dba0
<__emutls_get_address>
0x00000af892c771f0 <conn_backend_get+1456>: mov (%rax),%eax
0x00000af892c771f2 <conn_backend_get+1458>: shl $0x7,%rax
0x00000af892c771f6 <conn_backend_get+1462>: mov 951683(%rip),%rcx
# 0xaf892d5f780 <_DYNAMIC+480>
0x00000af892c771fd <conn_backend_get+1469>: lock addl $0x1,0x38(%rcx,%rax,1)
0x00000af892c77203 <conn_backend_get+1475>: mov $0x100000004,%rax
0x00000af892c7720d <conn_backend_get+1485>: lock sub %rax,(%r14)
0x00000af892c77211 <conn_backend_get+1489>: jmpq 0xaf892c76dd8
<conn_backend_get+408>
0x00000af892c77216 <conn_backend_get+1494>: add $0xffffffffffffffe8,%rax
0x00000af892c7721a <conn_backend_get+1498>: lea 0x8(%rax),%rcx
0x00000af892c7721e <conn_backend_get+1502>: mov 0x10(%rax),%rdx
0x00000af892c77222 <conn_backend_get+1506>: mov %rbx,%r12
0x00000af892c77225 <conn_backend_get+1509>: mov %rdx,0x60(%rbx)
0x00000af892c77229 <conn_backend_get+1513>: mov %r14,0x10(%rax)
0x00000af892c7722d <conn_backend_get+1517>: mov 0x60(%rbx),%rax
---Type <return> to continue, or q <return> to quit---
0x00000af892c77231 <conn_backend_get+1521>: mov %r14,(%rax)
0x00000af892c77234 <conn_backend_get+1524>: mov %rcx,0x58(%rbx)
0x00000af892c77238 <conn_backend_get+1528>: jmpq 0xaf892c771a6
<conn_backend_get+1382>
End of assembler dump.
(gdb)
