[Bug 279875] sockstat: segmentation fault
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=279875 Konstantin Belousov changed: What|Removed |Added CC||k...@freebsd.org --- Comment #2 from Konstantin Belousov --- Rebuild sockstat without optimization, like this: # make -C /usr/src/usr.bin/sockstat DEBUG_FLAGS=-g clean all install then get the backtrace from gdb. -- You are receiving this mail because: You are the assignee for the bug.
Re: TCP Success Story (was Re: TCP_RACK, TCP_BBR, and firewalls)
> On 19. Jul 2024, at 05:07, Junho Choi wrote: > > RACK is a loss detection algorithm and BBR is a congestion control algorithm > so it's on a different layer. > e.g. linux can configure them independently. > > However in FreeBSD it looks like it is using the same configuration sysctl > (net.inet.tcp.functions_default=tcp_rack|tcp_bbr), > so not able to set it both. FreeBSD supports multiple TCP stacks. One stack has the name RACK and does RACK loss discovery, but much more like pacing, advanced LRO and so on. You can configure CC modules like newreno, cubic or so to be used. Another TCP stack is called BBR. It does BBRv1 congestion control and other things. > > Is there any plan to improve it? or does tcp_bbr include tcp_rack's loss > probe behavior? I am not aware of a plan to improve the BBR stack right now. Best regards Michael > > A little confused. > > Best, > > > On Fri, Jul 19, 2024 at 4:23 AM wrote: >> On 18. Jul 2024, at 20:37, Alan Somers wrote: >> >> Coexist how? Do you mean that one socket can use one and a different >> socket uses the other? That makes sense. > Correct. > > Best regards > Michael >> >> On Thu, Jul 18, 2024 at 10:34 AM wrote: >>> On 18. Jul 2024, at 15:00, Junho Choi wrote: Alan - this is a great result to see. Thanks for experimenting. Just curious why bbr and rack don't co-exist? Those are two separate things. Is it a current bug or by design? >>> Technically RACK and BBR can coexist. The problem was with pf and/or LRO. >>> >>> But this is all fixed now in 14.1 and head. >>> >>> Best regards >>> Michael BR, On Thu, Jul 18, 2024 at 5:27 AM wrote: > On 17. Jul 2024, at 22:00, Alan Somers wrote: > > On Sat, Jul 13, 2024 at 1:50 AM wrote: >> >>> On 13. Jul 2024, at 01:43, Alan Somers wrote: >>> >>> I've been experimenting with RACK and BBR. In my environment, they >>> can dramatically improve single-stream TCP performance, which is >>> awesome. But pf interferes. I have to disable pf in order for them >>> to work at all. >>> >>> Is this a known limitation? If not, I will experiment some more to >>> determine exactly what aspect of my pf configuration is responsible. >>> If so, can anybody suggest what changes would have to happen to make >>> the two compatible? >> A problem with same symptoms was already reported and fixed in >> https://reviews.freebsd.org/D43769 >> >> Which version are you using? >> >> Best regards >> Michael >>> >>> -Alan > > TLDR; tcp_rack is good, cc_chd is better, and tcp_bbr is best > > I want to follow up with the list to post my conclusions. Firstly > tuexen@ helped me solve my problem: in FreeBSD 14.0 there is a 3-way > incompatibility between (tcp_bbr || tcp_rack) && lro && pf. I can > confirm that tcp_bbr works for me if I either disable LRO, disable PF, > or switch to a 14.1 server. > > Here's the real problem: on multiple production servers, downloading > large files (or ZFS send/recv streams) was slow. After ruling out > many possible causes, wireshark revealed that the connection was > suffering about 0.05% packet loss. I don't know the source of that > packet loss, but I don't believe it to be congestion-related. Along > with a 54ms RTT, that's a fatal combination for the throughput of > loss-based congestion control algorithms. According to the Mathis > Formula [1], I could only expect 1.1 MBps over such a connection. > That's actually worse than what I saw. With default settings > (cc_cubic), I averaged 5.6 MBps. Probably Mathis's assumptions are > outdated, but that's still pretty close for such a simple formula > that's 27 years old. > > So I benchmarked all available congestion control algorithms for > single download streams. The results are summarized in the table > below. > > AlgoPacket Loss RateAverage Throughput > vegas 0.05% 2.0 MBps > newreno 0.05% 3.2 MBps > cubic 0.05% 5.6 MBps > hd 0.05% 8.6 MBps > cdg 0.05% 13.5 MBps > rack0.04% 14 MBps > htcp0.05% 15 MBps > dctcp 0.05% 15 MBps > chd 0.05% 17.3 MBps > bbr 0.05% 29.2 MBps > cubic 10% 159 kBps > chd 10% 208 kBps > bbr 10% 5.7 MBps > > RACK seemed to achieve about the same maximum bandwidth as BBR, though > it took a lot longer to get there. Also, with RACK, wireshark > reported about 10x as many retransmissions as dropped packets, which > is suspicious. > > At one point, something went haywire and packet loss briefly spiked to > the neighborhood of 10%.
[Bug 279875] sockstat: segmentation fault
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=279875 --- Comment #3 from John Marshall --- (In reply to Konstantin Belousov from comment #2) Thank you. - Re-built and installed sockstat as per your instruction. - Built and installed devel/gdb rwsrv08# gdb sockstat GNU gdb (GDB) 14.1 [GDB v14.1 for FreeBSD] ... Reading symbols from sockstat... Reading symbols from /build/usr/lib/debug//usr/bin/sockstat.debug... (gdb) r Starting program: /usr/bin/sockstat [Detaching after fork from child process 85890] USER COMMANDPID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS root sockstat 85895 6 stream -> [85889 8] root sockstat 85894 6 stream -> [85889 7] ... root syslogd 2948 9 dgram /var/run/logpriv root gssd2810 3 stream /var/run/gssd.sock Program received signal SIGSEGV, Segmentation fault. Address not mapped to object. files_t_RB_FIND (head=, elm=) at /kits/src/usr.bin/sockstat/sockstat.c:181 181 RB_GENERATE_STATIC(files_t, file, file_tree, file_compare); (gdb) bt #0 files_t_RB_FIND (head=, elm=) at /kits/src/usr.bin/sockstat/sockstat.c:181 #1 displaysock (s=s@entry=0x80184b440, pos=, pos@entry=30) at /kits/src/usr.bin/sockstat/sockstat.c:1165 #2 0x0102671f in display () at /kits/src/usr.bin/sockstat/sockstat.c:1345 #3 0x01025c07 in main (argc=, argv=) at /kits/src/usr.bin/sockstat/sockstat.c:1577 (gdb) -- You are receiving this mail because: You are the assignee for the bug.
[Bug 279875] sockstat: segmentation fault
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=279875 --- Comment #4 from John Marshall --- (In reply to Konstantin Belousov from comment #2) Sorry. That re-compile still had -O2. I found that I was able to get it to compile with -O0 by adding 'CFLAGS=-O0' to /etc/make.conf. Is there a way to tell make(1) to use -O0 just for 'this' Makefile? Here is a more useful backtrace I hope. rwsrv08# rwsrv08# gdb sockstat GNU gdb (GDB) 14.1 [GDB v14.1 for FreeBSD] ... Reading symbols from sockstat... Reading symbols from /build/usr/lib/debug//usr/bin/sockstat.debug... (gdb) r Starting program: /usr/bin/sockstat [Detaching after fork from child process 88560] USER COMMANDPID FD PROTO LOCAL ADDRESS FOREIGN ADDRESS root sockstat 88565 6 stream -> [88559 8] root sockstat 88564 6 stream -> [88559 7] ... root syslogd 2948 9 dgram /var/run/logpriv root gssd2810 3 stream /var/run/gssd.sock Program received signal SIGSEGV, Segmentation fault. Address not mapped to object. 0x01029669 in displaysock (s=0x80184b200, pos=59) at /kits/src/usr.bin/sockstat/sockstat.c:1169 1169(u_long)f->xf_pid, f->xf_fd); (gdb) bt #0 0x01029669 in displaysock (s=0x80184b200, pos=59) at /kits/src/usr.bin/sockstat/sockstat.c:1169 #1 0x0102776b in display () at /kits/src/usr.bin/sockstat/sockstat.c:1345 #2 0x01024eab in main (argc=0, argv=0x7fffeb60) at /kits/src/usr.bin/sockstat/sockstat.c:1577 (gdb) -- You are receiving this mail because: You are the assignee for the bug.
[Bug 279875] sockstat: segmentation fault
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=279875
--- Comment #5 from Konstantin Belousov ---
Do you have set something that prevents visibility of processes/jails?
Like see_other_uids etc?
Anyway, please apply this debugging patch and report if the program still
SIGSEGVs with it.
diff --git a/usr.bin/sockstat/sockstat.c b/usr.bin/sockstat/sockstat.c
index 73b1f00a4481..20a0a5e65e0a 100644
--- a/usr.bin/sockstat/sockstat.c
+++ b/usr.bin/sockstat/sockstat.c
@@ -1164,6 +1164,7 @@ displaysock(struct sock *s, int pos)
f = RB_FIND(files_t, &ftree,
&(struct file){ .xf_data =
p->socket });
+ if (f != NULL)
pos += xprintf("[%lu %d]",
(u_long)f->xf_pid, f->xf_fd);
} else
@@ -1183,6 +1184,7 @@ displaysock(struct sock *s, int pos)
f = RB_FIND(files_t, &ftree,
&(struct file){ .xf_data =
p->socket });
+ if (f != NULL);
pos += xprintf("%s[%lu %d]",
fref ? "" : ",",
(u_long)f->xf_pid, f->xf_fd);
--
You are receiving this mail because:
You are the assignee for the bug.
[Bug 279875] sockstat: segmentation fault
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=279875 --- Comment #6 from John Marshall --- (In reply to Konstantin Belousov from comment #5) That debugging patch stops the segfault in my case. Program runs to completion. Yes, I have security restrictions in place. My sysctl.conf includes the following. -- kern.securelevel=0 # init(8) will raise this to 1 going multi-user. security.bsd.see_other_uids=0 security.bsd.see_other_gids=0 security.bsd.unprivileged_proc_debug=0 security.bsd.unprivileged_read_msgbuf=0 -- Thank you. -- You are receiving this mail because: You are the assignee for the bug.
[Bug 279875] sockstat: segmentation fault
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=279875 --- Comment #7 from Konstantin Belousov --- https://reviews.freebsd.org/D46050 -- You are receiving this mail because: You are the assignee for the bug.
