Re: Question on TCP reassembly counter
On 10/01/10 22:20, Andre Oppermann wrote: > On 01.10.2010 12:01, Sriram Gorti wrote: >> Hi, >> >> In the following is an observation when testing our XLR/XLS network >> driver with 16 concurrent instances of netperf on FreeBSD-CURRENT. >> Based on this observation, I have a question on which I hope to get >> some understanding from here. >> >> When running 16 concurrent netperf instances (each for about 20 >> seconds), it was found that after some number of runs performance >> degraded badly (almost by a factor of 5). All subsequent runs remained >> so. Started debugging this from TCP-side as other driver tests were >> doing fine for comparably long durations on same board+s/w. >> >> netstat indicated the following: >> >> $ netstat -s -f inet -p tcp | grep discarded >> 0 discarded for bad checksums >> 0 discarded for bad header offset fields >> 0 discarded because packet too short >> 7318 discarded due to memory problems >> >> Then, traced the "discarded due to memory problems" to the following >> counter: >> >> $ sysctl -a net.inet.tcp.reass >> net.inet.tcp.reass.overflows: 7318 >> net.inet.tcp.reass.maxqlen: 48 >> net.inet.tcp.reass.cursegments: 1594<--- // corresponds to >> V_tcp_reass_qsize variable >> net.inet.tcp.reass.maxsegments: 1600 >> >> Our guess for the need for reassembly (in this low-packet-loss test >> setup) was the lack of per-flow classification in the driver, causing >> it to spew incoming packets across the 16 h/w cpus instead of packets >> of a flow being sent to the same cpu. While we are working on >> addressing this driver limitation, debugged further to see how/why the >> V_tcp_reass_qsize grew (assuming that out-of-order segments should >> have dropped to zero at the end of the run). It was seen that this >> counter was actually growing up from the initial runs but only when it >> reached near to maxsgements, perf degradation was seen. Then, started >> looking at vmstat also to see how many of the reassembly segments were >> lost. But, there were no segments lost. We could not reconcile "no >> lost segments" with "growth of this counter across test runs". > > A patch is in the works to properly autoscale the reassembly queue > and should be comitted shortly. > >> $ sysctl net.inet.tcp.reass ; vmstat -z | egrep "FREE|mbuf|tcpre" >> net.inet.tcp.reass.overflows: 0 >> net.inet.tcp.reass.maxqlen: 48 >> net.inet.tcp.reass.cursegments: 147 >> net.inet.tcp.reass.maxsegments: 1600 >> ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP >> mbuf_packet:256, 0,4096,3200, 5653833, 0, 0 >> mbuf: 256, 0, 1,2048, 4766910, 0, 0 >> mbuf_cluster: 2048, 25600,7296, 6,7297, 0, 0 >> mbuf_jumbo_page: 4096, 12800, 0, 0, 0, 0, 0 >> mbuf_jumbo_9k: 9216, 6400, 0, 0, 0, 0, 0 >> mbuf_jumbo_16k: 16384, 3200, 0, 0, 0, 0, 0 >> mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0 >> tcpreass:20, 1690, 0, 845, 1757074, 0, 0 >> >> In view of these observations, my question is: is it possible for the >> V_tcp_reass_qsize variable to be unsafely updated on SMP ? (The >> particular flavor of XLS that was used in the test had 4 cores with 4 >> h/w threads/core). I see that the tcp_reass function assumes some lock >> is taken but not sure if it is the per-socket or the global tcp lock. > > The updating of the global counter is indeed unsafe and becomes obsolete > with the autotuning patch. > > The patch is reviewed by me and ready for commit. However lstewart@ is > currently writing his thesis and has only very little spare time. I'll > send you the patch in private email so you can continue your testing. Quick update on this: patch is blocked while waiting for Jeff to review some related UMA changes. As soon as I get the all clear I'll push everything into head. Cheers, Lawrence ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Current problem reports assigned to freebsd-net@FreeBSD.org
Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description o kern/150920 net[ixgbe][igb] Panic when packets are dropped with heade o kern/150557 net[igb] igb0: Watchdog timeout -- resetting o kern/150257 net[msk] watchdog timeout o kern/150251 net[patch] [ixgbe] Late cable insertion broken o kern/150249 net[ixgbe] Media type detection broken o kern/150247 net[patch] [ixgbe] Version in -current won't build on 7.x o bin/150224 netppp(8) does not reassign static IP after kill -KILL co o kern/150148 net[ath] Atheros 5424/2424 - AR2425 stopped working with o kern/150052 net[wi] wi(4) driver does not work with wlan(4) driver fo f kern/149969 net[wlan] [ral] ralink rt2661 fails to maintain connectio o kern/149937 net[ipfilter] [patch] kernel panic in ipfilter IP fragmen o kern/149786 net[bwn] bwn on Dell Inspiron 1150: connections stall o kern/149643 net[rum] device not sending proper beacon frames in ap mo o kern/149609 net[panic] reboot after adding second default route o kern/149539 net[ath] atheros ar9287 is not supported by ath_hal o kern/149516 net[ath] ath(4) hostap with fake MAC/BSSID results in sta o kern/149373 net[realtek/atheros]: None of my network card working o kern/149307 net[ath] Doesn't work Atheros 9285 o kern/149306 net[alc] Doesn't work Atheros AR8131 PCIe Gigabit Etherne o kern/149117 net[inet] [patch] in_pcbbind: redundant test o kern/149086 net[multicast] Generic multicast join failure in 8.1 o kern/148862 net[panic] page fault while in kernel mode at _mtx_lock_s o kern/148322 net[ath] Triggering atheros wifi beacon misses in hostap o kern/148317 net[ath] FreeBSD 7.x hostap memory leak in net80211 or At o kern/148078 net[ath] wireless networking stops functioning o kern/147985 net[alc] alc network driver + tso ( + vlan ? ) does not w o kern/147894 net[ipsec] IPv6-in-IPv4 does not work inside an ESP-only o kern/147862 net[wpi] Possible bug in the wpi driver. Network Manager o kern/147155 net[ip6] setfb not work with ipv6 o kern/146845 net[libc] close(2) returns error 54 (connection reset by o kern/146792 net[flowtable] flowcleaner 100% cpu's core load o kern/146759 net[cxgb] [patch] cxgb panic calling cxgb_set_lro() witho o kern/146719 net[pf] [panic] PF or dumynet kernel panic o kern/146534 net[icmp6] wrong source address in echo reply o kern/146517 net[ath] [wlan] device timeouts for ath wlan device on re o kern/146427 net[mwl] Additional virtual access points don't work on m o kern/146426 net[mwl] 802.11n rates not possible on mwl o kern/146425 net[mwl] mwl dropping all packets during and after high u f kern/146394 net[vlan] IP source address for outgoing connections o bin/146377 net[ppp] [tun] Interface doesn't clear addresses when PPP o kern/146358 net[vlan] wrong destination MAC address o kern/146165 net[wlan] [panic] Setting bssid in adhoc mode causes pani o kern/146082 net[ng_l2tp] a false invaliant check was performed in ng_ o kern/146037 net[panic] mpd + CoA = kernel panic o bin/145934 net[patch] add count option to netstat(1) o kern/145826 net[ath] Unable to configure adhoc mode on ath0/wlan0 o kern/145825 net[panic] panic: soabort: so_count o kern/145777 net[wpi] Intel 3945ABG driver breaks the connection after o kern/145728 net[lagg] Stops working lagg between two servers. o amd64/145654 netamd64-curent memory leak in kernel o kern/144987 net[wpi] [panic] injecting packets with wlaninject using o kern/144882 netMacBookPro =>4.1 does not connect to BSD in hostap wit o kern/144874 net[if_bridge] [patch] if_bridge frees mbuf after pfil ho o conf/144700 net[rc.d] async dhclient breaks stuff for too many people o kern/144642 net[rum] [panic] Enabling rum interface causes panic o kern/144616 net[nat] [panic] ip_nat panic FreeBSD 7.2 o kern/144572 net[carp] CARP preemption mode traffic partially goes to f kern/144315 net[ipfw] [panic] freebsd 8-stable reboot after add ipfw o kern/143939 net[ipfw] [em] ipfw nat and em interface rxcsum problem o kern/143874 net[wpi] Wireless 3945ABG error. wpi0 could not allocate o kern/143868 net[ath] [patch] [request] allow Atheros watchdog timeout o kern/143846
