How can I see if there is an active arp queue? Regarding the qdisc, I don't think we're bumping up against that (at least not in my tiny bench setup):
tc -s qdisc show qdisc fq_codel 0: dev eth0 root refcnt 2 limit 10240p flows 1024 quantum 1514 target 5.0ms interval 100.0ms ecn Sent 925035443 bytes 8988011 pkt (dropped 0, overlimits 0 requeues 3) backlog 0b 0p requeues 3 maxpacket 717 drop_overlimit 0 new_flow_count 1004 ecn_mark 0 new_flows_len 0 old_flows_len 0 I'm not sure I still 100% understand the relationship between the socket buffer (skb / wmem_default sysctl setting or SO_SNDBUF socket option), arp queue (arp_queue), and the unres_qlen_bytes sysctl setting. I've made a public google spreadsheet here to try and calculate this value based on some input and assumptions. Can you take a look and see if I got this somewhat correct? https://docs.google.com/spreadsheets/d/1t9_UowY6sok8xvK8Tx_La_jB4iqpewJT5X4WANj39gg/edit?usp=sharing On Thu, May 16, 2019 at 1:03 PM Eric Dumazet <eric.duma...@gmail.com> wrote: > > > > On 5/16/19 9:32 AM, Adam Urban wrote: > > Eric, thanks. Increasing wmem_default from 229376 to 2293760 indeed > > makes the issue go away on my test bench. What's a good way to > > determine the optimal value here? I assume this is in bytes and needs > > to be large enough so that the SO_SNDBUF doesn't fill up before the > > kernel drops the packets. How often does that happen? > > You have to count the max number of arp queues your UDP socket could hit. > > Say this number is X > > Then wmem_default should be set to X * unres_qlen_bytes + Y > > With Y = 229376 (the default wmem_default) > > Then, you might need to increase the qdisc limits. > > If no arp queue is active, all UDP packets could be in the qdisc and might > hit sooner > the qdisc limit, thus dropping packets on the qdisc. > > (This is assuming your UDP application can blast packets at a rate above the > link rate) > > > > > On Thu, May 16, 2019 at 12:14 PM Eric Dumazet <eric.duma...@gmail.com> > > wrote: > >> > >> > >> > >> On 5/16/19 9:05 AM, Eric Dumazet wrote: > >> > >>> We probably should add a ttl on arp queues. > >>> > >>> neigh_probe() could do that quite easily. > >>> > >> > >> Adam, all you need to do is to increase UDP socket sndbuf. > >> > >> Either by increasing /proc/sys/net/core/wmem_default > >> > >> or using setsockopt( ... SO_SNDBUF ... ) > >>
