Re: Does FreeBSD do proactive ARP refresh?
15.03.2018, 23:08, "sth...@nethelp.no" : > I have a reproducible problem on 11.1-STABLE where, during a longterm > iperf3 session, some packets are lost every time ARP is refreshed (every > net.link.ether.inet.max_age seconds). Checking with tcpdump, I can > indeed see that the packet loss is happening as the hosts are doing > ARP request/reply. I'll take a look. Indeed, the intended behaviour is to proactively refresh the record. Is the situation the same with forwarding and locally-originated traffic? With local TCP socket inpcb route caching might come into play. > > How to reproduce the problem: > > - pkg install iperf3-3.5 > - Decrease ARP aging timer to 120 (net.link.ether.inet.max_age=120) > - Run receiver as "iperf3 -s -i10" > - Run sender as "iperf3 -c -u -b4m -l100 -i10 -t3600" > - Observe packet loss of a few packets every 120 seconds on the > receiver > > I've tried this on several different hosts, with different Ethernet cards. > Same result. Also, reducing net.link.ether.inet.max_age isn't *needed* for > the problem to occur - it simply makes the problem visible faster. The > loss interval is clearly correlated with the net.link.ether.inet.max_age > value. > > Extract of typical iperf3 output on the receiver end showing loss: > > [ 5] 1810.00-1820.00 sec 4.77 MBytes 4.00 Mbits/sec 0.036 ms 0/5 (0%) > [ 5] 1820.00-1830.00 sec 4.77 MBytes 4.00 Mbits/sec 0.007 ms 0/5 (0%) > [ 5] 1830.00-1840.00 sec 4.77 MBytes 4.00 Mbits/sec 0.006 ms 4/5 (0.008%) > [ 5] 1840.00-1850.00 sec 4.77 MBytes 4.00 Mbits/sec 0.029 ms 0/49997 (0%) > [ 5] 1850.00-1860.00 sec 4.77 MBytes 4.00 Mbits/sec 0.007 ms 0/49998 (0%) > [ 5] 1860.00-1870.00 sec 4.77 MBytes 4.00 Mbits/sec 0.006 ms 0/5 (0%) > [ 5] 1870.00-1880.00 sec 4.77 MBytes 4.00 Mbits/sec 0.030 ms 0/5 (0%) > [ 5] 1880.00-1890.00 sec 4.77 MBytes 4.00 Mbits/sec 0.007 ms 0/5 (0%) > [ 5] 1890.00-1900.00 sec 4.77 MBytes 4.00 Mbits/sec 0.007 ms 0/5 (0%) > [ 5] 1900.00-1910.00 sec 4.77 MBytes 4.00 Mbits/sec 0.006 ms 0/4 (0%) > [ 5] 1910.00-1920.00 sec 4.77 MBytes 4.00 Mbits/sec 0.006 ms 0/49996 (0%) > [ 5] 1920.00-1930.00 sec 4.77 MBytes 4.00 Mbits/sec 0.007 ms 0/5 (0%) > [ 5] 1930.00-1940.00 sec 4.77 MBytes 4.00 Mbits/sec 0.013 ms 0/5 (0%) > [ 5] 1940.00-1950.00 sec 4.77 MBytes 4.00 Mbits/sec 0.007 ms 0/5 (0%) > [ 5] 1950.00-1960.00 sec 4.77 MBytes 4.00 Mbits/sec 0.006 ms 4/5 (0.008%) > [ 5] 1960.00-1970.00 sec 4.77 MBytes 4.00 Mbits/sec 0.007 ms 0/4 (0%) > [ 5] 1970.00-1980.00 sec 4.77 MBytes 4.00 Mbits/sec 0.007 ms 0/49996 (0%) > > I've tried to read some of the code in /sys/netinet/if_ether.c, and I > get impression that FreeBSD is supposed to (proactively) refresh an ARP > entry *before* it expires - specifically the arptimer() routine which > has the comment > > * Expiration time is approaching. > * Let's try to refresh entry if it is still > * in use. > > However, I'm uncertain of whether my reading here is correct. Can > somebody tell me if FreeBSD is supposed to proactively refresh an ARP > entry? > > My idea here is that as long as you have a valid ARP entry, it should > be possible to refresh the ARP entry *and* continue sending traffic > (with no packet drops) - i.e. the goal is to completely avoid the > drops I'm currently seeing on every ARP refresh. > > Steinar Haug, Nethelp consulting, sth...@nethelp.no > ___ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Does FreeBSD do proactive ARP refresh?
On 15.03.2018 23:05, sth...@nethelp.no wrote: > My idea here is that as long as you have a valid ARP entry, it should > be possible to refresh the ARP entry *and* continue sending traffic > (with no packet drops) - i.e. the goal is to completely avoid the > drops I'm currently seeing on every ARP refresh. > Can you show your config and stats after the test? # sysctl net.link.ether # netstat -sp arp # netstat -sp ip # netstat -sp udp -- WBR, Andrey V. Elsukov signature.asc Description: OpenPGP digital signature
Re: Does FreeBSD do proactive ARP refresh?
> > I have a reproducible problem on 11.1-STABLE where, during a longterm > > iperf3 session, some packets are lost every time ARP is refreshed (every > > net.link.ether.inet.max_age seconds). Checking with tcpdump, I can > > indeed see that the packet loss is happening as the hosts are doing > > ARP request/reply. > I'll take a look. Indeed, the intended behaviour is to proactively refresh > the record. > Is the situation the same with forwarding and locally-originated traffic? > With local TCP socket inpcb route caching might come into play. My testing is with locally-originated UDP traffic (iperf3 -u). Haven't tested what happens if the box is forwarding the traffic - however, I believe TCP socket inpcb route caching should not be relevant for the UDP traffic? (But there is also a TCP transaction between the iperf3 sender and the iperf3 receiver at the *start* of the iperf3 session.) In any case, I will also test what happens if the box is forwarding the traffic. Steinar Haug, Nethelp consulting, sth...@nethelp.no ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Does FreeBSD do proactive ARP refresh?,Re: Does FreeBSD do proactive ARP refresh?
> Can you show your config and stats after the test? > # sysctl net.link.ether > # netstat -sp arp > # netstat -sp ip > # netstat -sp udp OK, below is the output from these commands, run on the iperf3 sender, from before and after one test run. I ran the test for 370 seconds, and had 3 periods of packet loss, spaced 120 seconds apart. Steinar Haug, Nethelp consulting, sth...@nethelp.no -- # Before sthaug@pondus% sysctl net.link.ether net.link.ether.inet.allow_multicast: 0 net.link.ether.inet.log_arp_permanent_modify: 1 net.link.ether.inet.log_arp_movements: 1 net.link.ether.inet.log_arp_wrong_iface: 1 net.link.ether.inet.garp_rexmit_count: 0 net.link.ether.inet.max_log_per_second: 1 net.link.ether.inet.maxhold: 1 net.link.ether.inet.wait: 20 net.link.ether.inet.proxyall: 0 net.link.ether.inet.maxtries: 5 net.link.ether.inet.max_age: 120 net.link.ether.ipfw: 0 sthaug@pondus% netstat -sp arp arp: 67 ARP requests sent 812 ARP replies sent 54274 ARP requests received 63 ARP replies received 54337 ARP packets received 339 total packets dropped due to no ARP entry 63 ARP entrys timed out 0 Duplicate IPs seen sthaug@pondus% netstat -sp ip ip: 52434 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 51339 packets for this host 0 packets for unknown/unsupported protocol 33 packets forwarded (18 packets fast forwarded) 1 packet not forwardable 0 packets received for unknown multicast group 0 redirects sent 109204414 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 23551 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header sthaug@pondus% netstat -sp udp udp: 2750 datagrams received 0 with incomplete header 0 with bad data length field 0 with bad checksum 440 with no checksum 2138 dropped due to no socket 0 broadcast/multicast datagrams undelivered 0 dropped due to full socket buffers 0 not for hashed pcb 612 delivered 109160018 datagrams output 0 times multicast source filter matched # After sthaug@pondus% sysctl net.link.ether net.link.ether.inet.allow_multicast: 0 net.link.ether.inet.log_arp_permanent_modify: 1 net.link.ether.inet.log_arp_movements: 1 net.link.ether.inet.log_arp_wrong_iface: 1 net.link.ether.inet.garp_rexmit_count: 0 net.link.ether.inet.max_log_per_second: 1 net.link.ether.inet.maxhold: 1 net.link.ether.inet.wait: 20 net.link.ether.inet.proxyall: 0 net.link.ether.inet.maxtries: 5 net.link.ether.inet.max_age: 120 net.link.ether.ipfw: 0 sthaug@pondus% netstat -sp arp arp: 70 ARP requests sent 813 ARP replies sent 54297 ARP requests received 66 ARP replies received 54363 ARP packets received 351 total packets dropped due to no ARP entry 66 ARP entrys timed out 0 Duplicate IPs seen sthaug@pondus% netstat -sp ip ip: 52527 total packets received 0 bad header checksums 0 with size smaller than minimum 0 with data size < data length 0 with ip length > max ip packet size 0 with header length < data size 0 with data length < header length 0 with bad options 0 with incorrect version number 0 fragments received 0 fragments dropped (dup or out of space) 0 fragments dropped after timeout 0 packets reassembled ok 51432 packets for this host 0 packets for unknown/unsupported protocol 33 packets forwarded (18 packets fast forwarded) 1 packet not forwardable 0 packets received for unknown multicast group 0 redirects sent 111059559 packets sent from this host 0 packets sent with fabricated ip header 0 output packets dropped due to no bufs, etc. 23551 output packets discarded due to no route 0 output datagrams fragmented 0 fragments created 0 datagrams that can't be fragmented 0 tunneling packets that can't find gif 0 datagrams with bad address in header sthaug@pondus% netstat -sp udp udp: 2751 datagrams received 0 with incomp
Re: Does FreeBSD do proactive ARP refresh?
> > > I have a reproducible problem on 11.1-STABLE where, during a longterm > > > iperf3 session, some packets are lost every time ARP is refreshed (every > > > net.link.ether.inet.max_age seconds). Checking with tcpdump, I can > > > indeed see that the packet loss is happening as the hosts are doing > > > ARP request/reply. > > I'll take a look. Indeed, the intended behaviour is to proactively refresh > > the record. > > Is the situation the same with forwarding and locally-originated traffic? > > With local TCP socket inpcb route caching might come into play. > > My testing is with locally-originated UDP traffic (iperf3 -u). Haven't > tested what happens if the box is forwarding the traffic - however, I > believe TCP socket inpcb route caching should not be relevant for the > UDP traffic? (But there is also a TCP transaction between the iperf3 > sender and the iperf3 receiver at the *start* of the iperf3 session.) > > In any case, I will also test what happens if the box is forwarding > the traffic. And thank you for that suggestion! The packet loss during ARP refresh (of the destination address connected to the output interface) does *not* happen when the box is forwarding! It only happens with locally generated traffic. Steinar Haug, Nethelp consulting, sth...@nethelp.no ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Does FreeBSD do proactive ARP refresh?
> > > > I have a reproducible problem on 11.1-STABLE where, during a longterm > > > > iperf3 session, some packets are lost every time ARP is refreshed (every > > > > net.link.ether.inet.max_age seconds). Checking with tcpdump, I can > > > > indeed see that the packet loss is happening as the hosts are doing > > > > ARP request/reply. > > > I'll take a look. Indeed, the intended behaviour is to proactively > > > refresh the record. > > > Is the situation the same with forwarding and locally-originated traffic? > > > With local TCP socket inpcb route caching might come into play. > > > > My testing is with locally-originated UDP traffic (iperf3 -u). Haven't > > tested what happens if the box is forwarding the traffic - however, I > > believe TCP socket inpcb route caching should not be relevant for the > > UDP traffic? (But there is also a TCP transaction between the iperf3 > > sender and the iperf3 receiver at the *start* of the iperf3 session.) > > > > In any case, I will also test what happens if the box is forwarding > > the traffic. > > And thank you for that suggestion! The packet loss during ARP refresh > (of the destination address connected to the output interface) does > *not* happen when the box is forwarding! It only happens with locally > generated traffic. Checking once per second with "arp -n " I can see the following behavior with net.link.ether.inet.max_age=120: - Locally generated traffic: The ARP entry is refreshed after net.link.ether.inet.max_age seconds - which presumably means it actually expires first. And there is some packet loss. - Transit traffic (the box is forwarding): The ARP entry is refreshed 5 seconds *before* net.link.ether.inet.max_age has passed, and there is no packet loss. Steinar Haug, Nethelp consulting, sth...@nethelp.no ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Does FreeBSD do proactive ARP refresh?
On 16.03.2018 14:47, sth...@nethelp.no wrote:
>> And thank you for that suggestion! The packet loss during ARP refresh
>> (of the destination address connected to the output interface) does
>> *not* happen when the box is forwarding! It only happens with locally
>> generated traffic.
>
> Checking once per second with "arp -n " I can see the
> following behavior with net.link.ether.inet.max_age=120:
>
> - Locally generated traffic: The ARP entry is refreshed after
> net.link.ether.inet.max_age seconds - which presumably means it
> actually expires first. And there is some packet loss.
>
> - Transit traffic (the box is forwarding): The ARP entry is refreshed
> 5 seconds *before* net.link.ether.inet.max_age has passed, and there
> is no packet loss.
Can you test this patch? I did not tested it, but I think it should fix
this issue.
--
WBR, Andrey V. Elsukov
Index: sys/net/if_ethersubr.c
===
--- sys/net/if_ethersubr.c (revision 330791)
+++ sys/net/if_ethersubr.c (working copy)
@@ -317,6 +317,17 @@ ether_output(struct ifnet *ifp, struct mbuf *m,
phdr = lle->r_linkdata;
hlen = lle->r_hdrlen;
pflags = lle->r_flags;
+/*
+ * XXX: Check if we have feedback request
+ * from arptimer()/nd6_llinfo_timer().
+ */
+if ((pflags & RLLE_VALID) &&
+lle->r_skip_req != 0) {
+ LLE_REQ_LOCK(lle);
+ /* Notify that entry was used */
+ lle->r_skip_req = 0;
+ LLE_REQ_UNLOCK(lle);
+}
}
}
}
signature.asc
Description: OpenPGP digital signature
Re: Does FreeBSD do proactive ARP refresh?,Re: Does FreeBSD do proactive ARP refresh?
> >> And thank you for that suggestion! The packet loss during ARP refresh
> >> (of the destination address connected to the output interface) does
> >> *not* happen when the box is forwarding! It only happens with locally
> >> generated traffic.
> >
> > Checking once per second with "arp -n " I can see the
> > following behavior with net.link.ether.inet.max_age=120:
> >
> > - Locally generated traffic: The ARP entry is refreshed after
> > net.link.ether.inet.max_age seconds - which presumably means it
> > actually expires first. And there is some packet loss.
> >
> > - Transit traffic (the box is forwarding): The ARP entry is refreshed
> > 5 seconds *before* net.link.ether.inet.max_age has passed, and there
> > is no packet loss.
>
> Can you test this patch? I did not tested it, but I think it should fix
> this issue.
Looks like this is a patch against HEAD. I don't have any boxes
running HEAD, but I can get one installed and test the patch this
weekend (unless you want to produce a patch against 11.1-STABLE).
I tried the patch manually applied on 11.1-STABLE, but got lots of
errors during kernel compilation (see the end of this message).
Steinar Haug, Nethelp consulting, sth...@nethelp.no
--
cc -c -O2 -pipe -fno-strict-aliasing -g -nostdinc -I. -I../../..
-I../../../contrib/libfdt -D_KERNEL -DHAVE_KERNEL_OPTION_HEADERS -include
opt_global.h -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer -MD
-MF.depend.if_ethersubr.o -MTif_ethersubr.o -mcmodel=kernel -mno-red-zone
-mno-mmx -mno-sse -msoft-float -fno-asynchronous-unwind-tables -ffreestanding
-fwrapv -fstack-protector -gdwarf-2 -Wall -Wredundant-decls -Wnested-externs
-Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual
-Wundef -Wno-pointer-sign -D__printf__=__freebsd_kprintf__
-Wmissing-include-dirs -fdiagnostics-show-option -Wno-unknown-pragmas
-Wno-error-tautological-compare -Wno-error-empty-body
-Wno-error-parentheses-equality -Wno-error-unused-function
-Wno-error-pointer-sign -Wno-error-shift-negative-value
-Wno-error-address-of-packed-member -mno-aes -mno-avx -std=iso9899:1999
-Werror ../../../net/if_ethersubr.c
../../../net/if_ethersubr.c:288:18: error: unused variable 't'
[-Werror,-Wunused-variable]
struct pf_mtag *t;
^
../../../net/if_ethersubr.c:286:7: error: unused variable 'linkhdr'
[-Werror,-Wunused-variable]
char linkhdr[ETHER_HDR_LEN], *phdr;
^
../../../net/if_ethersubr.c:285:6: error: unused variable 'error'
[-Werror,-Wunused-variable]
int error = 0;
^
../../../net/if_ethersubr.c:289:6: error: unused variable 'loop_copy'
[-Werror,-Wunused-variable]
int loop_copy = 1;
^
../../../net/if_ethersubr.c:287:23: error: unused variable 'eh'
[-Werror,-Wunused-variable]
struct ether_header *eh;
^
../../../net/if_ethersubr.c:293:18: error: unused variable 'rt0'
[-Werror,-Wunused-variable]
struct rtentry *rt0 = NULL;
^
../../../net/if_ethersubr.c:335:4: error: control reaches end of non-void
function [-Werror,-Wreturn-type]
}
^
../../../net/if_ethersubr.c:336:3: error: extraneous closing brace ('}')
}
^
../../../net/if_ethersubr.c:337:3: error: type specifier missing, defaults to
'int' [-Werror,-Wimplicit-int]
rt0 = ro->ro_rt;
^
../../../net/if_ethersubr.c:337:9: error: use of undeclared identifier 'ro'
rt0 = ro->ro_rt;
^
../../../net/if_ethersubr.c:338:2: error: extraneous closing brace ('}')
}
^
../../../net/if_ethersubr.c:341:2: error: type specifier missing, defaults to
'int' [-Werror,-Wimplicit-int]
error = mac_ifnet_check_transmit(ifp, m);
^
../../../net/if_ethersubr.c:341:35: error: use of undeclared identifier 'ifp'
error = mac_ifnet_check_transmit(ifp, m);
^
../../../net/if_ethersubr.c:341:40: error: use of undeclared identifier 'm'
error = mac_ifnet_check_transmit(ifp, m);
^
../../../net/if_ethersubr.c:342:2: error: expected identifier or '('
if (error)
^
../../../net/if_ethersubr.c:343:3: error: expected identifier or '('
senderr(error);
^
../../../net/if_ethersubr.c:123:49: note: expanded from macro 'senderr'
#define senderr(e) do { error = (e); goto bad;} while (0)
^
../../../net/if_ethersubr.c:347:2: error: expected identifier or '('
if (ifp->if_flags & IFF_MONITOR)
^
../../../net/if_ethersubr.c:348:3: error: expected identifier or '('
senderr(ENETDOWN);
^
../../../net/if_ethersubr.c:123:49: note: expanded from macro 'senderr
Re: Does FreeBSD do proactive ARP refresh?,Re: Does FreeBSD do proactive ARP refresh?
On 16.03.2018 16:47, sth...@nethelp.no wrote:
>> Can you test this patch? I did not tested it, but I think it should fix
>> this issue.
>
> Looks like this is a patch against HEAD. I don't have any boxes
> running HEAD, but I can get one installed and test the patch this
> weekend (unless you want to produce a patch against 11.1-STABLE).
>
> I tried the patch manually applied on 11.1-STABLE, but got lots of
> errors during kernel compilation (see the end of this message).
I made the patch for stable/11, but I need some time to build
kernel-toolchain, since compiler from head/ doesn't want to build
stable/11 kernel :)
--
WBR, Andrey V. Elsukov
Index: stable/11/sys/net/if_ethersubr.c
===
--- stable/11/sys/net/if_ethersubr.c (revision 331055)
+++ stable/11/sys/net/if_ethersubr.c (working copy)
@@ -292,7 +292,6 @@ ether_output(struct ifnet *ifp, struct mbuf *m,
int hlen; /* link layer header length */
uint32_t pflags;
struct llentry *lle = NULL;
- struct rtentry *rt0 = NULL;
int addref = 0;
phdr = NULL;
@@ -320,9 +319,20 @@ ether_output(struct ifnet *ifp, struct mbuf *m,
phdr = lle->r_linkdata;
hlen = lle->r_hdrlen;
pflags = lle->r_flags;
+/*
+ * XXX: Check if we have feedback request
+ * from arptimer()/nd6_llinfo_timer().
+ */
+if ((pflags & RLLE_VALID) &&
+lle->r_skip_req != 0) {
+ LLE_REQ_LOCK(lle);
+ /* Notify that entry was used */
+ lle->r_skip_req = 0;
+ lle->lle_hittime = time_uptime;
+ LLE_REQ_UNLOCK(lle);
+}
}
}
- rt0 = ro->ro_rt;
}
#ifdef MAC
signature.asc
Description: OpenPGP digital signature
netmap ixgbevf mtu
I am having difficulties with netmap over top of ixgbevf when attempting to use a large MTU (say 9000 bytes). Does the ixgbevf driver use 2048 byte buffers for RX regardless of the MTU or netmap buffer size? I can send large frames just fine but inbound frames are passed via netmap as 2048 bytes max. It is possible that netmap is passing frames in multiple pieces I suppose, I haven't checked that yet -- my code is looking at the frame headers only at the moment so would toss trailing pieces. Joe Buehler ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: netmap ixgbevf mtu
Sorry, I should have added, this is LINUX if it matters. Joe Buehler wrote: > I am having difficulties with netmap over top of ixgbevf when attempting to > use a large MTU (say 9000 bytes). > > Does the ixgbevf driver use 2048 byte buffers for RX regardless of the MTU or > netmap buffer size? > > I can send large frames just fine but inbound frames are passed via netmap as > 2048 bytes max. It is possible that netmap is passing frames in multiple > pieces I suppose, I haven't checked that yet -- my code is looking at the > frame headers only at the moment so would toss trailing pieces. > > Joe Buehler > > ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
