Re: kern/176446: [netinet] [patch] Concurrency in ixgbe driving out-of-order packet process and spurious RST
The following reply was made to PR kern/176446; it has been noted by GNATS.
From: "Charbon, Julien"
To: Cc: bug-follo...@freebsd.org,
"De La Gueronniere, Marc"
Subject: Re: kern/176446: [netinet] [patch] Concurrency in ixgbe driving
out-of-order
packet process and spurious RST
Date: Tue, 12 Mar 2013 14:59:11 +0100
This is a multi-part message in MIME format.
--05060901050001050509
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
On 3/7/13 11:11 AM, Charbon, Julien wrote:
> On 2/28/13 8:10 PM, Charbon, Julien wrote:
>> On 2/28/13 4:57 PM, John Baldwin wrote:
>>> Can you try the fixes from
>>> http://svnweb.freebsd.org/base?view=revision&revision=240968?
>>
>> Actually, Marc (I CC'ed him) did find the r240968 fix for concurrency
>> between ixgbe_msix_que() and ixgbe_handle_que(), and made a backport for
>> release-8.3.0 (see patch [1] below). However, the issue was still
>> reproducible, then Marc found another place for concurrency from
>> ixgbe_local_timer() and fix it (see patch [2]). But it was still not
>> enough, and he found a last place for concurrency due to
>> ixgbe_rearm_queues() call (see patch [3]). We all these patches
>> applied, we were not able to reproduce this issue.
>
>Just for the record: As expected this issue is reproducible on
> 9.1-RELEASE:
Just for the record: As expected this issue is reproducible also on
10.0-CURRENT:
# uname -a
FreeBSD atlas 10.0-CURRENT FreeBSD 10.0-CURRENT #0 r248173M: Tue Mar 12
07:52:58 UTC 2013
jcharbon@atlas:/usr/obj/app/jcharbon/head/sys/GENERIC amd64
1. Enable TCP debug log:
# sysctl net.inet.tcp.log_debug=1
net.inet.tcp.log_debug: 1 -> 1
2. Load a TCP service with numerous small requests/responses:
3. Look in /var/log/debug.log for the pattern:
Mar 12 10:31:22 atlas kernel: TCP: [192.168.100.35]:4698 to
[192.168.100.152]:8080; syncache_socket: in_pcbconnect failed with error 48
Mar 12 10:31:22 atlas-dl360-4 kernel: TCP: [192.168.100.35]:4698 to
[192.168.100.152]:8080 tcpflags 0x10; tcp_input: Listen socket:
Socket allocation failed due to limits or memory shortage, sending RST
Mar 12 10:31:22 atlas-dl360-4 kernel: TCP: [192.168.100.35]:4698 to
[192.168.100.152]:8080 tcpflags 0x4; syncache_chkrst: Spurious RST
without matching syncache entry (possibly syncookie only), segment ignored
Joined the patch we use to fix this issue in 10-CURRENT.
--
Julien
--05060901050001050509
Content-Type: text/plain; charset=UTF-8; x-mac-type="0"; x-mac-creator="0";
name="ixgbe.c.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="ixgbe.c.patch"
Index: sys/dev/ixgbe/ixgbe.c
===
--- sys/dev/ixgbe/ixgbe.c (revision 248173)
+++ sys/dev/ixgbe/ixgbe.c (working copy)
@@ -2038,14 +2038,14 @@
(paused == 0))
++hung;
else if (txr->queue_status == IXGBE_QUEUE_WORKING)
- taskqueue_enqueue(que->tq, &que->que_task);
+ taskqueue_enqueue(que->tq, &txr->txq_task);
}
/* Only truely watchdog if all queues show hung */
if (hung == adapter->num_queues)
goto watchdog;
out:
- ixgbe_rearm_queues(adapter, adapter->que_mask);
+ // ixgbe_rearm_queues(adapter, adapter->que_mask);
callout_reset(&adapter->timer, hz, ixgbe_local_timer, adapter);
return;
@@ -4575,7 +4575,7 @@
** Schedule another interrupt if so.
*/
if ((staterr & IXGBE_RXD_STAT_DD) != 0) {
- ixgbe_rearm_queues(adapter, (u64)(1 << que->msix));
+ // ixgbe_rearm_queues(adapter, (u64)(1 << que->msix));
return (TRUE);
}
--05060901050001050509--
___
freebsd-net@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Problems of using ipv6 addresses on loc0
I have a FreeBSD 8.2-STABLE amd64. Network: vlan300: flags=8843 metric 0 mtu 1500 description: Uplink_1 inet inet6 vlan400: flags=8843 metric 0 mtu 1500 description: Uplink_2 inet inet6 vlan100: flags=8843 metric 0 mtu 1500 description: Internal_LAN inet inet6 2001:67c:21f0:: prefixlen 64 lo0: flags=8049 metric 0 mtu 16384 options=3 inet 127.0.0.1 netmask 0xff00 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9 inet6 2001:67c:21f0:f::1 prefixlen 64 nd6 options=3 Some services, such as named, refused to use ipv6 with lo0. Option query-source-v6 address 2001:67c:21f0::1; makes a mistake and named service will not start. Problems with the ipv6 address are on vlan100 - no. P.S. BIND 9.9.2-P1 -- Vladislav V. Prodan System & Network Administrator http://support.od.ua +380 67 4584408, +380 99 4060508 VVP88-RIPE ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
mbuf tuning on 9.1
Hi. I have a FreeBSD 9.1/amd64 machine. It runs HAProxy and runs out of various things frequently. Sometimes it's mbufs. netstat -m: 68202/1698/69900 mbufs in use (current/cache/total) 41449/1229/42678/2622144 mbuf clusters in use (current/cache/total/max) 41449/1175 mbuf+clusters out of packet secondary zone in use (current/cache) 26405/538/26943/3276800 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/6400 9k jumbo clusters in use (current/cache/total/max) 0/0/0/3200 16k jumbo clusters in use (current/cache/total/max) 205568K/5034K/210603K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile 0 calls to protocol drain routines How can I increase "mbufs," as they appear above, and "mbuf clusters," as they appear above? Thank you. -Boris ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: mbuf tuning on 9.1
> How can I increase "mbufs," as they appear above, and "mbuf clusters," > as they appear above? You can modify the sysctl's associated with mbufs to suit your needs. https://wiki.freebsd.org/NetworkPerformanceTuning The following link describes what mbufs are and sysctl's governing their operation. ~Paul This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: mbuf tuning on 9.1
Additionally, can someone clarify the meaning of "total" vs. "max" for these values? -Boris On 03/12/13 17:23, Paul A. Procacci wrote: >> How can I increase "mbufs," as they appear above, and "mbuf clusters," >> as they appear above? > You can modify the sysctl's associated with mbufs to suit your needs. > > https://wiki.freebsd.org/NetworkPerformanceTuning > > The following link describes what mbufs are and sysctl's governing > their operation. > > ~Paul > > > > This message may contain confidential or privileged information. If you are > not the intended recipient, please advise us immediately and delete this > message. See http://www.datapipe.com/legal/email_disclaimer/ for further > information on confidentiality and the risks of non-secure electronic > communication. If you cannot access these links, please notify us by reply > message and we will send the contents to you. > ___ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
ntpd bind() failure: Can't assign requested address
Hi! [First of all, I have posted this question already on the FreeBSD forum -- so far without replies -- and now my hope is that the set of subscribers here and those of the forum do not completely coincide.] I have installed FreeBSD 9.1 on my server (it's a virtualized qemu/kvm environment, in case that matters) and during boot the following messages appear: ntpd[766]: ntpd 4.2.4p5-a (1) ntpd[767]: bind() fd 23, family AF_INET6, port 123, scope 2, addr fe80::216:36ff:fe74:2076, mcast=0 flags=0x11 fails: Can't assign requested address ntpd[767]: unable to create socket on re0 (3) for fe80::216:36ff:fe74:2076#123 This happens with a GENERIC kernel. I have not touched any IPv6 related configuration after install, so everything should be in the default state. Neither have I changed the ntp/ntpd configuration. Although I could find some threads where people were discussing the same problem, none of the mentioned suggestions fixed this for me. In particular I tried setting ipv6_activate_all_interfaces="YES" in rc.conf -- didn't fix the problem. According to ifconfig, my interfaces, re0 in particular, have IPv6 addresses associated to them. So why would it fail that a process tries to bind to it? Note that I'm rather unexperienced wrt IPv6. Would be great if somebody could give me a hint into the right direction. Anybody else experiencing this? Thank you very much! ~ melanie ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ntpd bind() failure: Can't assign requested address
On 12 Mar 2013, at 22:42, "M. Schulte" wrote: > Hi! > > [First of all, I have posted this question already on the FreeBSD > forum -- so far without replies -- and now my hope is that the set of > subscribers here and those of the forum do not completely coincide.] > > I have installed FreeBSD 9.1 on my server (it's a virtualized qemu/kvm > environment, in case that matters) and during boot the following > messages appear: > > ntpd[766]: ntpd 4.2.4p5-a (1) > ntpd[767]: bind() fd 23, family AF_INET6, port 123, scope 2, addr > fe80::216:36ff:fe74:2076, mcast=0 flags=0x11 fails: Can't assign requested > address > ntpd[767]: unable to create socket on re0 (3) for > fe80::216:36ff:fe74:2076#123 > > This happens with a GENERIC kernel. I have not touched any IPv6 > related configuration after install, so everything should be in the > default state. Neither have I changed the ntp/ntpd configuration. > > Although I could find some threads where people were discussing the > same problem, none of the mentioned suggestions fixed this for me. > > In particular I tried setting > > ipv6_activate_all_interfaces="YES" > > in rc.conf -- didn't fix the problem. > > According to ifconfig, my interfaces, re0 in particular, have IPv6 > addresses associated to them. So why would it fail that a process > tries to bind to it? Note that I'm rather unexperienced wrt IPv6. > > Would be great if somebody could give me a hint into the right > direction. Anybody else experiencing this? Thank you very much! > > ~ melanie > I'll check tomorrow if we have the same issue at work, although it doesn't look familiar. We've got a 9.1 with a custom kernel (although nothing related to ipv6 changed) running in KVM. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Problems of using ipv6 addresses on loc0
On 12 Mar 2013, at 21:06, "Vladislav Prodan" wrote: > > I have a FreeBSD 8.2-STABLE amd64. > > Network: > vlan300: flags=8843 metric 0 mtu 1500 >description: Uplink_1 >inet >inet6 > > vlan400: flags=8843 metric 0 mtu 1500 >description: Uplink_2 >inet >inet6 > > vlan100: flags=8843 metric 0 mtu 1500 >description: Internal_LAN >inet >inet6 2001:67c:21f0:: prefixlen 64 > > lo0: flags=8049 metric 0 mtu 16384 >options=3 >inet 127.0.0.1 netmask 0xff00 >inet6 ::1 prefixlen 128 >inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9 >inet6 2001:67c:21f0:f::1 prefixlen 64 >nd6 options=3 > > Some services, such as named, refused to use ipv6 with lo0. > Option >query-source-v6 address 2001:67c:21f0::1; > makes a mistake and named service will not start. > Problems with the ipv6 address are on vlan100 - no. > > P.S. BIND 9.9.2-P1 > > -- > Vladislav V. Prodan > System & Network Administrator > http://support.od.ua > +380 67 4584408, +380 99 4060508 > VVP88-RIPE > The addresses for lo0 and BIND's query-source differ (extraneous :f), so unless this is a typo, that may be your problem. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
ipv6 default router Operation not permitted
Hi, I have a problem reaching my ipv6 default router. # ping6 -c 1 2001:41d0:2:e7ff:ff:ff:ff:ff PING6(56=40+8+8 bytes) 2001:41d0:2:e7c4::1 --> 2001:41d0:2:e7ff:ff:ff:ff:ff ping6: sendmsg: Operation not permitted ping6: wrote 2001:41d0:2:e7ff:ff:ff:ff:ff 16 chars, ret=-1 --- 2001:41d0:2:e7ff:ff:ff:ff:ff ping6 statistics --- 1 packets transmitted, 0 packets received, 100.0% packet loss It is the same issue as posted here : http://lists.freebsd.org/pipermail/freebsd-net/2012-February/031518.html However, I believe the problem Mr Dandy was experiencing was he should have changed his default gateway from: ipv6_defaultrouter="2a01:4f8:61:50c0::1" To: ipv6_defaultrouter="fe80::1%re0" As per Hertzner documentation. http://wiki.hetzner.de/index.php/FreeBSD_installieren/en#Network_Configuration I am not hosted with Hertzner but I have the same broken network configuration and am uncertain of the Correct Fix. I have: ::: /etc/rc.conf ::: ipv6_activate_all_interfaces="YES" ipv6_default_interface="re0" ipv6_network_interfaces="auto" ip6addrctl_policy="ipv6_prefer" ifconfig_re0_ipv6="inet6 2001:41D0:2:E7c4::1 prefixlen 64" ipv6_static_routes="ovhgw default" ipv6_route_ovhgw="2001:41d0:2:e7ff:ff:ff:ff:ff -prefixlen 128 -interface re0" ipv6_route_default="default 2001:41d0:2:e7ff:ff:ff:ff:ff" The default IPv6 router is _outside_ of my assigned /64 prefix, this is why an interface route is set and is configured this way to ensure the interface route is applied before the default IPv6 router is added to the routing table. If I do not do this I will receive an error on boot to say that the default gateway cannot be reached. There is also no firewall in place on the host OS. However this configuration does not work. # ping6 -c 1 2001:41d0:2:e7ff:ff:ff:ff:ff PING6(56=40+8+8 bytes) 2001:41d0:2:e7c4::1 --> 2001:41d0:2:e7ff:ff:ff:ff:ff ping6: sendmsg: Operation not permitted ping6: wrote 2001:41d0:2:e7ff:ff:ff:ff:ff 16 chars, ret=-1 --- 2001:41d0:2:e7ff:ff:ff:ff:ff ping6 statistics --- 1 packets transmitted, 0 packets received, 100.0% packet loss If I do # ifconfig re0 inet6 accept_rtadv It works : # ping6 -c 1 2001:41d0:2:e7ff:ff:ff:ff:ff PING6(56=40+8+8 bytes) 2001:41d0:2:e7c4::1 --> 2001:41d0:2:e7ff:ff:ff:ff:ff 16 bytes from 2001:41d0:2:e7ff:ff:ff:ff:ff, icmp_seq=0 hlim=64 time=4.935 ms --- 2001:41d0:2:e7ff:ff:ff:ff:ff ping6 statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 4.935/4.935/4.935/0.000 ms Alternatively, knowing the Linklayer Address of my default gateway, I could set a static ndp pair : # ifconfig re0 inet6 -accept_rtadv # ndp -s 2001:41d0:2:e7ff:ff:ff:ff:ff 00:05:73:a0:00:00 And it works: # ping6 -c 1 2001:41d0:2:e7ff:ff:ff:ff:ff PING6(56=40+8+8 bytes) 2001:41d0:2:e7c4::1 --> 2001:41d0:2:e7ff:ff:ff:ff:ff 16 bytes from 2001:41d0:2:e7ff:ff:ff:ff:ff, icmp_seq=0 hlim=64 time=3.582 ms --- 2001:41d0:2:e7ff:ff:ff:ff:ff ping6 statistics --- 1 packets transmitted, 1 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 3.582/3.582/3.582/0.000 ms If I do : # ndp -d 2001:41d0:2:e7ff:ff:ff:ff:ff It stops working again : # ping6 -c 1 2001:41d0:2:e7ff:ff:ff:ff:ff PING6(56=40+8+8 bytes) 2001:41d0:2:e7c4::1 --> 2001:41d0:2:e7ff:ff:ff:ff:ff ping6: sendmsg: Operation not permitted ping6: wrote 2001:41d0:2:e7ff:ff:ff:ff:ff 16 chars, ret=-1 --- 2001:41d0:2:e7ff:ff:ff:ff:ff ping6 statistics --- 1 packets transmitted, 0 packets received, 100.0% packet loss It seems the simple solution is to have re0 ACCEPT_RTADV but I am told that my default gateway does not perform router advertisements. In my own networks I have a static default route for inet6 and I am not required to set any interfaces ACCEPT_RTADV. Can anybody shed some light on the correct configuration? Who is at fault here ? - Me - Hosting - FreeBSD - The universe Cheers, C. -- +---+ Quidquid latine dictum sit, altum sonatur. ICQ: 112562229 GPG: http://www.konundrum.org/schro.asc pgp4hnMKHg5x8.pgp Description: PGP signature
Re[2]: Problems of using ipv6 addresses on loc0
> On 12 Mar 2013, at 21:06, "Vladislav Prodan" wrote: > > > lo0: flags=8049 metric 0 mtu 16384 > >options=3 > >inet 127.0.0.1 netmask 0xff00 > >inet6 ::1 prefixlen 128 > >inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9 > >inet6 2001:67c:21f0:f::1 prefixlen 64 > >nd6 options=3 > > > > Some services, such as named, refused to use ipv6 with lo0. > > Option > >query-source-v6 address 2001:67c:21f0::1; > > makes a mistake and named service will not start. > > Problems with the ipv6 address are on vlan100 - no. > > > > P.S. BIND 9.9.2-P1 > > > > The addresses for lo0 and BIND's query-source differ (extraneous :f), so > unless this is a typo, that may be your problem. Thanks. It's my typo! -- Vladislav V. Prodan System & Network Administrator http://support.od.ua +380 67 4584408, +380 99 4060508 VVP88-RIPE ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: mbuf tuning on 9.1
On Tue, Mar 12, 2013 at 5:31 PM, Boris Kochergin wrote: > Additionally, can someone clarify the meaning of "total" vs. "max" for > these values? > > -Boris > It measures the amount of memory being used as that type of data structure. This is the total of the number of allocated items (current) and the number of free items that are being cached by the allocater for faster allocations in the future (cached). ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ntpd bind() failure: Can't assign requested address
On Tue, Mar 12, 2013 at 2:42 PM, M. Schulte wrote: > Hi! > > [First of all, I have posted this question already on the FreeBSD > forum -- so far without replies -- and now my hope is that the set of > subscribers here and those of the forum do not completely coincide.] > > I have installed FreeBSD 9.1 on my server (it's a virtualized qemu/kvm > environment, in case that matters) and during boot the following > messages appear: > >ntpd[766]: ntpd 4.2.4p5-a (1) >ntpd[767]: bind() fd 23, family AF_INET6, port 123, scope 2, addr > fe80::216:36ff:fe74:2076, mcast=0 flags=0x11 fails: Can't assign requested > address >ntpd[767]: unable to create socket on re0 (3) for > fe80::216:36ff:fe74:2076#123 > > This happens with a GENERIC kernel. I have not touched any IPv6 > related configuration after install, so everything should be in the > default state. Neither have I changed the ntp/ntpd configuration. > > Although I could find some threads where people were discussing the > same problem, none of the mentioned suggestions fixed this for me. > > In particular I tried setting > >ipv6_activate_all_interfaces="**YES" > > in rc.conf -- didn't fix the problem. > > According to ifconfig, my interfaces, re0 in particular, have IPv6 > addresses associated to them. So why would it fail that a process > tries to bind to it? Note that I'm rather unexperienced wrt IPv6. > > Would be great if somebody could give me a hint into the right > direction. Anybody else experiencing this? Thank you very much! > > ~ melanie > What is the address space is the address you are seeing on the interface? Is it is a link-local address or one assigned by the router, a static from rc.conf or DHCPv6? You should have two inet6 addresses, one is link-local (starts with "fe") and another should start with "2". There may be more than two. -- R. Kevin Oberman, Network Engineer E-mail: rkober...@gmail.com ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: ipv6 default router Operation not permitted
On 12 Mar 2013, at 23:50, Schrodinger wrote: > Hi, > > I have a problem reaching my ipv6 default router. > > # ping6 -c 1 2001:41d0:2:e7ff:ff:ff:ff:ff > PING6(56=40+8+8 bytes) 2001:41d0:2:e7c4::1 --> 2001:41d0:2:e7ff:ff:ff:ff:ff > ping6: sendmsg: Operation not permitted > ping6: wrote 2001:41d0:2:e7ff:ff:ff:ff:ff 16 chars, ret=-1 > > --- 2001:41d0:2:e7ff:ff:ff:ff:ff ping6 statistics --- > 1 packets transmitted, 0 packets received, 100.0% packet loss > > It is the same issue as posted here : > > http://lists.freebsd.org/pipermail/freebsd-net/2012-February/031518.html > > However, I believe the problem Mr Dandy was experiencing was he should > have changed his default gateway from: > > ipv6_defaultrouter="2a01:4f8:61:50c0::1" > > To: > > ipv6_defaultrouter="fe80::1%re0" > > As per Hertzner documentation. > > http://wiki.hetzner.de/index.php/FreeBSD_installieren/en#Network_Configuration > > I am not hosted with Hertzner but I have the same broken network > configuration and am uncertain of the Correct Fix. > > I have: > > ::: /etc/rc.conf ::: > > ipv6_activate_all_interfaces="YES" > ipv6_default_interface="re0" > ipv6_network_interfaces="auto" > ip6addrctl_policy="ipv6_prefer" > > ifconfig_re0_ipv6="inet6 2001:41D0:2:E7c4::1 prefixlen 64" > > ipv6_static_routes="ovhgw default" > ipv6_route_ovhgw="2001:41d0:2:e7ff:ff:ff:ff:ff -prefixlen 128 -interface re0" > ipv6_route_default="default 2001:41d0:2:e7ff:ff:ff:ff:ff" > > The default IPv6 router is _outside_ of my assigned /64 prefix, this is > why an interface route is set and is configured this way to ensure the > interface route is applied before the default IPv6 router is added to > the routing table. If I do not do this I will receive an error on boot > to say that the default gateway cannot be reached. There is also no > firewall in place on the host OS. > > However this configuration does not work. > > # ping6 -c 1 2001:41d0:2:e7ff:ff:ff:ff:ff > PING6(56=40+8+8 bytes) 2001:41d0:2:e7c4::1 --> 2001:41d0:2:e7ff:ff:ff:ff:ff > ping6: sendmsg: Operation not permitted > ping6: wrote 2001:41d0:2:e7ff:ff:ff:ff:ff 16 chars, ret=-1 > > --- 2001:41d0:2:e7ff:ff:ff:ff:ff ping6 statistics --- > 1 packets transmitted, 0 packets received, 100.0% packet loss > > If I do > > # ifconfig re0 inet6 accept_rtadv > > It works : > > # ping6 -c 1 2001:41d0:2:e7ff:ff:ff:ff:ff > PING6(56=40+8+8 bytes) 2001:41d0:2:e7c4::1 --> 2001:41d0:2:e7ff:ff:ff:ff:ff > 16 bytes from 2001:41d0:2:e7ff:ff:ff:ff:ff, icmp_seq=0 hlim=64 time=4.935 ms > > --- 2001:41d0:2:e7ff:ff:ff:ff:ff ping6 statistics --- > 1 packets transmitted, 1 packets received, 0.0% packet loss > round-trip min/avg/max/std-dev = 4.935/4.935/4.935/0.000 ms > > > Alternatively, knowing the Linklayer Address of my default gateway, I > could set a static ndp pair : > > # ifconfig re0 inet6 -accept_rtadv > # ndp -s 2001:41d0:2:e7ff:ff:ff:ff:ff 00:05:73:a0:00:00 > > And it works: > > # ping6 -c 1 2001:41d0:2:e7ff:ff:ff:ff:ff > PING6(56=40+8+8 bytes) 2001:41d0:2:e7c4::1 --> 2001:41d0:2:e7ff:ff:ff:ff:ff > 16 bytes from 2001:41d0:2:e7ff:ff:ff:ff:ff, icmp_seq=0 hlim=64 time=3.582 ms > > --- 2001:41d0:2:e7ff:ff:ff:ff:ff ping6 statistics --- > 1 packets transmitted, 1 packets received, 0.0% packet loss > round-trip min/avg/max/std-dev = 3.582/3.582/3.582/0.000 ms > > If I do : > > # ndp -d 2001:41d0:2:e7ff:ff:ff:ff:ff > > It stops working again : > > # ping6 -c 1 2001:41d0:2:e7ff:ff:ff:ff:ff > PING6(56=40+8+8 bytes) 2001:41d0:2:e7c4::1 --> 2001:41d0:2:e7ff:ff:ff:ff:ff > ping6: sendmsg: Operation not permitted > ping6: wrote 2001:41d0:2:e7ff:ff:ff:ff:ff 16 chars, ret=-1 > > --- 2001:41d0:2:e7ff:ff:ff:ff:ff ping6 statistics --- > 1 packets transmitted, 0 packets received, 100.0% packet loss > > It seems the simple solution is to have re0 ACCEPT_RTADV but I am told > that my default gateway does not perform router advertisements. > > In my own networks I have a static default route for inet6 and I am not > required to set any interfaces ACCEPT_RTADV. > > Can anybody shed some light on the correct configuration? > > Who is at fault here ? > > - Me > - Hosting > - FreeBSD > - The universe > > Cheers, > C. > The network is actually /48 and you get assigned a /64 inside it. Set your interface to use the /48 prefix and voodoo will happen (I can assure you with a 97% certainty that your default GW is inside the /48). Of course, using the /48 prefix doesn't mean you can/may use IPs from outside the /64 that was given you. Also, remove all the junk from rc.conf to only set the ipv6 default gw, none of these static routes anymore. Kindly reply with topic "SOLVED" if that fixed you up, that googlers in the future may find the solution easily. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: FreeBSD 9.1-RELEASE + bge0 == watchdog timeout
On Thu, Feb 28, 2013 at 12:37:03PM +0100, Miroslav Lachman wrote: > YongHyeon PYUN wrote: > >On Wed, Feb 27, 2013 at 12:09:28PM +0100, Miroslav Lachman wrote: > > [...] > > >>I can provide you full access to this machine (if you want) or let me > >>know, what version I should check. Older versions (6.x - 8.3) are > >>working fine with hw.bge.allow_asf="1" in loader.conf. I didn't test > >>newer releases on these old machines. > > > >The reporter said the machine was Sun Fire X2200 M2 so I guess you > >may see the same issue on both stable/9 and stable/8. Ideally the > >loader tunable hw.bge.allow_asf should not be there and driver > >should take care of it by checking the existence of ASF/IPMI > >firmware. > > > >Can you setup a remote debugging environments(+ IPMI access) like > >the following URL? > >http://people.freebsd.org/~yongari/remote_debugging.txt > > The one Sun Fire X2100 M2 is idling in datacenter and connected to > internet, so I can remotely reinstall it to stable/9 withing day or two > and give you full access to it (ssh user, root, BMC / IPMI admin account > with remote KVM + remote media). But as I understand, you need another > machine connected to it with serial and another ethernet. It will take > me some more time, as I will need to go to the datacenter, find some > serial cable etc. > > Let me know if ssh + ipmi access to X2100 alone is useful for you to > start, or only full remote debugging setup is needed. > > Can you point me to the original problem report with X2200 M2? > I had been working on fixing the IPMI regression with the help of Miroslav. It was fixed in r248226. Many thanks to Miroslav for providing full remote debugging environments. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: FreeBSD 9.1-RELEASE + bge0 == watchdog timeout
On Thu, Mar 07, 2013 at 05:15:48PM +0900, YongHyeon PYUN wrote: > On Thu, Mar 07, 2013 at 01:14:03PM +0600, Eugene M. Zheganin wrote: > > Hi. > > > > On 07.03.2013 12:23, YongHyeon PYUN wrote: > > > On Thu, Mar 07, 2013 at 11:08:50AM +0600, Eugene M. Zheganin wrote: > > >> It was definitely older than "months". It was running something similar > > >> to "FreeBSD 8.2-STABLE #0: Mon Sep 19 08:10:00 YEKST 2011", this is the > > >> uname from a neighbor machine. > > >> > > >> I have, as I said, identical servers running FreeBSD. Here are some of > > >> the unames that I don't see timeouts on: > > >> > > >> 8.3-STABLE #2: Wed Aug 29 13:00:02 YEKT 2012 (up 187 days) > > >> 8.3-PRERELEASE #1: Thu Mar 29 16:14:11 MSK 2012 (up 15 days, previous > > >> uptime around 180 days) > > > These servers do not have 5718/5719/5720 changes. > > > > > >> 8.2-STABLE #0: Wed Dec 14 16:56:11 YEKT 2011 (up 99 days) > > > This server has the bge(4) change but it didn't trigger watchdog > > > timeouts. Does this server use the same controller? If yes, the > > > issue didn't come from bge(4) change. > > > > > How's that ? It's running even older version than previous two. I guess > > you misread the year. > > Oops, you're right. If your controller supports ASF/IPMI access please apply r248226 to stable/8 and let me know whether that makes any difference. I believe ignoring ASF/IPMI firmware is not good idea since the ASF/IPMI firmware will run regardless of hw.bge.allow_asf loader tunable configuration. You may have to set hw.bge.allow_asf=1 since it's off by default on stable/8. ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Limits on jumbo mbuf cluster allocation
Garrett Wollman wrote: > < said: > > > To be honest, I'd consider seeing a lot of non-empty receive queues > > for TCP connections to the NFS server to be an indication that it is > > near/at its load limit. (Sure, if you do netstat a lot, you will > > occasionally > > see a non-empty queue here or there, but I would not expect to see a > > lot > > of them non-empty a lot of the time.) If that is the case, then the > > question becomes "what is the bottleneck?". Below I suggest getting > > rid > > of the DRC in case it is the bottleneck for your server. > > The problem is not the DRC in "normal" operation, but the DRC when it > gets into the livelocked state. I think we've talked about a number > of solutions to the livelock problem, but I haven't managed to > implement or test these ideas yet. I have a duplicate server up now, > so I hope to do some testing this week. > > In normal operation, the server is mostly idle, and the nfsd threads > that aren't themselves idle are sleeping deep in ZFS waiting for > something to happen on disk. When the arrival rate exceeds the rate > at which requests are cleared from the DRC, *all* of the nfsd threads > will spin, either waiting for the DRC mutex or walking the DRC finding > that there is nothing that can be released yet. *That* is the > livelock condition -- the spinning that takes over all nfsd threads is > what causes the receive buffers to build up, and the large queues then > maintain the livelocked condition -- and that is why it clears > *immediately* when the DRC size is increased. (It's possible to > reproduce this condition on a loaded server by simply reducing the > tcphighwater to less than the current size.) Unfortunately, I'm at > the NFSRC_FLOODSIZE limit right now (64k), so there is no room for > further increases until I recompile the kernel. It's probably a bug > that the sysctl definition in drc3.patch doesn't check the new value > against this limit. > > Note that I'm currently running 64 nfsd threads on a 12-core > (24-thread) system. In the livelocked condition, as you would expect, > the system goes to 100% CPU utilization and the load average peaks out > at 64, while goodput goes to nearly nil. > Ok, I think I finally understand what you are referring to by your livelock. Basically, you are at the tcphighwater mark and the nfsd threads don't succeed in freeing up many cache entries so each nfsd thread tries to trim the cache for each RPC and that slows the server right down. I suspect it is the cached entries from dismounted clients that are filling up the cache (you did mention clients using amd at some point in the discussion, which implies frequent mounts/dismounts). I'm guessing that the tcp cache timeout needs to be made a lot smaller for your case. > > For either A or B, I'd suggest that you disable the DRC for TCP > > connections > > (email if you need a patch for that), which will have a couple of > > effects: > > I would like to see your patch, since it's more likely to be correct > than one I might dream up. > > The alternative solution is twofold: first, nfsrv_trimcache() needs to > do something to ensure forward progress, even when that means dropping > something that hasn't timed out yet, and second, the server code needs > to ensure that nfsrv_trimcache() is only executing on one thread at a > time. An easy way to do the first part would be to maintain an LRU > queue for TCP in addition to the UDP LRU, and just blow away the first > N (>NCPU) entries on the queue if, after checking all the TCP replies, > the DRC is still larger than the limit. The second part is just an > atomic_cmpset_int(). > I've attached a patch that has assorted changes. I didn't use an LRU list, since that results in a single mutex to contend on, but I added a second pass to the nfsrc_trimcache() function that frees old entries. (Approximate LRU, using a histogram of timeout values to select a timeout value that frees enough of the oldest ones.) Basically, this patch: - allows setting of the tcp timeout via vfs.nfsd.tcpcachetimeo (I'd suggest you go down to a few minutes instead of 12hrs) - allows TCP caching to be disabled by setting vfs.nfsd.cachetcp=0 - does the above 2 things you describe to try and avoid the livelock, although not quite using an lru list - increases the hash table size to 500 (still a compile time setting) (feel free to make it even bigger) - sets nfsrc_floodlevel to at least nfsrc_tcphighwater, so you can grow vfs.nfsd.tcphighwater as big as you dare The patch includes a lot of drc2.patch and drc3.patch, so don't try and apply it to a patched kernel. Hopefully it will apply cleanly to vanilla sources. Tha patch has been minimally tested. If you'd rather not apply the patch, you can change NFSRVCACHE_TCPTIMEOUT and set the variable nfsrc_tcpidempotent == 0 to get a couple of the changes. (You'll have to recompile the kernel for these changes to take effect.) Good luck with it, rick > -GAWollman > _
Re: Limits on jumbo mbuf cluster allocation
< said: > Basically, this patch: > - allows setting of the tcp timeout via vfs.nfsd.tcpcachetimeo > (I'd suggest you go down to a few minutes instead of 12hrs) > - allows TCP caching to be disabled by setting vfs.nfsd.cachetcp=0 > - does the above 2 things you describe to try and avoid the livelock, > although not quite using an lru list > - increases the hash table size to 500 (still a compile time setting) > (feel free to make it even bigger) > - sets nfsrc_floodlevel to at least nfsrc_tcphighwater, so you can > grow vfs.nfsd.tcphighwater as big as you dare Thanks, this looks very good. One quibble with the last bit: I'd do that in a sysctl() handler rather than checking it every time through. If somebody uses a debugger rather than sysctl to change tcphighwater, they deserve what's coming to them. Also, I might suggest adding a counter for how many times we had to go through the "try harder" phase, so that the sysadmin has some indication that the defaults need adjustment. I will test this out later this week and see how it performs. I have a user who has been able to reproducibly clobber servers before, so if he has time and cycles available it should be pretty easy to tell whether it's working or not. -GAWollman ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: kern/176884: [re] re0 flapping up/down
Old Synopsis: re0 flapping up/down New Synopsis: [re] re0 flapping up/down Responsible-Changed-From-To: freebsd-bugs->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Wed Mar 13 05:30:13 UTC 2013 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=176884 ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
NDP prefix list locking
Hi Mark, In the patch that you provided to freebsd-net, http://lists.freebsd.org/pipermail/freebsd-net/2013-February/034695.html, I have a question. nd6_timer() acquires the write lock using ND_DEFRTR_WLOCK. However, defrouter_select again tries to acquire a read lock using ND_DEFRTR_RLOCK. This will lead to a deadlock as the read lock can't be acquired as the write lock is already acquired. The solution is to check if the write lock is already acquired. If not the read lock can be acquired, else there is no need to acquire the read lock. Can you share your thoughts on the same. Dheeraj ___ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
