I believe that this problem has been solved in the latest master and various release branches with commit: 8e21fbb9d8dd495630350ef1831809d6249ee022 "datapath/flow_netlink: Fix NDP flow mask validation"
However, this is specific to IPv6 neighbor discovery packets, so while it should avoid the errors seen below I doubt that it will fix the other ones in the thread. Therefore, if problems continue, please post the logs using master, which has additional logging. On Thu, Jul 3, 2014 at 1:34 PM, Bannai, Vinay <vban...@paypal.com> wrote: > Hello Jesse, > > We are running a large Openstack deployment with several hundred > hypervisors running Openstack Havana with OVS. There are currently around > several thousand VM's running in this environment. > Here are the details: > > root@pxxx:~# ovs-vsctl --version > ovs-vsctl (Open vSwitch) 2.0.1.32944 > Compiled Jan 28 2014 23:52:12 > root@xxx:~# > > Some of the VM's are not reachable. When we further debugged the VM and > the hypervisor it is running on, we seeing the following messages in > ovs-vswitchd.log > > ovs-vswitchd.log.9.gz:2014-06-25T08:37:21.975Z|4012804|dpif|WARN|system@ovs > -system: failed to put[create] (Invalid argument) > skb_priority(0),tunnel(tun_id=0xbe3e000003ed68,src=10.87.194.102,dst=10.42. > 224.36,tos=0x0,ttl=57,flags(csum,key)),in_port(3),skb_mark(0),eth(src=fa:16 > :3e:6e:d4:78,dst=33:33:ff:52:92:ae),eth_type(0x86dd),ipv6(src=fe80::e853:17 > 6d:3ada:3b2b/::,dst=ff02::1:ff52:92ae/::,label=0/0,proto=58/0xff,tclass=0/0 > ,hlimit=255/0,frag=no/0xff),icmpv6(type=135,code=0),nd(target=fe80::31ae:d0 > 00:6f52:92ae/::,sll=fa:16:3e:6e:d4:78/ff:ff:ff:ff:ff:ff), actions:drop > ovs-vswitchd.log.9.gz:2014-06-25T10:11:31.540Z|4014779|dpif|WARN|system@ovs > -system: failed to put[create] (Invalid argument) > skb_priority(0),tunnel(tun_id=0x16329000003ed68,src=10.87.196.10,dst=10.42. > 224.36,tos=0x0,ttl=57,flags(csum,key)),in_port(3),skb_mark(0),eth(src=54:52 > :00:c1:fb:a0,dst=33:33:ff:da:3b:2b),eth_type(0x86dd),ipv6(src=fe80::31ae:d0 > 00:6f52:92ae/::,dst=ff02::1:ffda:3b2b/::,label=0/0,proto=58/0xff,tclass=0/0 > ,hlimit=255/0,frag=no/0xff),icmpv6(type=135,code=0),nd(target=fe80::e853:17 > 6d:3ada:3b2b/::,sll=54:52:00:c1:fb:a0/ff:ff:ff:ff:ff:ff), actions:drop > ovs-vswitchd.log.9.gz:2014-06-25T10:41:31.585Z|4015411|dpif|WARN|system@ovs > -system: failed to put[create] (Invalid argument) > skb_priority(0),tunnel(tun_id=0x16329000003ed68,src=10.87.196.10,dst=10.42. > 224.36,tos=0x0,ttl=57,flags(csum,key)),in_port(3),skb_mark(0),eth(src=54:52 > :00:c1:fb:a0,dst=33:33:ff:da:3b:2b),eth_type(0x86dd),ipv6(src=fe80::31ae:d0 > 00:6f52:92ae/::,dst=ff02::1:ffda:3b2b/::,label=0/0,proto=58/0xff,tclass=0/0 > ,hlimit=255/0,frag=no/0xff),icmpv6(type=135,code=0),nd(target=fe80::e853:17 > 6d:3ada:3b2b/::,sll=54:52:00:c1:fb:a0/ff:ff:ff:ff:ff:ff), actions:drop > ovs-vswitchd.log.9.gz:2014-06-25T12:26:31.726Z|4017607|dpif|WARN|system@ovs > -system: failed to put[create] (Invalid argument) > skb_priority(0),tunnel(tun_id=0xbe3e000003ed68,src=10.87.194.102,dst=10.42. > 224.36,tos=0x0,ttl=57,flags(csum,key)),in_port(3),skb_mark(0),eth(src=fa:16 > :3e:6e:d4:78,dst=33:33:ff:52:92:ae),eth_type(0x86dd),ipv6(src=fe80::e853:17 > 6d:3ada:3b2b/::,dst=ff02::1:ff52:92ae/::,label=0/0,proto=58/0xff,tclass=0/0 > ,hlimit=255/0,frag=no/0xff),icmpv6(type=135,code=0),nd(target=fe80::31ae:d0 > 00:6f52:92ae/::,sll=fa:16:3e:6e:d4:78/ff:ff:ff:ff:ff:ff), actions:drop > > > When we try to do ovs-dpctl dump-flows and grep for the MAC address > associated with the VMs, we see the flow has action set to drop. > Tcpdump on the tap interface of the VM' shows that it is sending DHCP > packets but the ovs bridge is dropping the flow. > > Vinay > > eBay/PayPal > > -----Original Message----- > From: Jesse Gross <je...@nicira.com> > Date: Tuesday, July 1, 2014 at 5:46 PM > To: Federico Iezzi <fie...@enter.it> > Cc: "discuss@openvswitch.org" <discuss@openvswitch.org> > Subject: Re: [ovs-discuss] failed to put[create][modify] > >>Thanks for the additional information. >> >>I'm still having a hard time seeing how a flow could be invalid >>without generating any log messages so I just sent out a patch to >>hopefully cover the remaining cases (CC'ed to you). The patch is >>against master - is it possible for you to try it and report any >>messages that show up in dmesg? >> >>Also, can you send a copy of what both your kernel and userspace flow >>tables look like when this happens (ovs-dpctl dump-flows and ovs-ofctl >>dump-flows <BR> respectively)? >> >>On Mon, Jun 30, 2014 at 1:08 PM, Federico Iezzi <fie...@enter.it> wrote: >>> >>> Hi Jesse, >>> >>> >>> Here we go, I¹m really sorry for this huge delay but I had some trouble. >>> So I tested all OVS version from 2.1.0 (under advise of Salvatore >>>Orlando) to 2.3, all of this with kernel 3.11 to 3.13 and I had the >>>following result. >>> >>> With Kernel 3.12 or 3.13 AND OVS 2.1 (datapath come from Kernel) or >>>with OVS 2.2 2.3 AND Kernel 3.11 (datapath come from OVS) I have the >>>error ³failed to put² into log. During this message I have packet loss >>>and retransmission. >>> With Kernel 3.11 and OVS compiled from branch-2.1 I don¹t have any >>>issue. The datapath came from OVS. >>> >>> The following commit means my OVS version, because I compile and >>>generate the DEB file from GIT branch: >>> >>> - OVS 2.1 -> >>>https://github.com/openvswitch/ovs/commit/daded0421c7b746c4ce54b6f7906191 >>>8ed440edf >>> - OVS 2.2 -> >>>https://github.com/openvswitch/ovs/commit/d90edfa1251b04dea3820af126c1852 >>>7de2aa12e >>> - OVS 2.3 -> >>>https://github.com/openvswitch/ovs/commit/803485697dd690490395a81685dd547 >>>bf38810af >>> >>> I¹m sorry but I didn¹t have any time to test OVS 2.0.1 LTS with kernel >>>3.11. >>> >>> Also as OS I¹m using Ubuntu Server 12.04.4 x64 full upgrade, as Kernel >>>I tested the following version: >>> >>> Kernel 3.11 >>> >>>-http://changelogs.ubuntu.com/changelogs/pool/main/l/linux/linux_3.11.0-1 >>>9.33/ >>> >>>-http://changelogs.ubuntu.com/changelogs/pool/main/l/linux/linux_3.11.0-2 >>>4.41/ >>> >>> Kernel 3.13 >>> - >>>http://changelogs.ubuntu.com/changelogs/pool/main/l/linux/linux_3.13.0-27 >>>.50/ >>> - >>>http://changelogs.ubuntu.com/changelogs/pool/main/l/linux/linux_3.13.0-29 >>>.53/ >>> >>> Kernel 3.12, the vanilla version ³3.12.23² provided by Linus from >>>kernel.org >>> >>> All my hardware is based on Intel Xeon E5-2620 v2, E5-2630, E5-2650 v2 >>>and E5-2670 v2. >>> >>> The following are my version of the build dependencies >>> dpkg -l | grep -E >>>"(debhelper|autoconf|automake|libssl-dev|bzip2|openssl|graphviz|python-al >>>l|procps|python-qt4)" >>> >>> ii autoconf 2.68-1ubuntu2 >>> automatic configure script builder >>> ii automake 1:1.11.3-1ubuntu2 >>> Tool for generating GNU Standards-compliant Makefiles >>> ii automake1.10 1:1.10.3-1ubuntu1 >>> A tool for generating GNU Standards-compliant Makefiles >>> ii bzip2 1.0.6-1 >>> high-quality block-sorting file compressor - utilities >>> ii debhelper 9.20120115ubuntu3 >>> helper programs for debian/rules >>> ii dh-apparmor 2.7.102-0ubuntu3.9 >>> AppArmor debhelper routines >>> ii graphviz 2.26.3-10ubuntu1.1 >>> rich set of graph drawing tools >>> ii libgnutls-openssl27 2.12.14-5ubuntu3.8 >>> GNU TLS library - OpenSSL wrapper >>> ii libssl-dev 1.0.1-4ubuntu5.15 >>> SSL development libraries, header files and documentation >>> ii openssl 1.0.1-4ubuntu5.15 >>> Secure Socket Layer (SSL) binary and related cryptographic tools >>> ii procps 1:3.2.8-11ubuntu6.3 >>> /proc file system utilities >>> ii python-all 2.7.3-0ubuntu2.2 >>> package depending on all supported Python runtime versions >>> ii python-openssl 0.12-1ubuntu2.1 >>> Python wrapper around the OpenSSL library >>> ii python-qt4 4.9.1-2ubuntu1 >>> Python bindings for Qt4 >>> >>> As Ethernet NIC I¹m using Mellanox Connect X-3 EN Pro and only EN 10 >>>and 40/56Gbps with latest OFED and Firmware. >>> As Ethernet Switch I¹m using Mellanox SX1024 with a special OS and >>>Firmware version. >>> >>> During the issue I don¹t any problem in my internal network >>>communication (ICMP, ICMPv6, TCP and UDP). >>> >>> If you need more detail please tell me. >>> If you think to be found a workaround or a probable solution tell me >>>and I¹ll try to provide some test nodes. >>> >>> Regards, >>> Federico >>> >>> >>> >>> >>> >>> >>> Il giorno 27/giu/2014, alle ore 18:23, Jesse Gross <je...@nicira.com> >>>ha scritto: >>> >>> On Thu, Jun 26, 2014 at 12:33 PM, Federico Iezzi <fie...@enter.it> >>>wrote: >>> >>> Eugene, >>> >>> >>> >>> The only way that I found is roll-back to a Kernel version < 3.12. >>> >>> Today I did many tests with vanilla Kernel 3.11.10 and Canonical >>>3.11.0-24 I haven't anymore problem using datapath from OVS source. I¹m >>>using VXLAN so I can¹t test OVS with kernel module because only from >>>kernel 3.12 there is VXLAN tunnel support. At the beginning >>> I said that I had the problem with kernel 3.11 and OVS datapath from >>>source but this consideration come from a wrong analysis. ** My fault, >>>sorry guys! ** >>> >>> From my tests during the message ³failed to put² I have packet loss. >>> >>> >>> >>> Tomorrow I going to test the latest OVS LTS 2.0.1 with Kernel >=3.12 >>>but I'm pretty sure that I will have the same problem. >>> >>> >>> >>> >>> My guess is that this is related only to the OVS datapath and not the >>> >>> rest of the running kernel since flow parsing doesn't really have any >>> >>> interaction with the rest of the network stack. It sounds like it also >>> >>> it's not impacted by userspace version since it sounds like you have >>> >>> tested several versions with the same results. >>> >>> >>> >>> Can you clarify exactly which OVS kernel versions you have tested and >>> >>> the results? It would also be helpful if you can hold other factors >>> >>> constant as it is a little bit hard to follow what is going on. >>> >>> >>> >>> >>> >>> >>> >>> >>_______________________________________________ >>discuss mailing list >>discuss@openvswitch.org >>http://openvswitch.org/mailman/listinfo/discuss > _______________________________________________ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
