Hi Daniele, I have a way to mitigate the problem by stopping rte_eth_rx_burst until all the rules have settled. I do this with an options: argument in other_config (amongst other arguments using this free form field - like dedup and defrag features i am experimenting with) My rules are static so this is ok for me to drop packets while rules are all being added as configuration is an infrequent event for my implementation.
but occasionally after setting port options on a netdev using pmd328 via ovsdb 'options' in other_config i get: 2015-09-26T19:41:14.186Z|00297|ovs_rcu|WARN|blocked 1000 ms waiting for pmd327 to quiesce from which it never recovers but exponentially backs off waiting. Cheers, Dave On Fri, Sep 25, 2015 at 10:17 PM, David Evans <davidjoshuaev...@gmail.com> wrote: > yes i suspect a packet leak. > As the drop count when it stops is usually about 520K packets. :( > Where do i start looking in the source for that kind of symptom? > > On Fri, Sep 25, 2015 at 9:58 PM, David Evans <davidjoshuaev...@gmail.com> > wrote: > >> Hi Daniele! >> Thank you for your reply. >> I find that it reliably starts too if i have 6 or less NIC ports. >> >> Any more than 6 ports (even if there is only one receiving traffic) and >> the one receivng traffic will just keep upping the rx_errors. >> >> Ovs is automatically adding 11pmd's because i have a FFF mask for core >> usage and it is using 10G worth of 1G hugepages. I have about 524288 9k >> packet buffers on one numa node memory. and 262K on the other. >> >> If i have my 12 ports included, and if i wait till it has completed >> starting up before sending any traffic it easily copes with at least >> 80+Gbps throughput. But alas if traffic is running on startup i just get >> rx_errors. >> >> On my bridge i add a drop all flow at low priority on table 3 and have >> various other rules in table 0, 1 and 2 and 3 for >> detection/filtering/forwarding. The problem is exacerbated if i add more >> bridges and connect the dpdk ports to other bridges in the same switch. >> >> Could it have leaked out all it's packet buffers? >> Or the threads be busy wait somewhere else because of a race condition >> with all those PMD's? >> Maybe a thread synchronize, or quiescing issue? >> >> Can you think of a neat way i can track thread synchronization & packet >> usage? >> >> btw i am using ovs 2.3.2+29bae5412ea3c6a1e20d79afd25c11b968dffa05-r0 >> >> Regards, >> >> Dave. >> >> >> On Fri, Sep 25, 2015 at 11:50 AM, Daniele Di Proietto < >> diproiet...@vmware.com> wrote: >> >>> I regularly start ovs with dpdk while traffic is flowing. >>> >>> Is there anything suspicious in the ovs-vswitchd log? >>> >>> Have you tried other DPDK apps (e.g. l2fwd)? >>> >>> >>> On 25/09/2015 17:28, "David Evans" <davidjoshuaev...@gmail.com> wrote: >>> >>> >Hi all, >>> > >>> > >>> >Has anyone seen that when starting ovs (with dpdk) that the port will >>> >show many rx-errors and the device will stop receiving traffic? >>> >Is there a mitigation for this. >>> >I have tried adding drop all flows to the bridge before all my other >>> >flows are set up. >>> >I have tried disabling the port rx too. But still get this behaviour >>> when >>> >starting ovs if there is traffic running to it. >>> > >>> > >>> >Everything works fine if I have no traffic running to the dpdk port at >>> >startup and initiate the traffic after the switch is steady and >>> >configured. >>> > >>> > >>> >Regards, >>> >Dave. >>> > >>> > >>> >>> >> >
_______________________________________________ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
