I just sent a fix for this issue in OVS master to the OVS dev list. You can see the patches also in patchwork:
http://patchwork.ozlabs.org/patch/610337/ http://patchwork.ozlabs.org/patch/610338/ http://patchwork.ozlabs.org/patch/610339/ http://patchwork.ozlabs.org/patch/610340/ Jarno > On Apr 13, 2016, at 12:36 PM, Maurya, Alok Kumar (HP Networking) > <alok-kumar.mau...@hpe.com> wrote: > > Hi Jarno, > > I have 4 CPU , 8 GB RAM VM . > I am running Ubuntu 14.04 on the VM . > > With 200K flows , yes in my system also its 2-3 minutes , however when > I have around 350K flows , it took 29-30 mins (sorry the data which I > had given was with 350K flows not with 200K flows) . > When 350 Flows ae there in the system try to delete 17K flows in batch . > > I can try to remove in_port=1 from match and see if that makes any > difference , but flows will have same in_port as we expect packets > with match to come from this port . > > > ### script #### > echo "Adding 200K Flows" > date > for v in {1..20} > > do > > for i in {1..250} > do > > for j in {1..70} > do > > ovs-ofctl add-flow br-int > hard_timeout=0,idle_timeout=0,priority=10,ip,cookie=0x5c38cfa3a668$v,table=0,dl_dst=fa:16:3e:af:69:a3,dl_vlan=$v,nw_src=9.16.$i.$j/32,in_port=1,actions=output:1 > > done > done > > done > date > echo "Flows are added" > > echo "Dump-Flows" > ovs-ofctl dump-flows br-int |wc > date > > #### end script ##### > > Thanks & Regards, > Alok > > -----Original Message----- > From: Jarno Rajahalme [mailto:ja...@ovn.org] > Sent: Thursday, April 14, 2016 12:44 AM > To: Maurya, Alok Kumar (HP Networking) <alok-kumar.mau...@hpe.com> > Cc: Ben Pfaff <b...@ovn.org>; discuss@openvswitch.org > Subject: Re: [ovs-discuss] OVS 2.4 and 2.4 Flow deletion issue > > >> On Apr 12, 2016, at 1:05 AM, Maurya, Alok Kumar (HP Networking) >> <alok-kumar.mau...@hpe.com> wrote: >> >> Thanks Jarno and Ben for looking into this issue . >> >> I had tried to backtrace this from 2.3 to 2.4 . >> >> >> I found that I had started seeing this issue after >> f2c214029e1ac0c9bc0f7c2013fe178a6b08dd7f commit , however in that >> ovs-ofctl del-flows br-int dl_vlan=1 >> > > I was able to reproduce the issue with the exact flow pattern you reported, > thanks! > > The issue is that each of the flows has exactly the same metadata portion in > the match (in_port=1). I’m looking into how to alleviate this, but in the > meanwhile you could try to remove fields from the matches that make no > difference to the outcome of the match. > >> took long time to execute ( more than 30 mins) . >> > > On my 4-core VM the delete of the 5% of the flows took almost two minutes, > so 30 minutes sounds like a long time. Could you tell a little bit more about > your setup where you are executing this? OS, memory, # of cores, etc? > >> In case of flow delete now , looks like ovs-vswitchd is sending >> "barrier-reply" even before flow deletion is completed . >> > > The barrier-reply is returned as soon as the flows have been made invisible, > but some of the clean-up work is postponed and can safely happen while > following OpenFlow messages are processed. However, in this case the clean-up > takes a mutex for a duration of removing the 10000 rules, and due to all the > 200000 rules having the exactly same metadata match this now takes a long > time. > > Regards, > > Jarno > >> Thanks & Regards, >> Alok >> >> -----Original Message----- >> From: Jarno Rajahalme [mailto:ja...@ovn.org] >> Sent: Tuesday, April 12, 2016 7:02 AM >> To: Ben Pfaff <b...@ovn.org> >> Cc: Maurya, Alok Kumar (HP Networking) <alok-kumar.mau...@hpe.com>; >> discuss@openvswitch.org >> Subject: Re: [ovs-discuss] OVS 2.4 and 2.4 Flow deletion issue >> >> I tried with a bit simpler flows, like this: >> >> ovs-ofctl add-flow br-int >> hard_timeout=0,idle_timeout=0,priority=10,ip,cookie=0x$v,table=0,dl_vl >> an=$v,nw_src=9.16.$i.$j,actions=output:1 >> >> And while adding them one-by-one in my VM took 12 minutes (my dev VM has >> been very slow recently), the runtime of the deletion was not noticeable, >> i.e., a following flow dump returned right away. Deleting all 200 flows took >> maybe 2-3 seconds. >> >> I’m trying again with the exact flow reported, but I’d be surprised if the >> additional match fields would make a difference. >> >> This on tip of master. I’ll have to try again with releases 2.4 and 2.5. >> >> Jarno >> >>> On Apr 11, 2016, at 5:15 PM, Jarno Rajahalme <ja...@ovn.org> wrote: >>> >>> I’m looking into this, >>> >>> Jarno >>> >>>> On Apr 11, 2016, at 3:41 PM, Ben Pfaff <b...@ovn.org> wrote: >>>> >>>> Jarno, it seems likely that this has something to do with the >>>> transactional classifier implementation. Do you think so? Do you >>>> have any idea why deletions would be so slow? >>>> >>>> Thanks, >>>> >>>> Ben. >>>> >>>> On Mon, Apr 11, 2016 at 09:51:45PM +0000, Maurya, Alok Kumar (HP >>>> Networking) wrote: >>>>> Hi All, >>>>> >>>>> I am seeing that in OVS 2.4 and ovs 2.5 Bulk flow deletion is >>>>> taking huge time more than (30 mins) compared to ovs 2.3 where >>>>> it used to take 1-2 secs . >>>>> >>>>> Below are steps which I am following to reproduce the issue : >>>>> >>>>> >>>>> 1. Add 200K Flows >>>>> >>>>> ####### Sample Script to add 200 K Flows ###### >>>>> >>>>> >>>>> echo "Adding 200K Flows" >>>>> date >>>>> for v in {1..20} >>>>> >>>>> do >>>>> >>>>> for i in {1..250} >>>>> do >>>>> >>>>> for j in {1..40} >>>>> do >>>>> >>>>> ovs-ofctl add-flow br-int >>>>> hard_timeout=0,idle_timeout=0,priority=10,ip,cookie=0x5c38cfa3a668$ >>>>> v >>>>> ,table=0,dl_dst=fa:16:3e:af:69:a3,dl_vlan=$v,nw_src=9.16.$i.$j/32,i >>>>> n >>>>> _port=1,actions=output:1 >>>>> >>>>> done >>>>> done >>>>> >>>>> done >>>>> date >>>>> echo "Flows are added" >>>>> >>>>> echo "Dump-Flows" >>>>> ovs-ofctl dump-flows br-int |wc >>>>> date >>>>> >>>>> >>>>> ######## End Script #### >>>>> >>>>> >>>>> 2. Try to delete 10 k Flows () >>>>> >>>>> ovs-ofctl del-flows br-int dl_vlan=1 >>>>> >>>>> 3. Above command gets executed successfully , but looks like >>>>> deletion keep on happening in background . >>>>> 4. It takes more than 30 mins to complete it , during this >>>>> period if anyone executes any of below command it hangs without >>>>> any output . >>>>> >>>>> >>>>> ovs-ofctl dump-flows br-int >>>>> >>>>> or >>>>> >>>>> ovs-ofctl show br-int >>>>> >>>>> >>>>> 5. This issue is not seen in ovs 2.3 >>>>> >>>>> >>>>> Is there any solution for this issue ? >>>>> >>>>> >>>>> >>>>> >>>>> Thanks & Regards, >>>>> Alok >>>> >>>>> _______________________________________________ >>>>> discuss mailing list >>>>> discuss@openvswitch.org >>>>> http://openvswitch.org/mailman/listinfo/discuss >>>> >>> >> > _______________________________________________ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
