On Nov 16, 2012, at 3:31 PM, Kyle Mestery (kmestery) <kmest...@cisco.com> wrote: > On Nov 16, 2012, at 3:18 PM, Jesse Gross <je...@nicira.com> wrote: >> On Fri, Nov 16, 2012 at 1:00 PM, Kyle Mestery (kmestery) >> <kmest...@cisco.com> wrote: >>> On Nov 15, 2012, at 3:32 PM, Kyle Mestery (kmestery) <kmest...@cisco.com> >>> wrote: >>>> On Nov 15, 2012, at 3:13 PM, Kyle Mestery (kmestery) <kmest...@cisco.com> >>>> wrote: >>>>> On Nov 15, 2012, at 1:03 PM, Kyle Mestery (kmestery) <kmest...@cisco.com> >>>>> wrote: >>>>>> Jesse: >>>>>> >>>>>> I modified the source port hashing for the VXLAN patch I submitted a few >>>>>> days ago, >>>>>> but I've noticed when using the upstream source port hashing routine, >>>>>> performance >>>>>> drops off by 3.5 times when using iperf between two VMs. From what I can >>>>>> tell, it >>>>>> has to be that all skbuffs coming into the VXLAN tunnel have not already >>>>>> had their >>>>>> rxhash set, and this function is what's killing performance. Let me >>>>>> share the details: >>>>>> >>>>> I think I figured this out. The upstream source port selection algorithm >>>>> is exploding flows >>>>> in the fast path. Here are iperf runs with both and subsequent "ovs-dpctl >>>>> dump-flows" >>>>> commands for comparison. The first one is with the upstream version, the >>>>> second is >>>>> with the one in my patch. Note that I just piped "ovs-dpctl dump-flows" >>>>> into wc to >>>>> summarize the flow count. >>>>> >>>>> Upstream verison: >>>>> [root@linux-br ~]# iperf -c 10.1.2.14 && ovs-dpctl dump-flows | wc >>>> >>>> >>>> Figured this out, fixing it now, will repost the patch with only this >>>> change soon. >>>> >>> >>> So after looking at this, the upstream source port selection function will >>> cause an explosion >>> of fast path flows due to an ever changing skbuff->rxhash. By using the >>> flow->hash, we >>> don't see this problem. Jesse, any comments on this particular issue? I >>> think using the >>> upstream function will allow for greater spreading across links depending >>> on the hashing >>> algorithm used on upstream switches, but will cause this flow explosion on >>> the host itself. >> >> Generally speaking, the OVS flow extraction pulls out more fields than >> are usually used by skb_get_rxhash() so I'm surprised that it would >> have this effect (of course, neither is supposed to be so fine grained >> that it breaks down a single 'real' flow). It sounds to me like there >> is a bug that is pulling in random data that's not supposed to be part >> of the flow if it changes on a per packet basis. >> > Right, I agree. I'll keep digging to see what I can find. > I figured this one out. What was throwing the hash off was the UDP source port for the VXLAN packets was apparently random garbage. Clearing it to zero before calling the routine to acquire one (which calls skb_get_rxhash() itself) fixed the problem. I'll submit the latest patch (which has this as the only change) soon.
Thanks, Kyle >> Do you know if you are getting a value from skb_get_rxhash() or is it >> falling back to the Ethernet headers? > > It's definitely getting a value from skb_get_rxhash(), because I replaced the > ethernet headers fallback with using OVS_CB(skb)->flow->hash, which would > have resulted in the better performance, but it the performance was still > awful. > Just to be sure, I added some debug to verify, and it's returning a value. > > Thanks, > Kyle > _______________________________________________ > dev mailing list > dev@openvswitch.org > http://openvswitch.org/mailman/listinfo/dev _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
