Thanks Sugesh, i was missing the rxqueue. now its correctly spreading across the cores.
Thanks Praveen On Tue, Dec 1, 2015 at 11:32 AM, Chandran, Sugesh <sugesh.chand...@intel.com > wrote: > Praveen, > > > > You need to pin the PMD threads on specific CPU cores explicitly and > create that many number of RX queues to share the load across cores. > > Please refer Performance Tuning section in INSTALL.DPDK.md for more > details. > > > > > > *Regards* > > *_Sugesh* > > > > *From:* Praveen MANKARA RADHAKRISHNAN [mailto:praveen.mank...@6wind.com] > *Sent:* Monday, November 30, 2015 1:36 PM > *To:* Chandran, Sugesh <sugesh.chand...@intel.com> > *Cc:* b...@openvswitch.org; Daniel Virassamy <daniel.virass...@6wind.com> > > *Subject:* Re: [ovs-discuss] Test with vxlan using ovs-dpdk (2.4) > > > > Hi Sugesh, > > > > I made this scenario work with 2 physical machines. > > > > I am not seeing packets distributed across cores when i send multiple > flows from ixia. > > it seems all the packets from a port is handled by only a single core. its > not spreading across the cores. > > is there any specific configuration is needed for enabling the packet > spreading across different cores. > > > > given below is the description of my test set up. > > > > 2 servers are connected via 1 10Gb interface. > > each server 1 ixia port is connected. > > > > from the ixia vlan traffic is coming to dpdk port 0 of server 1. it strips > valn go to a vxlan tunnel through dpdk port 1 to the next server. there > vxla is stripped and through the next going to the ixia port2. > > > > same for reverse direction also. > > > > when i tried the test i have the following observations. > > packets are not distributing across the cores. because of this performance > is not scaling. > > is there any configuration i should specifically do to have the scaling. > > > > Please find the details about my test. > > > > Configuration: > > ----------------------- > > > > Server 1 > > ------------ > > > > ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev > > ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk > > ovs-vsctl add-port br0 vxlan0 -- set interface vxlan0 type=vxlan > options:remote_ip=10.200.0.6 > > > > ovs-vsctl add-br br1 -- set bridge br1 datapath_type=netdev > > ovs-vsctl add-port br1 dpdk1 -- set Interface dpdk1 type=dpdk > > > > ip link set br0 up > > ip link set br1 up > > ip a a 10.200.0.5/24 dev br1 > > > > ovs-ofctl add-flow br0 in_port=1,actions=strip_vlan,output:2 > > ovs-ofctl add-flow br0 in_port=2,actions=output:1 > > > > Server 2 > > ------------ > > > > ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev > > ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk > > ovs-vsctl add-port br0 vxlan0 -- set interface vxlan0 type=vxlan > options:remote_ip=10.200.0.5 > > > > ovs-vsctl add-br br1 -- set bridge br1 datapath_type=netdev > > ovs-vsctl add-port br1 dpdk1 -- set Interface dpdk1 type=dpdk > > > > ip link set br0 up > > ip link set br1 up > > ip a a 10.200.0.6/24 dev br1 > > > > ovs-ofctl add-flow br0 in_port=1,actions=strip_vlan,output:2 > > ovs-ofctl add-flow br0 in_port=2,actions=output:1 > > > > 1C_2T ~~~~~~~~ ovs-vsctl set Open_vSwitch . > other_config:pmd-cpu-mask=2000002 > > > > [root@controller ~]# ovs-appctl dpif-netdev/pmd-stats-show main thread: > emc hits:0 megaflow hits:0 miss:0 lost:0 polling cycles:844386 (100.00%) > processing cycles:0 (0.00%) pmd thread numa_id 1 core_id 1: emc > hits:140438952 megaflow hits:126 miss:2 lost:0 polling cycles:11260105290 > (21.84%) processing cycles:40302774189 (78.16%) avg cycles per packet: > 367.15 (51562879479/140439080) avg processing cycles per packet: 286.98 > (40302774189/140439080) pmd thread numa_id 1 core_id 25: emc hits:148663170 > megaflow hits:126 miss:2 lost:0 polling cycles:11918926350 (23.16%) > processing cycles:39554751984 (76.84%) avg cycles per packet: 346.24 > (51473678334/148663298) avg processing cycles per packet: 266.07 > (39554751984/148663298) > > > > Here traffic is coming from both ixia ports so bothcores are getting > packets. > > > > Changed the threads to 2C_4T : > > > > [root@redhat7 ~]# ovs-vsctl set Open_vSwitch . > other_config:pmd-cpu-mask=a00000a > > > > [root@redhat7 ~]# ovs-appctl dpif-netdev/pmd-stats-show > > main thread: > > emc hits:0 > > megaflow hits:0 > > miss:0 > > lost:0 > > polling cycles:1105692 (100.00%) > > processing cycles:0 (0.00%) > > pmd thread numa_id 1 core_id 1: > > emc hits:174248214 > > megaflow hits:126 > > miss:2 > > lost:0 > > polling cycles:41551018158 (49.30%) > > processing cycles:42726992931 (50.70%) > > avg cycles per packet: 483.67 (84278011089/174248342) > > avg processing cycles per packet: 245.21 (42726992931/174248342) > > pmd thread numa_id 1 core_id 3: > > emc hits:182375144 > > megaflow hits:126 > > miss:2 > > lost:0 > > polling cycles:39728724567 (46.68%) > > processing cycles:45381262518 (53.32%) > > avg cycles per packet: 466.68 (85109987085/182375272) > > avg processing cycles per packet: 248.83 (45381262518/182375272) > > pmd thread numa_id 1 core_id 27: > > emc hits:0 > > megaflow hits:0 > > miss:0 > > lost:0 > > pmd thread numa_id 1 core_id 25: > > emc hits:0 > > megaflow hits:0 > > miss:0 > > lost:0 > > [root@redhat7 ~]# > > > > As you can see still only 2 cores were used. > > > > > > > > The same is for the other server also. > > from the ixia i have sending multiple flows more than 50 flows by changing > the source ip. > > the problem is its as if 1 port is bind to the 1 core. its not > distributing the packets. > > > > > > Thanks > > Praveen > > > > On Tue, Nov 3, 2015 at 11:33 AM, Praveen MANKARA RADHAKRISHNAN < > praveen.mank...@6wind.com> wrote: > > Hi Sugesh, > > > > Thanks for the information. > > by adding the flow in reverse direction OVS learns the arp and the > scenario works correctly. > > > > Thanks > > Praveen > > > > > > On Wed, Oct 28, 2015 at 5:37 PM, Chandran, Sugesh < > sugesh.chand...@intel.com> wrote: > > Hi Praveen, > > Thank you sharing your setup details! > > It looks like the issue here is ovs doesn’t have the arp entry for remote > end point i.e 10.200.0.6. > > Can you let OVS to learn the mac address for 10.200.0.6?? You can send a > arp reply from Ixia port-2 to make it learn this value. Alternatively you > can do the same with tnl/arp/set as well, but it looks like to me that the > command not yet available on your OVS version. > > And also please remove the static flow rule that configured lately. > > > > You can verify the rules being used in the datapath with > > ovs-appctl dpctl/dump-flows > > > > There must be rules for `tnl_push` for pushing the vxlan header and also > for forward the packet to physical interface. > > > > *Regards* > > *_Sugesh* > > > > *From:* Praveen MANKARA RADHAKRISHNAN [mailto:praveen.mank...@6wind.com] > *Sent:* Wednesday, October 28, 2015 11:17 AM > > > *To:* Chandran, Sugesh > *Cc:* b...@openvswitch.org; Daniel Virassamy > *Subject:* Re: [ovs-discuss] Test with vxlan using ovs-dpdk (2.4) > > > > Hi Sugesh, > > > > Thanks. but its still the same.It seems the packets are not getting > forwarded from the vxlan interface. > > Please find the logs attached. > > > > root@ubuntu1404:~# ovs-vsctl show > > e2127cda-4b01-4020-8a6c-b5ff6be8f2fa > > Bridge "br0" > > Port "dpdk1" > > Interface "dpdk1" > > type: dpdk > > Port "vxlan0" > > Interface "vxlan0" > > type: vxlan > > options: {remote_ip="10.200.0.6"} > > Port "br0" > > Interface "br0" > > type: internal > > Bridge "br1" > > Port "dpdk0" > > Interface "dpdk0" > > type: dpdk > > Port "br1" > > Interface "br1" > > type: internal > > > > > > root@ubuntu1404:~# ovs-ofctl dump-flows br0 > > NXST_FLOW reply (xid=0x4): > > cookie=0x0, duration=341.284s, table=0, n_packets=1039, n_bytes=68574, > idle_age=0, dl_vlan=1,dl_dst=02:11:22:44:55:22 actions=strip_vlan,output:2 > > cookie=0x0, duration=427.090s, table=0, n_packets=8, n_bytes=648, > idle_age=324, priority=0 actions=NORMAL > > > > > > root@ubuntu1404:~# ovs-ofctl dump-flows br1 > > NXST_FLOW reply (xid=0x4): > > cookie=0x0, duration=66.576s, table=0, n_packets=0, n_bytes=0, > idle_age=66, in_port=LOCAL actions=output:1 > > cookie=0x0, duration=401.153s, table=0, n_packets=8, n_bytes=648, > idle_age=325, priority=0 actions=NORMAL > > > > root@ubuntu1404:~# ovs-appctl tnl/arp/show > > IP MAC Bridge > > ============================================= > > 10.200.0.5 b8:ca:3a:6f:16:c8 br1 > > > > root@ubuntu1404:~# ovs-ofctl show br0 > > OFPT_FEATURES_REPLY (xid=0x2): dpid:0000b8ca3a6f16ca > > n_tables:254, n_buffers:256 > > capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP > > actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src > mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst > > 1(dpdk1): addr:b8:ca:3a:6f:16:ca > > config: 0 > > state: 0 > > current: 10GB-FD > > supported: 1GB-FD FIBER AUTO_PAUSE > > speed: 10000 Mbps now, 1000 Mbps max > > 2(vxlan0): addr:46:6f:4a:fd:94:75 > > config: 0 > > state: 0 > > speed: 0 Mbps now, 0 Mbps max > > LOCAL(br0): addr:b8:ca:3a:6f:16:ca > > config: 0 > > state: 0 > > current: 10MB-FD COPPER > > speed: 10 Mbps now, 0 Mbps max > > OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0 > > > > > > As you can see in the br0 packets are reaching the vxlan interface and its > not forwarding form there it seems dropped at vxlan interface. > > > > The same configurations is working without the dpdk. > > please refer the attached diagram for my set up. > > > > Thanks > > Praveen > > > > > > On Wed, Oct 28, 2015 at 11:39 AM, Chandran, Sugesh < > sugesh.chand...@intel.com> wrote: > > Hi Praveen, > > > > The arp entries are learned properly, if the packets are reaching at br1. > > Can you please confirm what flow rules are populated on br1 using , > > ovs-ofctl dump-flows br1 > > > > If there is no rules present, please add a rule to forward packet to the > physical port as follows, > > ovs-ofctl add-flow br1 idle_timeout=0,in_port=LOCAL,action=output:1 > > > > > > *Regards* > > *_Sugesh* > > > > *From:* Praveen MANKARA RADHAKRISHNAN [mailto:praveen.mank...@6wind.com] > *Sent:* Wednesday, October 28, 2015 10:05 AM > *To:* Chandran, Sugesh > *Cc:* b...@openvswitch.org; Daniel Virassamy > *Subject:* Re: [ovs-discuss] Test with vxlan using ovs-dpdk (2.4) > > > > Hi Sugesh, > > > > Thanks for the reply. > > arp entries are show in vs-appctl tnl/arp/show command. but it seems o > packet is coming to the bridge with physical port. > > > > I have already tried to set using tnl/arp/set command but it seems set is > not present in openvswitch 2.4 > > it returns the following error. > > > > " > > ovs-appctl tnl/arp/set br0 10.200.0.6 02:11:22:44:55:11 > > "tnl/arp/set" is not a valid command > > ovs-appctl: ovs-vswitchd: server returned an error " > > > > One important thing is to note that i was using dpdk-2.0.0. does that make > any difference? > > > > Thanks > > Praveen > > > > On Tue, Oct 27, 2015 at 6:28 PM, Chandran, Sugesh < > sugesh.chand...@intel.com> wrote: > > > > > > *Regards* > > *_Sugesh* > > > > *From:* discuss [mailto:discuss-boun...@openvswitch.org] *On Behalf Of > *Praveen > MANKARA RADHAKRISHNAN > *Sent:* Monday, October 26, 2015 5:23 PM > *To:* b...@openvswitch.org > *Cc:* Daniel Virassamy > *Subject:* [ovs-discuss] Test with vxlan using ovs-dpdk (2.4) > > > > HI, > > > > I am Praveen, > > > > I am trying a specific test scenario using ovs-dpdk. > > > > Test set up > > ------------------- > > > > from the traffgen vlan packets are coming to dpdk interface its stripped > using ovs command. > > and sending to a vxlan interface which is comnnected in theo other dpdk > interface and send back to traffgen. > > > > i have done the vxlan set up similar to the sample test case. > > > > https://github.com/openvswitch/ovs/blob/master/README-native-tunneling.md > > > > the issue is that from the sumpflows i can see that packets are send to > vxlan but it is not going to the other bridge. > > > > the same scenario is working without dpdk. > > > > I am using openvswitch-2.4.0. > > > > Linux (without dpdk) > > --------------------------------- > > > > ovs-vsctl add-br br0 > > ovs-vsctl add-port br0 eth0 > > ovs-vsctl add-port br0 vxlan0 -- set interface vxlan0 type=vxlan > options:remote_ip=10.200.0.6 > > ovs-ofctl add-flow br0 > dl_vlan=1,dl_dst=02:11:22:44:55:22,actions=strip_vlan,output:2 > > > > ovs-vsctl add-br br1 > > ovs-vsctl add-port br1 eth1 > > > > ip a a 10.200.0.5/24 dev br1 > > arp -s 10.200.0.6 02:11:22:44:55:11 -i eth2 > > > > and from the traffgen i am sending traffic with dst mac 02:11:22:44:55:22 > and vlan id 1. > > it is correctly working. > > > > ovs-dpdk > > ------------------ > > > > started ovs-dpdk :: > > ovs-vswitchd --dpdk -c 0x1 -n 4 -- unix:$DB_SOCK --pidfile --detach > > > > ovs-vsctl add-br br0 -- set bridge br0 datapath_type=netdev > > ovs-vsctl add-port br0 dpdk0 -- set Interface dpdk0 type=dpdk > > > > ovs-vsctl add-br br1 -- set bridge br1 datapath_type=netdev > > ovs-vsctl add-port br1 dpdk1 -- set Interface dpdk1 type=dpdk > > > > ovs-vsctl add-port br0 vxlan0 -- set interface vxlan0 type=vxlan > options:remote_ip=10.200.0.6 > > ovs-ofctl add-flow br0 > dl_vlan=1,dl_dst=02:11:22:44:55:22,actions=strip_vlan,output:2 > > > > ip a a 10.200.0.5/24 dev br1 > > arp -s 10.200.0.6 02:11:22:44:55:11 -i eth2 > > > > > > here i have started the ovs-dpddk and do the same set up > > but packets are not seen in bridge1 (br1) and in traffgen. > > > > is there any specif configuration i need to do in ovs-dpdk > > could it be because of the arp? > > Is there a way to set static arp to the ovs-dpdk? > > >>>> Does the arp entries shown properly in “ovs-appctl tnl/arp/show”??? > > >>>>> You can set the arp entries using ovs-appctl tnl/arp/set <bridge> > <ip> <mac>. In OVS-DPDK the DPDK interfaces are not managed by kernel so > assigning static arp to physical interface may not work. > > Could you help in identifying the problem? > > > > Thanking you in advance > > Praveen > > > > > > > > > > > > > > >
_______________________________________________ discuss mailing list discuss@openvswitch.org http://openvswitch.org/mailman/listinfo/discuss
