Just checking back on this thread to see where things are. Are we saying this is a bug in dpdk or vpp ?. That part wasn't quite clear to me.
-Chandra On Thu, May 26, 2016 at 3:56 PM, John Daley (johndale) <johndale at cisco.com> wrote: > John, > As just discussed, what you suggest was considered but there are other > apps depending a different behavior of the flag, so we thought the best > thing to do is deprecate it. That is part of what Olivier's patch discussed > in http://www.dpdk.org/ml/archives/dev/2016-May/038786.html does. Adding > a new ptype for VLAN tagged packets after the patch is accepted would allow > VPP to check if the ptype is supported by the PMD and if so, use it to > determine if the delivered packet actually has a VLAN tag in it. No need to > know if stripping is enabled/disabled. If the ptype is not supported, the > app would have check the packet in SW. > > > -----Original Message----- > > From: dev [mailto:dev-bounces at dpdk.org] On Behalf Of John Lo (loj) > > Sent: Thursday, May 26, 2016 11:11 AM > > To: Ananyev, Konstantin <konstantin.ananyev at intel.com>; Wiles, Keith > > <keith.wiles at intel.com>; Chandrasekar Kannan <ckannan at console.to> > > Cc: vpp-dev <vpp-dev at lists.fd.io>; Zhang, Helin <helin.zhang at > > intel.com>; > > dev at dpdk.org > > Subject: Re: [dpdk-dev] [vpp-dev] VLAN packets dropped... ? > > > > Hi Konstantin, > > > > Thanks for the link to DPDK discussion wrt this VLAN offload flag > > PKT_RX_VLAN_PKT. It seems the proposal was to deprecate > > PKT_RX_VLAN_PKT and introduce two new flags PKT_RX_VLAN_STRIPPED > > and PKT_RX_QINQ_STRIPPED. > > > > It would be a really good idea to keep PKT_RX_VLAN_PKT to indicate at > least > > one VLAN tag is present on the packet. For the case of i40e where its HW > > cannot detect VLAN tag if strip is not enabled, it should be reasonable > for the > > i40e DPDK driver software to make a check and set this flag. I would > think the > > overhead to check the 1st ethertype field in the MAC header against dot1q > > or dot1ad values in software be pretty minimal. > > > > Regards, > > John > > > > > > -----Original Message----- > > From: Ananyev, Konstantin [mailto:konstantin.ananyev at intel.com] > > Sent: Wednesday, May 25, 2016 3:23 PM > > To: John Lo (loj); Wiles, Keith; Chandrasekar Kannan > > Cc: vpp-dev; Zhang, Helin; dev at dpdk.org > > Subject: RE: [vpp-dev] VLAN packets dropped... ? > > > > > > > I suppose this has to do with what is expected usage of the > > > PKT_RX_VLAN_PKT offload flag. Is it set only for VLAN packets with the > > > VLAN stripped or should always be set if VLAN is/was present in the > > > received packet. It seems that different DPDK drivers are behaving > > differently which will make it really hard for VPP to take advantage of > NIC > > and driver offload capability to provide better performance. > > > > Yes, that's true ixgbe/igb from one side and i40e do raise > PKT_RX_VLAN_PKT > > in a different manner. > > There is an attempt to make it unified across all supported devices: > > http://dpdk.org/dev/patchwork/patch/12938/ > > > > Though, I am not sure it will help you with your issue. > > As I understand, for you the desired behaviour is: > > If it is a vlan packet, keep the packet intact (don't strip the vlan) > and raise > > PKT_RX_VLAN_PK inside mbuf->ol_flags. > > That's what ixgbe is doing with rte_eth_conf.rxmode.hw_vlan_strip==0. > > Correct? > > As far as I know, i40e HW doesn't provide such ability. > > i40e Receive HW Descriptor can only flag was VLAN tag stripped from the > > packet or not, but if stripping is disabled it wouldn't indicate in any > way is > > VLAN tag is present inside the packet or not. > > I am CC-ing it to dpdk.org in case I am missing something here. > > Probably someone knows a way to make it work in that way. > > Konstantin > > > > > > > > If VPP cannot rely on offload flags for VLAN so packets always have to > go > > through ethernet-input node, there is a performance cost. > > > For the 10GE case, before the inverse patch of the ixgbe driver, 10GE > > > Rx-vector path removed support of offload flag with DPDK 16.04 and so > > > ethernet-input node is always used. The 10GE IPv4 throughput rate > > > dropped from 6.17MPPSx2 to 4.92MPPSx2 for bidirectional traffic (2 > > > streams each with a single IP address as destination) on a single core > > > on my server. Konstantin suggested at that time to use scalar mode > which > > does support offload flags properly. The scalar mode did by-pass > ethernet- > > input and provided throughput of 5.72MPPS. We ended up inverse patched > > the ixgbe driver to restore vector mode offload flag support as the > original > > restriction (the reason offload flag support was removed) would not > affect > > VPP. > > > > > > I think for 40GE driver to provide offload flag support in vector mode > > > but not give indication of presence of VLAN tag is just wrong. This > make the > > offload flag support useless for VPP. > > > > > > Regards, > > > John > > > > > > -----Original Message----- > > > From: Ananyev, Konstantin [mailto:konstantin.ananyev at intel.com] > > > Sent: Wednesday, May 25, 2016 11:30 AM > > > To: John Lo (loj); Wiles, Keith; Chandrasekar Kannan > > > Cc: vpp-dev; Zhang, Helin > > > Subject: RE: [vpp-dev] VLAN packets dropped... ? > > > > > > > > > > > > > > I see what you are getting at, Konstantin. The VPP init code does > > > > not enable VLAN strip for Intel NICs as VLAN tag must be in the > > > > packet for sub-interface lookup by ethernet-input node. I agree if > > > > we enable VLAN tag strip for the i40e driver, we can get around this > > > > problem > > > but then all packets will be considered as received on the main > interface. > > > > > > I see... > > > As far as I know, when VLAN stripping is disabled, i40e RXD doesn't > contain > > information does that packet contain a VLAN or not. > > > So, if enabling vlan stripping is not an option for you guys, then I > > > don't see any other way on i40e to recognise is that VLAN packet or > not, > > except then parse the packet in SW. > > > Helin, please correct me here, if I am missing something here. > > > Thanks > > > Konstantin > > > > > > > > > > > Regards, > > > > John > > > > > > > > -----Original Message----- > > > > From: Ananyev, Konstantin [mailto:konstantin.ananyev at intel.com] > > > > Sent: Wednesday, May 25, 2016 10:35 AM > > > > To: John Lo (loj); Wiles, Keith; Chandrasekar Kannan > > > > Cc: vpp-dev > > > > Subject: RE: [vpp-dev] VLAN packets dropped... ? > > > > > > > > > > > > > > > > > > Since this is the XL710 40GE NIC, I suppose the DPDK driver > involved > > would be the i40e driver and not ixgbe for 10GE NICs. > > > > > > > > Yes, I understand that you are facing problem on i40e, not ixgbe. > > > > And the problem is that for i40e PKT_RX_VLAN_PKT flag is not set in > > mbuf->ol_flags, correct? > > > > That's why I asked: are you running it with > > rte_eth_conf.rxmode.hw_vlan_strip==0 or not? > > > > If yes, you can try with rte_eth_conf.rxmode.hw_vlan_strip=1 and see > > would it help you. > > > > > > > > > > > > > > As explained before, ixgbe driver had the inverse patch added. It > > > > > does recognize VLAN tag in the packet and set PKT_RX_VLAN_PKT > > offload flag properly: > > > > > > > > That patch has nothing to do with PKT_RX_VLAN_PKT and i40e. > > > > So I don't think it is related to that problem at all. > > > > Konstantin > > > > > > > > > > > > > > 00:01:02:132370: dpdk-input > > > > > TenGigabitEthernet5/0/0 rx queue 0 > > > > > buffer 0x44cff: current data 0, length 96, free-list 0, > totlen-nifb 0, trace > > 0x0 > > > > > PKT MBUF: port 3, nb_segs 1, pkt_len 96 > > > > > buf_len 2176, data_len 96, ol_flags 0x1, > > > > > packet_type 0x210 > > > > > Packet Offload Flags > > > > > PKT_RX_VLAN_PKT (0x0001) RX packet is a 802.1q VLAN packet > > > > > Packet Types > > > > > RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension > > headers > > > > > RTE_PTYPE_L4_UDP (0x0200) UDP packet > > > > > IP4: 00:00:00:00:00:aa -> 90:e2:ba:6e:e7:dc vlan 1101 > > > > > UDP: 28.0.0.1 -> 29.0.0.254 > > > > > tos 0x00, ttl 64, length 78, checksum 0x40a1 > > > > > fragment id 0x0000 > > > > > UDP: 63 -> 63 > > > > > length 58, checksum 0x5b76 > > > > > 00:01:02:132391: ethernet-input > > > > > IP4: 00:00:00:00:00:aa -> 90:e2:ba:6e:e7:dc vlan 1101 > > > > > 00:01:02:132408: error-drop > > > > > ethernet-input: unknown vlan > > > > > > > > > > You can see from above the packet was passed to ethernet-input > > > > > node and dropped because sub-interface for VLAN 1101 is not set up. > > > > > > > > > > Regards, > > > > > John > > > > > > > > > > -----Original Message----- > > > > > From: Ananyev, Konstantin [mailto:konstantin.ananyev at intel.com] > > > > > Sent: Wednesday, May 25, 2016 4:46 AM > > > > > To: John Lo (loj); Wiles, Keith; Chandrasekar Kannan > > > > > Cc: vpp-dev > > > > > Subject: RE: [vpp-dev] VLAN packets dropped... ? > > > > > > > > > > > > > > > Hi, > > > > > I don?t think ptype recognition on ixgbe vRX is somehow related to > that. > > > > > Are you guys running FVL with > > rte_eth_conf.rxmode.hw_vlan_strip==0? > > > > > If so, can you set it to 1 and see would it help? > > > > > Konstantin > > > > > > > > > > From: vpp-dev-bounces at lists.fd.io > > > > > [mailto:vpp-dev-bounces at lists.fd.io] > > > > > On Behalf Of John Lo (loj) > > > > > Sent: Wednesday, May 25, 2016 1:56 AM > > > > > To: Wiles, Keith; Chandrasekar Kannan > > > > > Cc: vpp-dev > > > > > Subject: Re: [vpp-dev] VLAN packets dropped... ? > > > > > > > > > > I did put in a reverse patch, provided by Damjan, for this in > fd.io: > > > > > https://gerrit.fd.io/r/#/c/941/1/dpdk/dpdk-16.04_patches/0013-Reve > > > > > rt -i xgbe-fix-packet-type-from-vector-Rx.patch > > > > > > > > > > I know this fixed the 10GE NIC behavior. Perhaps the 40GE NIC is > using a > > different DPDK driver? > > > > > > > > > > Regards, > > > > > John > > > > > > > > > > From: Wiles, Keith [mailto:keith.wiles at intel.com] > > > > > Sent: Tuesday, May 24, 2016 8:30 PM > > > > > To: John Lo (loj); Chandrasekar Kannan > > > > > Cc: vpp-dev > > > > > Subject: Re: [vpp-dev] VLAN packets dropped... ? > > > > > > > > > > > > > > > The commit ID is > > > > > > > > > > commit d9a2009a81089093645fea2e04b51dd37edf3e6f > > > > > Author: Konstantin Ananyev <konstantin.ananyev at intel.com> > > > > > Date: Tue Mar 22 14:30:17 2016 +0000 > > > > > > > > > > ixgbe: fix packet type from vector Rx > > > > > > > > > > Current vector RX can't always set the packet_type properly. > > > > > To be more specific: > > > > > a) it never sets RTE_PTYPE_L2_ETHER > > > > > b) it doesn't handle tunnel ipv4/ipv6 case correctly. > > > > > c) it doesn't check is IXGBE_RXDADV_PKTTYPE_ETQF set or not. > > > > > While a) is pretty easy to fix, b) and c) are not that > > > > > straightforward > > > > > in terms of SIMD ops (specially b). > > > > > So far I wasn't able to make vRX support packet_type properly > > > > > without > > > > > noticeable performance loss. > > > > > So for now, just remove that functionality from vector RX and > > > > > update dev_supported_ptypes_get(). > > > > > > > > > > Fixes: 396254175854 ("mbuf: redefine packet type") > > > > > > > > > > Signed-off-by: Konstantin Ananyev > > > > > <konstantin.ananyev at intel.com> > > > > > Acked-by: Cunming Liang <cunming.liang at intel.com> > > > > > > > > > > It appears the change was only down to the vector version in the RX > > routine, could move to the none-vector version I guess. > > > > > > > > > > I would think adding a patch or adding the code to VPP DPDK init > to use > > the RX callback to setup things correctly. > > > > > > > > > > Regards, > > > > > Keith > > > > > > > > > > > > > > > From: <vpp-dev-bounces at lists.fd.io> on behalf of Keith Wiles > > > > > <keith.wiles at intel.com> > > > > > Date: Tuesday, May 24, 2016 at 7:19 PM > > > > > To: "John Lo (loj)" <loj at cisco.com>, Chandrasekar Kannan > > > > > <ckannan at console.to> > > > > > Cc: vpp-dev <vpp-dev at lists.fd.io> > > > > > Subject: Re: [vpp-dev] VLAN packets dropped... ? > > > > > > > > > > > > > > > Here is the starting thread: > > > > > http://dpdk.org/ml/archives/dev/2016-April/037837.html > > > > > > > > > > > > > > > Regards, > > > > > Keith > > > > > > > > > > > > > > > From: <vpp-dev-bounces at lists.fd.io> on behalf of Keith Wiles > > > > > <keith.wiles at intel.com> > > > > > Date: Tuesday, May 24, 2016 at 7:16 PM > > > > > To: "John Lo (loj)" <loj at cisco.com>, Chandrasekar Kannan > > > > > <ckannan at console.to> > > > > > Cc: vpp-dev <vpp-dev at lists.fd.io> > > > > > Subject: Re: [vpp-dev] VLAN packets dropped... ? > > > > > > > > > > The flags in the rte_mbuf were changed a before 16.04, which > > > > > removed the ptype testing and possible setting this flag. I > thought a > > patch to revert that change was done, did that not happen. > > > > > > > > > > The other solution is to add an option to DPDK to restore the > > > > > previous behavior for this type of problem. The only other way is > > > > > to have a installed in the PMD to setup the flags correctly for > VPP. > > > > > This would require adding a routine to VPP and setting the call > > > > > back in the > > > > driver. > > > > > > > > > > I really thought a patch was created, does anyone know? > > > > > > > > > > Regards, > > > > > Keith > > > > > > > > > > > > > > > From: "John Lo (loj)" <loj at cisco.com> > > > > > Date: Tuesday, May 24, 2016 at 6:53 PM > > > > > To: Chandrasekar Kannan <ckannan at console.to> > > > > > Cc: Keith Wiles <keith.wiles at intel.com>, vpp-dev > > > > > <vpp-dev at lists.fd.io> > > > > > Subject: RE: [vpp-dev] VLAN packets dropped... ? > > > > > > > > > > I believe the problem is due to 40GE DPDK driver not parsing VLAN > > > > > packet properly and setting the packet offload flag > > > > > PKT_RX_VLAN_PKT, causing dpdk-input node to send packet directly > > > > > to ip4-input instead of ethernet-input. If this flag is set, then > > > > > the packet will be passed to ethernet-input which will then parse > > > > > the VLAN tag in the > > > > packet to perform sub-interface lookup and find the start of IP > > > > header in the packet properly. Following is an example trace of a > correctly > > marked packet: > > > > > > > > > > 23:11:09:855916: dpdk-input > > > > > TenGigE0/0/0/0 rx queue 0 > > > > > buffer 0x23f6c0: current data 0, length 118, free-list 0, > > > > > totlen-nifb 0, trace 0x0 > > > > > PKT MBUF: port 0, nb_segs 1, pkt_len 118 > > > > > buf_len 2176, data_len 118, ol_flags 0x1, > > > > > packet_type 0x10 > > > > > Packet Offload Flags > > > > > PKT_RX_VLAN_PKT (0x0001) RX packet is a 802.1q VLAN packet > > > > > Packet Types > > > > > RTE_PTYPE_L3_IPV4 (0x0010) IPv4 packet without extension > > > > > headers > > > > > IP4: 74:a0:2f:e2:d5:14 -> 90:e2:ba:9f:8b:cc vlan 101 > > > > > ICMP: 130.1.1.1 -> 130.1.1.2 > > > > > tos 0x00, ttl 255, length 100, checksum 0xac4b > > > > > fragment id 0x0948 > > > > > ICMP echo_request checksum 0x900d > > > > > > > > > > Hoping someone familiar with the 40GE DPDK driver can comment > > further. > > > > > > > > > > Regards, > > > > > John > > > > > > > > > > From: Chandrasekar Kannan [mailto:ckannan at console.to] > > > > > Sent: Tuesday, May 24, 2016 6:57 PM > > > > > To: John Lo (loj) > > > > > Cc: Wiles, Keith; vpp-dev > > > > > Subject: Re: [vpp-dev] VLAN packets dropped... ? > > > > > > > > > > Yes. Subinterface has been configured like this ... > > > > > > > > > > vppctl set interface ip address del FortyGigabitEthernet83/0/0 all > > > > > vppctl create sub FortyGigabitEthernet83/0/0 900 vppctl set > > > > > interface ip address FortyGigabitEthernet83/0/0.900 6.0.0.1/24 > > > > > vppctl set interface ip address FortyGigabitEthernet83/0/0.900 > > > > > 7.0.0.1/24 vppctl set interface state FortyGigabitEthernet83/0/0 > > > > > up vppctl set interface state FortyGigabitEthernet83/0/0.900 up > > > > > vppctl set ip arp static > > > > > FortyGigabitEthernet83/0/0.900 7.0.0.2 3c:fd:fe:9c:c3:88 > > > > > > > > > > -------------------------------------------------- > > > > > Packet 4 > > > > > > > > > > 00:02:23:366879: dpdk-input > > > > > FortyGigabitEthernet83/0/0 rx queue 0 > > > > > buffer 0x10e6b: current data 14, length 50, free-list 0, > > > > > totlen-nifb 0, trace 0x3 > > > > > PKT MBUF: port 0, nb_segs 1, pkt_len 64 > > > > > buf_len 2176, data_len 64, ol_flags 0x0, > > > > > packet_type 0x291 > > > > > Packet Types > > > > > RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet > > > > > RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or > > > > > without extension headers > > > > > RTE_PTYPE_L4_UDP (0x0200) UDP packet > > > > > IP4: 3c:fd:fe:9c:c3:88 -> 3c:fd:fe:9d:a0:20 vlan 900 > > > > > UDP: 6.0.0.2 -> 7.0.0.2 > > > > > tos 0x00, ttl 64, length 46, checksum 0x2dbc > > > > > fragment id 0x0000, flags DONT_FRAGMENT > > > > > UDP: 1024 -> 2001 > > > > > length 26, checksum 0x0000 > > > > > 00:02:23:366894: ip4-input-no-checksum > > > > > IP6_HOP_BY_HOP_OPTIONS: 64.17.45.188 -> 6.0.0.2 > > > > > version 0, header length 12 > > > > > tos 0x84, ttl 0, length 2048, checksum 0x4000 (should be > > > > > 0xaf4d) > > > > > fragment id 0x4500 offset 368, flags > > > > > 00:02:23:366910: error-drop > > > > > ip4-input: ip4 length > l2 length > > > > > > > > > > -------------------------------------------------------------- > > > > > > > > > > > > > > > > > > > > On Tue, May 24, 2016 at 3:12 PM, John Lo (loj) <loj at cisco.com> > wrote: > > > > > Was a sub-interface created with the expected VLAN tag 900? An > > > > > example of creating a sub-interface and bring it up would be > something > > like: > > > > > > > > > > create sub GigabitEthernet1/0/0 900 set int state > > > > > GigabitEthernet1/0/0 up set int state > > > > > GigabitEthernet1/0/0.900 up > > > > > > > > > > Regards, > > > > > John > > > > > > > > > > > > > > > From: vpp-dev-bounces at lists.fd.io > > > > > [mailto:vpp-dev-bounces at lists.fd.io] > > > > > On Behalf Of Wiles, Keith > > > > > Sent: Tuesday, May 24, 2016 5:59 PM > > > > > To: Chandrasekar Kannan; vpp-dev > > > > > Subject: Re: [vpp-dev] VLAN packets dropped... ? > > > > > > > > > > I am starting to think the unknown VLAN is not accounted for in the > > processing of the input frame in the VPP DPDK interface code. > > > > > vnet/vnet/devices/dpdk/node.c > > > > > > > > > > The version and header length are wrong in the output below, which > > > > > leads me to believe the starting IPv4 header location is wrong. I > > > > > have not parsed all of the code in node.c:dpdk_device_input() yet, > > > > > but I do not think the l3_offset is being adjusted to the correct > > > > offset???? > > > > > > > > > > Regards, > > > > > Keith > > > > > > > > > > > > > > > From: <vpp-dev-bounces at lists.fd.io> on behalf of Chandrasekar > > > > > Kannan <ckannan at console.to> > > > > > Date: Tuesday, May 24, 2016 at 3:40 PM > > > > > To: vpp-dev <vpp-dev at lists.fd.io> > > > > > Subject: [vpp-dev] VLAN packets dropped... ? > > > > > > > > > > Hi, > > > > > > > > > > I was trying to setup VLAN subinterfaces with VPP but it appears > > > > > the packets are dropped at ingress. I'm tried with both > > > > > pktgen-dpdk and trex for traffic generation just to rule out any > > > > > corruption with > > > > pkt generation. But both are producing the same errors with vpp. > > > > > > > > > > yaml file from trex... > > > > > > > > > > ------------------------------ > > > > > [root at node04 scripts]# cat cap2/ipv4_vlan.yaml > > > > > - duration : 10 > > > > > generator : > > > > > distribution : "seq" > > > > > clients_start : "16.0.0.1" > > > > > clients_end : "16.0.1.255" > > > > > servers_start : "48.0.0.1" > > > > > servers_end : "48.0.0.255" > > > > > clients_per_gb : 201 > > > > > min_clients : 101 > > > > > dual_port_mask : "1.0.0.0" > > > > > tcp_aging : 1 > > > > > udp_aging : 1 > > > > > mac : [0x0,0x0,0x0,0x1,0x0,0x00] > > > > > vlan : { enable : 1 , vlan0 : 900 , vlan1 : 1100 } > > > > > cap_info : > > > > > - name: cap2/udp_64B.pcap > > > > > cps : 1000.0 > > > > > ipg : 10000 > > > > > rtt : 10000 > > > > > w : 4 > > > > > limit : 20 > > > > > [root at node04 scripts]# > > > > > --------------------------------------------------------------- > > > > > > > > > > vpp trace... > > > > > > > > > > 00:02:21:576810: dpdk-input > > > > > FortyGigabitEthernet83/0/0 rx queue 0 > > > > > buffer 0x10e6b: current data 14, length 50, free-list 0, > > > > > totlen-nifb 0, trace 0x1 > > > > > PKT MBUF: port 0, nb_segs 1, pkt_len 64 > > > > > buf_len 2176, data_len 64, ol_flags 0x0, > > > > > packet_type 0x291 > > > > > Packet Types > > > > > RTE_PTYPE_L2_ETHER (0x0001) Ethernet packet > > > > > RTE_PTYPE_L3_IPV4_EXT_UNKNOWN (0x0090) IPv4 packet with or > > > > > without extension headers > > > > > RTE_PTYPE_L4_UDP (0x0200) UDP packet > > > > > IP4: 3c:fd:fe:9c:c3:88 -> 3c:fd:fe:9d:a0:20 vlan 900 > > > > > UDP: 16.0.1.0 -> 48.0.0.128 > > > > > tos 0x00, ttl 64, length 46, checksum 0xf93f > > > > > fragment id 0x0000, flags DONT_FRAGMENT > > > > > UDP: 1024 -> 2001 > > > > > length 26, checksum 0x0000 > > > > > 00:02:21:576817: ip4-input-no-checksum > > > > > IP6_HOP_BY_HOP_OPTIONS: 64.17.249.63 -> 16.0.1.0 > > > > > version 0, header length 12 > > > > > tos 0x84, ttl 0, length 2048, checksum 0x4000 (should be > > > > > 0xaf4d) > > > > > fragment id 0x4500 offset 368, flags > > > > > 00:02:21:576828: error-drop > > > > > ip4-input: ip4 length > l2 length > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
