Hi Ceara For testpmd command line, txqflags = 0xf01 should be set for receiving packets which needs more than one mbufs. I am not sure if it is helpful for you here. Please have a try!
Regards, Helin > -----Original Message----- > From: Take Ceara [mailto:dumitru.ceara at gmail.com] > Sent: Tuesday, July 26, 2016 4:38 PM > To: Xing, Beilei <beilei.xing at intel.com> > Cc: Zhang, Helin <helin.zhang at intel.com>; Wu, Jingjing <jingjing.wu at > intel.com>; > dev at dpdk.org > Subject: Re: [dpdk-dev] [dpdk-users] RSS Hash not working for XL710/X710 NICs > for some RX mbuf sizes > > Hi Beilei, > > On Mon, Jul 25, 2016 at 12:04 PM, Take Ceara <dumitru.ceara at gmail.com> > wrote: > > Hi Beilei, > > > > On Mon, Jul 25, 2016 at 5:24 AM, Xing, Beilei <beilei.xing at intel.com> > > wrote: > >> Hi, > >> > >>> -----Original Message----- > >>> From: Take Ceara [mailto:dumitru.ceara at gmail.com] > >>> Sent: Friday, July 22, 2016 8:32 PM > >>> To: Xing, Beilei <beilei.xing at intel.com> > >>> Cc: Zhang, Helin <helin.zhang at intel.com>; Wu, Jingjing > >>> <jingjing.wu at intel.com>; dev at dpdk.org > >>> Subject: Re: [dpdk-dev] [dpdk-users] RSS Hash not working for > >>> XL710/X710 NICs for some RX mbuf sizes > >>> > >>> I was using the test-pmd "txonly" implementation which sends fixed > >>> UDP packets from 192.168.0.1:1024 -> 192.168.0.2:1024. > >>> > >>> I changed the test-pmd tx_only code so that it sends traffic with > >>> incremental destination IP: 192.168.0.1:1024 -> [192.168.0.2, > >>> 192.168.0.12]:1024 > >>> I also dumped the source and destination IPs in the "rxonly" > >>> pkt_burst_receive function. > >>> Then I see that packets are indeed sent to different queues but the > >>> mbuf->hash.rss value is still 0. > >>> > >>> ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i > >>> --coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 1152 --txpkts 1024 > >>> --enable-rx-cksum --rss-udp > >>> > >>> [...] > >>> > >>> - Receive queue=0xf > >>> PKT_RX_RSS_HASH > >>> src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - > >>> length=1024 - nb_segs=1 - RSS queue=0xa - (outer) L2 type: ETHER - > >>> (outer) L3 type: IPV4_EXT_UNKNOWN SIP=C0A80001 DIP=C0A80006 - > >>> (outer) > >>> L4 type: UDP - Tunnel type: Unknown - RSS hash=0x0 - Inner L2 type: > >>> Unknown - RSS queue=0xf - RSS queue=0x7 - (outer) L2 type: ETHER - > >>> (outer) L3 type: IPV4_EXT_UNKNOWN SIP=C0A80001 DIP=C0A80007 - > >>> (outer) > >>> L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown - Inner > >>> L3 type: Unknown - Inner L4 type: Unknown > >>> - Receive queue=0x7 > >>> PKT_RX_RSS_HASH > >>> src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - (outer) L2 type: > >>> ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN SIP=C0A80001 DIP=C0A80009 > >>> - > >>> type=0x0800 - length=1024 - nb_segs=1 - Inner L3 type: Unknown - > >>> Inner > >>> L4 type: Unknown - RSS hash=0x0 - (outer) L4 type: UDP - Tunnel type: > >>> Unknown - Inner L2 type: Unknown - Inner L3 type: Unknown - RSS > >>> queue=0x7 - Inner L4 type: Unknown > >>> > >>> [...] > >>> > >>> testpmd> stop > >>> ------- Forward Stats for RX Port= 0/Queue= 0 -> TX Port= 1/Queue= 0 > ------- > >>> RX-packets: 0 TX-packets: 32 TX-dropped: 0 > >>> ------- Forward Stats for RX Port= 0/Queue= 1 -> TX Port= 1/Queue= 1 > ------- > >>> RX-packets: 59 TX-packets: 32 TX-dropped: 0 > >>> ------- Forward Stats for RX Port= 0/Queue= 2 -> TX Port= 1/Queue= 2 > ------- > >>> RX-packets: 48 TX-packets: 32 TX-dropped: 0 > >>> ------- Forward Stats for RX Port= 0/Queue= 3 -> TX Port= 1/Queue= 3 > ------- > >>> RX-packets: 0 TX-packets: 32 TX-dropped: 0 > >>> ------- Forward Stats for RX Port= 0/Queue= 4 -> TX Port= 1/Queue= 4 > ------- > >>> RX-packets: 59 TX-packets: 32 TX-dropped: 0 > >>> ------- Forward Stats for RX Port= 0/Queue= 5 -> TX Port= 1/Queue= 5 > ------- > >>> RX-packets: 0 TX-packets: 32 TX-dropped: 0 > >>> ------- Forward Stats for RX Port= 0/Queue= 6 -> TX Port= 1/Queue= 6 > ------- > >>> RX-packets: 0 TX-packets: 32 TX-dropped: 0 > >>> ------- Forward Stats for RX Port= 0/Queue= 7 -> TX Port= 1/Queue= 7 > ------- > >>> RX-packets: 48 TX-packets: 32 TX-dropped: 0 > >>> ------- Forward Stats for RX Port= 0/Queue= 8 -> TX Port= 1/Queue= 8 > ------- > >>> RX-packets: 0 TX-packets: 32 TX-dropped: 0 > >>> ------- Forward Stats for RX Port= 0/Queue= 9 -> TX Port= 1/Queue= 9 > ------- > >>> RX-packets: 59 TX-packets: 32 TX-dropped: 0 > >>> ------- Forward Stats for RX Port= 0/Queue=10 -> TX Port= 1/Queue=10 > ------- > >>> RX-packets: 48 TX-packets: 32 TX-dropped: 0 > >>> ------- Forward Stats for RX Port= 0/Queue=11 -> TX Port= 1/Queue=11 > ------- > >>> RX-packets: 0 TX-packets: 32 TX-dropped: 0 > >>> ------- Forward Stats for RX Port= 0/Queue=12 -> TX Port= 1/Queue=12 > ------- > >>> RX-packets: 59 TX-packets: 32 TX-dropped: 0 > >>> ------- Forward Stats for RX Port= 0/Queue=13 -> TX Port= 1/Queue=13 > ------- > >>> RX-packets: 0 TX-packets: 32 TX-dropped: 0 > >>> ------- Forward Stats for RX Port= 0/Queue=14 -> TX Port= 1/Queue=14 > ------- > >>> RX-packets: 0 TX-packets: 32 TX-dropped: 0 > >>> ------- Forward Stats for RX Port= 0/Queue=15 -> TX Port= 1/Queue=15 > ------- > >>> RX-packets: 48 TX-packets: 32 TX-dropped: 0 > >>> ---------------------- Forward statistics for port 0 > >>> ---------------------- > >>> RX-packets: 428 RX-dropped: 84 RX-total: 512 > >>> TX-packets: 0 TX-dropped: 0 TX-total: 0 > >>> > >>> -------------------------------------------------------------------- > >>> -------- > >>> > >>> ---------------------- Forward statistics for port 1 > >>> ---------------------- > >>> RX-packets: 0 RX-dropped: 0 RX-total: 0 > >>> TX-packets: 512 TX-dropped: 0 TX-total: 512 > >>> > >>> -------------------------------------------------------------------- > >>> -------- > >>> > >>> +++++++++++++++ Accumulated forward statistics for all > >>> ports+++++++++++++++ > >>> RX-packets: 428 RX-dropped: 84 RX-total: 512 > >>> TX-packets: 512 TX-dropped: 0 TX-total: 512 > >>> > >>> > +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ > >>> +++++++++++++++ > >>> > >>> I checked all the RSS hash values for all the 10 different incoming > >>> streams and they're all 0. Also, the fact that the traffic is > >>> actually distributed seems to suggest that RSS itself is working but > >>> the mbuf hash field is (I guess) either not written or corrupted. > >>> > >> > >> I tried to reproduce the problem with the same steps you used on 16.04 and > 16.07, but I really didn't replicate it. > >> I think you can try follow ways: > >> 1. apply the patch I told you last time and check if the problem still > >> exists. > > > > I applied the changes in the patch manually to 16.04. The RSS=0 > > problem still exists while the FDIR issue is fixed. > > > >> 2. update the codebase and check if the problem still exists. > > > > I updated the codebase to the latest version on > > http://dpdk.org/git/dpdk. I still see the RSS=0 issue. > > > >> 3. disable vector when you run testpmd, and check if the problem still > >> exists. > > > > I recompiled the latest dpdk code with > > CONFIG_RTE_LIBRTE_I40E_INC_VECTOR=n and the RSS=0 issue is still > > there. > > > > My current command line is: > > ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i > > --coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 1152 --txpkts 1024 > > --rss-udp > > > > Not sure if relevant but I'm running kernel 4.2.0-27: > > $ uname -a > > Linux jspg2 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 > > 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux > > > > Is there anything else that might help you identify the cause of the > > problem? > > > > Thanks, > > Dumitru > > After some debugging in the i40e DPDK driver I figured out the problem When > receiving packets with i40e_recv_scattered_pkts, which gets called in my case > because the incoming packet is bigger than 1 full mbuf (the 4 bytes CRC goes > in > the second mbuf of the chain), the pkt_flags, hash, etc are set only when > processing the last mbuf in the packet chain. However, when the hash.rss > field is > set, instead of setting it in the first mbuf of the packet it gets set in the > current > mbuf (rxm). This can also cause a lot of unpredictable behavior if the last > mbuf > only contained the CRC that was stripped as rxm would have already been freed > by then. The line I'm referring to is: > > if (pkt_flags & PKT_RX_RSS_HASH) > rxm->hash.rss = > rte_le_to_cpu_32(rxd.wb.qword0.hi_dword.rss); > > I changed it to setting the rss field in first_seg instead of rxm and it > works fine > now. > > As far as I see this is the only place where we can receive scattered packets > and > all the other places where the RSS hash is set seem to be fine. > Should I submit a proper patch for this or will you do it as you're more > familiar to > the code? > > Thanks, > Dumitru > > > > >> > >>> > > >>> >> > >>> >> If I use a different mbuf-size, for example 2048, everything is fine: > >>> >> > >>> >> ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i > >>> >> --coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 2048 --txpkts > >>> >> 1024 > >>> >> -- enable-rx-cksum --rss-udp [...] > >>> >> testpmd> set verbose 1 > >>> >> Change verbose level from 0 to 1 > >>> >> testpmd> set fwd rxonly > >>> >> Set rxonly packet forwarding mode > >>> >> testpmd> start tx_first > >>> >> rxonly packet forwarding - CRC stripping disabled - packets/burst=32 > >>> >> nb forwarding cores=16 - nb forwarding ports=2 > >>> >> RX queues=16 - RX desc=128 - RX free threshold=32 > >>> >> RX threshold registers: pthresh=8 hthresh=8 wthresh=0 > >>> >> TX queues=16 - TX desc=512 - TX free threshold=32 > >>> >> TX threshold registers: pthresh=32 hthresh=0 wthresh=0 > >>> >> TX RS bit threshold=32 - TXQ flags=0xf01 port 0/queue 1: > >>> >> received > >>> >> 32 packets > >>> >> src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - > >>> >> length=1024 - nb_segs=1 - RSS hash=0x5263c3f2 - RSS queue=0x1 - > >>> >> (outer) L2 type: ETHER - (outer) L3 type: IPV4_EXT_UNKNOWN - > >>> >> (outer) > >>> >> L4 type: UDP - Tunnel type: Unknown - Inner L2 type: Unknown - > >>> >> Inner > >>> >> L3 type: Unknown - Inner L4 type: Unknown > >>> >> - Receive queue=0x1 > >>> >> PKT_RX_RSS_HASH > >>> >> > >>> > > >>> > Did you send the same packet as before to port 0? > >>> > > >>> >> Another weird thing I noticed is when I run test-pmd without > >>> >> --enable-rx- cksum (which is the default mode) then the RSS flag > >>> >> doesn get > >>> set at all. > >>> >> Instead the PKT_RX_FDIR flag gets set. This happens even though > >>> >> fdir_mode is set to RTE_FDIR_MODE_NONE in the device > >>> >> configuration: > >>> >> > >>> >> ./testpmd -c ffff1 -n 4 -w 0000:01:00.3 -w 0000:81:00.3 -- -i > >>> >> --coremask=0xffff0 --rxq=16 --txq=16 --mbuf-size 1152 --txpkts > >>> >> 1024 > >>> >> --rss- udp [...] > >>> >> testpmd> set verbose 1 > >>> >> Change verbose level from 0 to 1 > >>> >> testpmd> set fwd rxonly > >>> >> Set rxonly packet forwarding mode > >>> >> testpmd> start tx_first > >>> >> rxonly packet forwarding - CRC stripping disabled - packets/burst=32 > >>> >> nb forwarding cores=16 - nb forwarding ports=2 > >>> >> RX queues=16 - RX desc=128 - RX free threshold=32 > >>> >> RX threshold registers: pthresh=8 hthresh=8 wthresh=0 > >>> >> TX queues=16 - TX desc=512 - TX free threshold=32 > >>> >> TX threshold registers: pthresh=32 hthresh=0 wthresh=0 > >>> >> TX RS bit threshold=32 - TXQ flags=0xf01 port 0/queue 1: > >>> >> received > >>> >> 16 packets > >>> >> src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - > >>> >> length=1024 - nb_segs=2 - FDIR matched hash=0xc3f2 ID=0x5263 > >>> >> Unknown packet type > >>> >> - Receive queue=0x1 > >>> >> PKT_RX_FDIR > >>> >> src=68:05:CA:38:6D:63 - dst=02:00:00:00:00:01 - type=0x0800 - > >>> >> length=1024 - nb_segs=2 - FDIR matched hash=0xc3f2 ID=0x5263 > >>> >> Unknown packet type > >>> >> - Receive queue=0x1 > >>> >> PKT_RX_FDIR > >>> >> > >>> > > >>> > For this issue, I think following patch can solve your problem, > >>> > please apply this > >>> patch. > >>> > http://dpdk.org/dev/patchwork/patch/13593/ > >>> > > >>> > >>> I tried to apply it directly on 16.04 but it can't be applied. I see > >>> it's been applied to dpdk-next-net/rel_16_07. Do you happen to have > >>> one that would work on the latest stable 16.04 release? > >> > >> Sorry, I haven't. If there's conflict with R16.04, I think you can resolve > >> it by > going through the patch. > >> > >> Beilei > >> > >>> > >>> Thanks, > >>> Dumitru > >>> > >>> > Beilei > >>> > > >>> >> Please let me know if there's more debug info that might be of > >>> >> interest in troubleshooting the RSS=0 issue. > >>> >> > >>> >> Thanks, > >>> >> Dumitru > >>> >> > >>> >> > /Beilei > >>> >> > > >>> >> >> Thanks, > >>> >> >> Dumitru > >>> >> >> > > > > > > > > -- > > Dumitru Ceara > > > > -- > Dumitru Ceara
