Hi, I tested Rx latency with testpmd and ixia, found that the interval configuration works on X710.
With the default configuration, the latency is about 32us. When RTE_LIBRTE_I40E_ITR_INTERVAL=0, the max latency is < 8us. When RTE_LIBRTE_I40E_ITR_INTERVAL=8160, the max latency is about 8ms. My test steps: 1. connect a X710 port with ixia, and bind the port to igb_uio 2. run testpmd and start io forwding 3. send a burst (5 packets) with ixia to X710 port 4. check the latency on ixia. Best Regards, Beilei > -----Original Message----- > From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Wu, Jingjing > Sent: Friday, December 1, 2017 1:47 AM > To: Hanoch Haim (hhaim) <hh...@cisco.com> > Cc: dev@dpdk.org > Subject: Re: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling > setting in PF > > Hi, Hanoch > > Thanks a lot for the trying. Are you using igb_uio to bind the device? > > I guess it would be because that fix is not complete, the overlap with Rx > interrupt mode is not considered. We will look into it. > And it would be great if you can have a try on vfio_pci cases. > > Thanks > Jingjing > > From: Hanoch Haim (hhaim) [mailto:hh...@cisco.com] > Sent: Friday, December 1, 2017 12:48 AM > To: Wu, Jingjing <jingjing...@intel.com> > Cc: dev@dpdk.org; Hanoch Haim (hhaim) <hh...@cisco.com> > Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling > setting in PF > > Hi Jingjing, > I did that and see the results, It does not work as expected > > TRex command: > $sudo ./t-rex-64 -f astf/http_simple.py -m 10000 -l 1000 -d 1000 --astf -c 1 > > > > 1) with the issue (without the patch) > *itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | *#define > RTE_LIBRTE_I40E_ITR_INTERVAL -1 > > > latency : 40usec > > -Latency stats enabled > Cpu Utilization : 0.2 % > if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter > max > window > | , , , , average , max , (usec) > > --------------------------------------------------------------------------------------------- > ------------------- > 0 | 5603, 5603, 0, 0, 6 , 43, 1 > | 0 0 0 0 41 41 34 40 41 > 33 43 8 8 > 1 | 5603, 5603, 0, 0, 12 , 44, 24 > | 0 0 0 0 39 41 41 34 40 > 33 33 8 8 > 2 | 5603, 5603, 0, 0, 8 , 43, 5 > | 0 0 0 0 38 41 42 40 40 > 40 42 9 8 > 3 | 5603, 5603, 0, 0, 6 , 43, 1 > | 0 0 0 0 36 41 34 42 43 > 8 35 40 41 > *** TRex is shutting down - cause: 'CTRL + C detected' > > 2) with RTE_LIBRTE_I40E_ITR_INTERVAL 4 > > itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | #define > RTE_LIBRTE_I40E_ITR_INTERVAL 4 > > latency : 40usec > > -Latency stats enabled > Cpu Utilization : 0.2 % > if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter > max > window > | , , , , average , max , (usec) > > --------------------------------------------------------------------------------------------- > ------------------- > 0 | 5034, 5034, 0, 0, 10 , 42, 0 > | 0 0 0 0 0 42 40 38 24 > 32 23 24 23 > 1 | 5034, 5034, 0, 0, 8 , 43, 0 > | 0 0 0 0 0 43 37 38 19 > 20 30 18 21 > 2 | 5034, 5034, 0, 0, 8 , 45, 0 > | 0 0 0 0 0 37 40 41 40 > 41 41 40 45 > 3 | 5034, 5034, 0, 0, 8 , 48, 0 > | 0 0 0 0 0 42 36 43 44 > 43 43 44 48 > * > > > 3) RTE_LIBRTE_I40E_ITR_INTERVAL 0 > > itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | #define > RTE_LIBRTE_I40E_ITR_INTERVAL 0 > > latency : 40usec > > -Latency stats enabled > Cpu Utilization : 0.2 % > if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter > max > window > | , , , , average , max , (usec) > > --------------------------------------------------------------------------------------------- > ------------------- > 0 | 5034, 5034, 0, 0, 10 , 42, 0 > | 0 0 0 0 0 42 40 38 24 > 32 23 24 23 > 1 | 5034, 5034, 0, 0, 8 , 43, 0 > | 0 0 0 0 0 43 37 38 19 > 20 30 18 21 > 2 | 5034, 5034, 0, 0, 8 , 45, 0 > | 0 0 0 0 0 37 40 41 40 > 41 41 40 45 > 3 | 5034, 5034, 0, 0, 8 , 48, 0 > | 0 0 0 0 0 42 36 43 44 > 43 43 44 48 > * > > 4) TRex patch issue solved > > I40E_QINT_RQCTL_ITR_INDX_MASK > #define RTE_LIBRTE_I40E_ITR_INTERVAL -1 > > latency : 8usec > > -Latency stats enabled > Cpu Utilization : 0.1 % > if| tx_ok , rx_ok , rx check ,error, latency (usec) , Jitter > max > window > | , , , , average , max , (usec) > > --------------------------------------------------------------------------------------------- > ------------------- > 0 | 9501, 9501, 0, 0, 7 , 21, 0 > | 21 9 9 12 9 9 14 12 8 9 > 9 9 8 > 1 | 9501, 9501, 0, 0, 7 , 25, 0 > | 22 8 9 12 9 9 15 12 8 8 > 9 9 8 > 2 | 9501, 9501, 0, 0, 6 , 26, 0 > | 22 9 10 9 10 10 15 13 8 > 9 9 9 8 > 3 | 9501, 9501, 0, 0, 6 , 32, 0 > | 22 8 9 9 9 9 15 12 8 7 > 9 9 9 > > Thanks, > Hanoh > > From: Wu, Jingjing [mailto:jingjing...@intel.com] > Sent: Tuesday, November 28, 2017 5:01 PM > To: Hanoch Haim (hhaim) > Cc: dev@dpdk.org<mailto:dev@dpdk.org> > Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling > setting in PF > > Hi, Hanoch > > If DPDK PF, the commit affects that because it introduces an argument > (itr_idx) for i40e_vsi_enable_queues_intr. And use the default itr_idx with > default value 32us. > > If you'd like to get the descriptor write back immediately, you can set > "CONFIG_RTE_LIBRTE_I40E_ITR_INTERVAL=0" in config/common_base file. > > Or you can just change the definition of > I40E_QUEUE_ITR_INTERVAL_DEFAULT like: > #define I40E_QUEUE_ITR_INTERVAL_DEFAULT 0 /* 0 us */ > > Thanks > Jingjing > > From: Hanoch Haim (hhaim) [mailto:hh...@cisco.com] > Sent: Tuesday, November 28, 2017 9:14 PM > To: Wu, Jingjing <jingjing...@intel.com<mailto:jingjing...@intel.com>> > Cc: dev@dpdk.org<mailto:dev@dpdk.org>; Hanoch Haim (hhaim) > <hh...@cisco.com<mailto:hh...@cisco.com>> > Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling > setting in PF > > Hi Jingjing, > > 1. The issue is with DPDK PF. > > 2. The rate is high ~10gb, one DP core, one latency core. > > 3. The fix is here > > /* Bind all RX queues to allocated MSIX interrupt */ > for (i = 0; i < nb_queue; i++) { > val = (msix_vect << > I40E_QINT_RQCTL_MSIX_INDX_SHIFT) | > #ifdef TREX_PATCH > I40E_QINT_RQCTL_ITR_INDX_MASK > | > << low latency 11b = NoITR > #else > itr_idx << I40E_QINT_RQCTL_ITR_INDX_SHIFT | > << high spkies > #endif > > I40E_WRITE_REG(hw, I40E_QINT_RQCTL(base_queue + i), val); > > The Interrupt Throttling ITR is configure using a different setting using a > different register here : > > > 4. The ITR_INTERVAL is 32 usec and it affect a different PF register > > #define I40E_ITR_INDEX_DEFAULT 0 > #define I40E_ITR_INDEX_NONE 3 > #define I40E_QUEUE_ITR_INTERVAL_DEFAULT 32 /* 32 us */ > > > 5. My question is why the VF configuration affects PF INT_INTERVAL ? Can > I remove my patch and fix this latency issue in the different way? > > Thanks, > Hanoh > > From: Wu, Jingjing [mailto:jingjing...@intel.com] > Sent: Tuesday, November 28, 2017 2:50 PM > To: Hanoch Haim (hhaim); dev@dpdk.org<mailto:dev@dpdk.org> > Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling > setting in PF > > Hi, Hanoch > > Are you talking about i40 VF's latency? And you are using DPDK PF as host > driver? > > In this case, we are setting the Interrupt Throttling (ITR) to be maximum. > That is to say, if the packet rate is very slow , the receive descriptor is > written > back when ITR timeout, otherwise it is written back when cache line is full > (4 > descriptors/packets). I think that's why you saw the latency is varying. > > If we change the ITR to minor, then huge number of interrupts will coming to > core which impact performance. > > > Thanks > Jingjing > > From: Hanoch Haim (hhaim) [mailto:hh...@cisco.com] > Sent: Friday, November 24, 2017 7:25 PM > To: dev@dpdk.org<mailto:dev@dpdk.org> > Cc: Wu, Jingjing <jingjing...@intel.com<mailto:jingjing...@intel.com>> > Subject: RE: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling > setting in PF > > Re-sending > > Hanoh > > From: Hanoch Haim (hhaim) > Sent: Monday, November 20, 2017 5:19 PM > To: dev@dpdk.org<mailto:dev@dpdk.org> > Cc: Wu, Jingjing (jingjing...@intel.com<mailto:jingjing...@intel.com>); > Hanoch Haim (hhaim) > Subject: [dpdk-dev] net/i40e: latency issue due fix interrupt throttling > setting > in PF > > Hi All, > While integrating dpdk17.11 into TRex latest code a new latency issue is > observed (i40e is very sensitive because it has very good resolution due to > Qos configuration). > git bitsec found the following commit. > With this commit we observe high spikes of Rx latency (~40usec) vs (~8usec). > Any idea why? > I can send how to reproduce this, it is very simple. > > cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e is the first bad commit > commit cfd662d22e7bddb4ba41dbd1384f8497f38c2b4e > Author: Jingjing Wu <jingjing...@intel.com<mailto:jingjing...@intel.com>> > Date: Thu Aug 24 09:57:51 2017 +0800 > > net/i40e: fix interrupt throttling setting in PF > > As no matter the PF host driver is DPDK or other kernel drivers, > they are sharing the same virtchnnl interfaces to communicate to VFs. > To follow the generic interface, DPDK PF need to set Interrupt > Throttling (ITR) index according to the rxitr_idx from virtchnnl > instead of ITR_NONE. > > Fixes: 6d59e4ea74a6 ("net/i40e: change version number to support Linux > VF") > Cc: sta...@dpdk.org<mailto:sta...@dpdk.org> > > Signed-off-by: Jingjing Wu > <jingjing...@intel.com<mailto:jingjing...@intel.com>> > > > > Thanks, > Hanoh
