Hey Nishit, The larger packets are less of a load on the PCIe bus so it can handle more traffic before you start overloading it.
If you are willing to sacrifice CPU cycles you could lower your buffer size like you doing earlier. Of course I would expect that to adversely affect the CPU usage. Thanks, -Don Skidmore <[email protected]> > -----Original Message----- > From: Nishit Shah [mailto:[email protected]] > Sent: Friday, May 17, 2013 5:22 AM > To: Skidmore, Donald C > Cc: [email protected] > Subject: Re: [E1000-devel] 82599 latency increase with rx_missed_errors > > > Hi Donald, > > IOMMU is disabled. > I do have one another observation. We don't have latency issue with > higher packet sizes. But when we reach the wire-line on higher packet sizes, > we are again seeing increase in latency. At that time CPU is free and no > rx_missed and rx_no_dma errors. > Is there anything trigger at that point ? > > Rgds, > Nishit Shah. > > On 5/3/2013 4:24 AM, Skidmore, Donald C wrote: > > Hey Nishit, > > > > Another engineer reminded me that IOMMU is never good for latency. > You should also make sure that is disabled. > > > > Thanks, > > -Don Skidmore<[email protected]> > > > > > >> -----Original Message----- > >> From: Skidmore, Donald C > >> Sent: Thursday, May 02, 2013 2:50 PM > >> To: Nishit Shah > >> Cc: [email protected] > >> Subject: RE: [E1000-devel] 82599 latency increase with > >> rx_missed_errors > >> > >> Hey Nishit, > >> > >> I'm not sure what I can tell you. It looks like your device is > >> loaded down with small packets we know this will overload the PCI > >> bus, this will of course adversely event latency. You could try > >> something like DCB. Put your traffic that needs low latency on one > >> TC and the small packet traffic on another and use FC on that TC. > >> This of course would require switches that support DCB and is > >> assuming your small packet traffic and latency dependent are > >> different flows. If they are not then you need some other mechanism to > throttle your traffic before it saturates the bus. > >> > >> Thanks, > >> -Don Skidmore<[email protected]> > >> > >>> -----Original Message----- > >>> From: Nishit Shah [mailto:[email protected]] > >>> Sent: Thursday, May 02, 2013 8:52 AM > >>> To: Skidmore, Donald C > >>> Cc: [email protected] > >>> Subject: Re: [E1000-devel] 82599 latency increase with > >>> rx_missed_errors > >>> > >>> > >>> Hi Don, > >>> > >>> Thanks for all the explanations. It will be a great help. > >>> Can you suggest some pointers to look more into PCIe bus > >>> configurations/tuning ? > >>> > >>> Rgds, > >>> Nishit Shah. > >>> > >>> On 5/2/2013 3:41 AM, Skidmore, Donald C wrote: > >>>> Hey Nishit, > >>>> > >>>> I replied inline below. > >>>> > >>>> Thanks, > >>>> -Don Skidmore<[email protected]> > >>>> > >>>>> -----Original Message----- > >>>>> From: Nishit Shah [mailto:[email protected]] > >>>>> Sent: Tuesday, April 30, 2013 11:46 PM > >>>>> To: Skidmore, Donald C > >>>>> Cc: [email protected] > >>>>> Subject: Re: [E1000-devel] 82599 latency increase with > >>>>> rx_missed_errors > >>>>> > >>>>> > >>>>> Hi Don, > >>>>> > >>>>> On 5/1/2013 3:40 AM, Skidmore, Donald C wrote: > >>>>>> Hi Nishit, > >>>>>> > >>>>>> The rx_no_dma_resources means we are dropping packets because > >> we > >>>>> don't have any free descriptors in the RX Queue. While > >>>>> rx_missed_errors are due to insufficient space to store an ingress > >>>>> packet, so basically we ran out of buffers or bandwidth on the PCIe > bus. > >>>>> > >>>>> Thanks for the explanation. Is there any way to verify whether we > >>>>> are running out of buffers or bandwidth on the PCIe bus ? > >>>> To a large extent these are transparent to the driver. You could > >>>> put on FC > >>> and if you still see that same number of rx_missed_errors this would > >>> infer that you are overloading your bus. > >>>>> (ixgbe loading shows PCIe 5.0GT/s:Width x8) > >>>>> > >>>>> # dmesg | grep "0000:06:00" > >>>>> ixgbe 0000:06:00.0: PCI->APIC IRQ transform: INT C -> IRQ > >>>>> 18 > >>>>> ixgbe 0000:06:00.0: setting latency timer to 64 > >>>>> ixgbe 0000:06:00.0: irq 40 for MSI/MSI-X > >>>>> ixgbe 0000:06:00.0: irq 41 for MSI/MSI-X > >>>>> ixgbe 0000:06:00.0: irq 42 for MSI/MSI-X > >>>>> ixgbe 0000:06:00.0: (PCI Express:5.0GT/s:Width x8) > >>>>> 00:90:fb:45:f1:76 > >>>>> ixgbe 0000:06:00.0: eth0: MAC: 2, PHY: 14, SFP+: 5, PBA No: > >>>>> ixgbe 0000:06:00.0: eth0: Enabled Features: RxQ: 2 TxQ: > >>>>> 2 FdirHash > >>> RSS > >>>>> ixgbe 0000:06:00.0: eth0: Intel(R) 10 Gigabit Network > >>>>> Connection > >>>>> ixgbe 0000:06:00.1: PCI->APIC IRQ transform: INT D -> IRQ > >>>>> 19 > >>>>> ixgbe 0000:06:00.1: setting latency timer to 64 > >>>>> ixgbe 0000:06:00.1: irq 43 for MSI/MSI-X > >>>>> ixgbe 0000:06:00.1: irq 44 for MSI/MSI-X > >>>>> ixgbe 0000:06:00.1: irq 45 for MSI/MSI-X > >>>>> ixgbe 0000:06:00.1: (PCI Express:5.0GT/s:Width x8) > >>>>> 00:90:fb:45:f1:77 > >>>>> ixgbe 0000:06:00.1: eth1: MAC: 2, PHY: 14, SFP+: 6, PBA No: > >>>>> ixgbe 0000:06:00.1: eth1: Enabled Features: RxQ: 2 TxQ: > >>>>> 2 FdirHash > >>> RSS > >>>>> ixgbe 0000:06:00.1: eth1: Intel(R) 10 Gigabit Network > >>>>> Connection > >>>>> > >>>>> One another interesting observation. > >>>>> When we have changed the packet buffer from 512 KB to 128 > >>>>> KB (by changing rx_pb_size in ixgbe_82599.c), per packet latency > >>>>> is reduced from > >>>>> 500 microseconds to 100 microseconds for 64 bytes packets. > >>>>> Does it mean some kind of relation with size of packet buffer ? > >>>> This most likely shows that your small packet flow is overloading > >>>> the PCIe > >>> bus. Since shrinking the packet buffer would me it would take less > >>> time backfill while you wait on the saturated PCIe bus. So when you > >>> shrink the packet buffer you don't wait as long queued up for PCIe > >>> bus that can't handle your load. So I'm wouldn't think this would > >>> be a solution for you the real issue here is that the bus can't keep > >>> up with small > >> packets loads. > >>>>>> All that said when you see the rx_no_dma_resources errors is > >>>>>> their rate > >>>>> comparable with what you were seeing for rx_missed_errors? Both > >>>>> will lead to the same thing, dropped packets. > >>>>> > >>>>> I have find out the frame size from where we are getting > >>>>> rx_missed_errors and below is a rate at those sizes. > >>>>> > >>>>> frame size 110 bytes (avg. latency 45 microseconds) > >>>>> - no rx_missed_errors. > >>>>> - rx_no_dma_resources increase rate is 8200000/sec > >>>>> > >>>>> frame size 108 bytes (avg. latency 345 microseconds) > >>>>> - rx_missed_errors increase rate is 207000/sec > >>>>> - rx_no_dma_resources increase rate is 8300000/sec > >>>> I assume you crossed some boundary here with the lower frame size > >>>> you > >>> have more PCI transaction for the same data throughput. So once you > >>> crossed to 108 bytes you start running out of PCIe bandwidth. > >>>>>> Also what does 'lspci -vvv' show, I'm looking to see if you are > >>>>>> getting the full > >>>>> PCIe bandwidth. You could also try to turn on FC which should > >>>>> lower these types of overflow occurrences. > >>>>> > >>>>> Enabling flow control is even increasing the latency. Seems > >>>>> to be tester machine is not understanding the PAUSE frames and FC > >>>>> also clears the DROP_EN bit that is again increasing the latency. > >>>>> lspci -vvv output is attached with the mail. > >>>>> > >>>>>> Thanks, > >>>>>> -Don Skidmore<[email protected]> > >>>>> Rgds, > >>>>> Nishit Shah. > >>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: Nishit Shah [mailto:[email protected]] > >>>>>>> Sent: Tuesday, April 30, 2013 9:07 AM > >>>>>>> To: [email protected] > >>>>>>> Subject: [E1000-devel] 82599 latency increase with > >>>>>>> rx_missed_errors > >>>>>>> > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> We are measuring packet latencies at various packet > >>>>>>> sizes > >>>>>>> (64 bytes to > >>>>>>> 1518 bytes) with 82599 card with ixgbe driver 3.7.21. > >>>>>>> > >>>>>>> Setup: > >>>>>>> > >>>>>>> Spirent test center sender machine with > >>>>>>> 82599 > >>>>>>> (ixgbe 3.7.21 and vanilla 2.6.39.4) Spirent test center > >>>>>>> receiver > >>>>>>> > >>>>>>> 10 G<------------------------> 10G > >>>>>>> 10G<------------------------------> 10G > >>>>>>> > >>>>>>> When we don't have an increase in "rx_missed_errors" and > >>>>>>> "rx_no_dma_resources", we are getting per packet latency around > >>>>>>> 40-70 microseconds. ("rx_no_buffer_count" is not increasing) > >>>>>>> When we have an increase in "rx_no_dma_resources", we > >>>>>>> are still > >>>>> getting > >>>>>>> per packet latency around 40-70 microseconds. > >>>>>>> ("rx_no_buffer_count" is not increasing) > >>>>>>> When we have an increase in "rx_missed_errors", we are > >>>>>>> getting per packet latency around 500 microseconds. > >>>>>>> (rx_no_buffer_count is not > >>>>>>> increasing) > >>>>>>> > >>>>>>> Is there any specific reason for latency increase when > >>>>> "rx_missed_errors" > >>>>>>> are increased ? > >>>>>>> Is there a way to control it ? > >>>>>>> > >>>>>>> Below is a machine detail. > >>>>>>> > >> > ========================================================== > >>>>>>> =============================================== > >>>>>>> Machine details. > >>>>>>> > >>>>>>> CPU: Dual Core Intel(R) Celeron(R) CPU G540 @ > >>>>>>> 2.50GHz > >>>>>>> Memory: 2 GB > >>>>>>> kernel: vanilla 2.6.39.4 > >>>>>>> Interface tuning parameters: > >>>>>>> Auto Negotiation is off (DROP_EN is set.) > >>>>>>> ethtool -G eth0 rx 64 tx 128 ; ethtool -G eth1 rx > >>>>>>> 64 tx 128 > >>>>>>> rx-usecs is set to 50. > >>>>>>> ethtool and lspci for bus information: > >>>>>>> > >>>>>>> # ethtool -i eth0 > >>>>>>> driver: ixgbe > >>>>>>> version: 3.7.21-NAPI > >>>>>>> firmware-version: 0x80000345 > >>>>>>> bus-info: 0000:06:00.0 > >>>>>>> # > >>>>>>> # ethtool -i eth1 > >>>>>>> driver: ixgbe > >>>>>>> version: 3.7.21-NAPI > >>>>>>> firmware-version: 0x80000345 > >>>>>>> bus-info: 0000:06:00.1 > >>>>>>> > >>>>>>> 06:00.0 Class 0200: Device 8086:10fb (rev 01) > >>>>>>> Subsystem: Device 15bb:30e0 > >>>>>>> Control: I/O+ Mem+ BusMaster+ SpecCycle- > >>>>>>> MemWINV- > >>>>> VGASnoop- > >>>>>>> ParErr- Stepping- SERR- FastB2B- DisINTx+ > >>>>>>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast > >>>>>>> >TAbort-<TAbort-<MAbort->SERR-<PERR- INTx- > >>>>>>> Latency: 0, Cache Line Size: 64 bytes > >>>>>>> Interrupt: pin A routed to IRQ 18 > >>>>>>> Region 0: Memory at f7520000 (64-bit, > >>>>>>> non-prefetchable) > >>>>> [size=128K] > >>>>>>> Region 2: I/O ports at 8020 [size=32] > >>>>>>> Region 4: Memory at f7544000 (64-bit, > >>>>>>> non-prefetchable) > >>>>> [size=16K] > >>>>>>> Capabilities: [40] Power Management version 3 > >>>>>>> Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA > >>>>>>> PME(D0+,D1-,D2-,D3hot+,D3cold-) > >>>>>>> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 > PME- > >>>>>>> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ > >>>>>>> 64bit+ > >>>>>>> Address: 0000000000000000 Data: 0000 > >>>>>>> Masking: 00000000 Pending: 00000000 > >>>>>>> Capabilities: [70] MSI-X: Enable+ Count=64 Masked- > >>>>>>> Vector table: BAR=4 offset=00000000 > >>>>>>> PBA: BAR=4 offset=00002000 > >>>>>>> Capabilities: [a0] Express (v2) Endpoint, MSI 00 > >>>>>>> DevCap: MaxPayload 512 bytes, PhantFunc 0, > >>>>>>> Latency L0s<512ns, > >>>>> L1 > >>>>>>> <64us > >>>>>>> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ > >>>>>>> FLReset+ > >>>>>>> DevCtl: Report errors: Correctable- > >>>>>>> Non-Fatal- > >>>>>>> Fatal- > >>>>>>> Unsupported- > >>>>>>> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- > >>>>>>> NoSnoop+ > >> FLReset- > >>>>>>> MaxPayload 128 bytes, MaxReadReq 512 bytes > >>>>>>> DevSta: CorrErr+ UncorrErr- FatalErr- > >>>>>>> UnsuppReq+ > >>>>>>> AuxPwr- TransPend- > >>>>>>> LnkCap: Port #1, Speed 5GT/s, Width x8, ASPM > >>>>>>> L0s, Latency > >>>>> L0<2us, > >>>>>>> L1<32us > >>>>>>> ClockPM- Surprise- LLActRep- BwNot- > >>>>>>> LnkCtl: ASPM Disabled; RCB 64 bytes > >>>>>>> Disabled- > >>>>>>> Retrain- > >>>>>>> CommClk- > >>>>>>> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > >>>>>>> LnkSta: Speed 5GT/s, Width x8, TrErr- Train- > >>>>>>> SlotClk+ > >>>>>>> DLActive- BWMgmt- ABWMgmt- > >>>>>>> DevCap2: Completion Timeout: Range ABCD, > TimeoutDis+ > >>>>>>> DevCtl2: Completion Timeout: 50us to 50ms, > >>>>>>> TimeoutDis- > >>>>>>> LnkCtl2: Target Link Speed: 5GT/s, > >>>>>>> EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB > >>>>>>> Transmit Margin: Normal Operating > >>>>>>> Range, > >>>>>>> EnterModifiedCompliance- ComplianceSOS- > >>>>>>> Compliance De-emphasis: -6dB > >>>>>>> LnkSta2: Current De-emphasis Level: -6dB > >>>>>>> Capabilities: [e0] Vital Product Data > >>>>>>> Unknown small resource type 06, will not decode > >>>>>>> more. > >>>>>>> Kernel driver in use: ixgbe > >>>>>>> Kernel modules: ixgbe > >>>>>>> > >>>>>>> 06:00.1 Class 0200: Device 8086:10fb (rev 01) > >>>>>>> Subsystem: Device 15bb:30e0 > >>>>>>> Control: I/O+ Mem+ BusMaster+ SpecCycle- > >>>>>>> MemWINV- > >>>>> VGASnoop- > >>>>>>> ParErr- Stepping- SERR- FastB2B- DisINTx+ > >>>>>>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast > >>>>>>> >TAbort-<TAbort-<MAbort->SERR-<PERR- INTx- > >>>>>>> Latency: 0, Cache Line Size: 64 bytes > >>>>>>> Interrupt: pin B routed to IRQ 19 > >>>>>>> Region 0: Memory at f7500000 (64-bit, > >>>>>>> non-prefetchable) > >>>>> [size=128K] > >>>>>>> Region 2: I/O ports at 8000 [size=32] > >>>>>>> Region 4: Memory at f7540000 (64-bit, > >>>>>>> non-prefetchable) > >>>>> [size=16K] > >>>>>>> Capabilities: [40] Power Management version 3 > >>>>>>> Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA > >>>>>>> PME(D0+,D1-,D2-,D3hot+,D3cold-) > >>>>>>> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 > PME- > >>>>>>> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ > >>>>>>> 64bit+ > >>>>>>> Address: 0000000000000000 Data: 0000 > >>>>>>> Masking: 00000000 Pending: 00000000 > >>>>>>> Capabilities: [70] MSI-X: Enable+ Count=64 Masked- > >>>>>>> Vector table: BAR=4 offset=00000000 > >>>>>>> PBA: BAR=4 offset=00002000 > >>>>>>> Capabilities: [a0] Express (v2) Endpoint, MSI 00 > >>>>>>> DevCap: MaxPayload 512 bytes, PhantFunc 0, > >>>>>>> Latency L0s<512ns, > >>>>> L1 > >>>>>>> <64us > >>>>>>> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ > >>>>>>> FLReset+ > >>>>>>> DevCtl: Report errors: Correctable- > >>>>>>> Non-Fatal- > >>>>>>> Fatal- > >>>>>>> Unsupported- > >>>>>>> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- > >>>>>>> NoSnoop+ > >> FLReset- > >>>>>>> MaxPayload 128 bytes, MaxReadReq 512 bytes > >>>>>>> DevSta: CorrErr+ UncorrErr- FatalErr- > >>>>>>> UnsuppReq+ > >>>>>>> AuxPwr- TransPend- > >>>>>>> LnkCap: Port #1, Speed 5GT/s, Width x8, ASPM > >>>>>>> L0s, Latency > >>>>> L0<2us, > >>>>>>> L1<32us > >>>>>>> ClockPM- Surprise- LLActRep- BwNot- > >>>>>>> LnkCtl: ASPM Disabled; RCB 64 bytes > >>>>>>> Disabled- > >>>>>>> Retrain- > >>>>>>> CommClk- > >>>>>>> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > >>>>>>> LnkSta: Speed 5GT/s, Width x8, TrErr- Train- > >>>>>>> SlotClk+ > >>>>>>> DLActive- BWMgmt- ABWMgmt- > >>>>>>> DevCap2: Completion Timeout: Range ABCD, > TimeoutDis+ > >>>>>>> DevCtl2: Completion Timeout: 50us to 50ms, > >>>>>>> TimeoutDis- > >>>>>>> LnkCtl2: Target Link Speed: 2.5GT/s, > >>>>>>> EnterCompliance- SpeedDis- > >>>>> , > >>>>>>> Selectable De-emphasis: -6dB > >>>>>>> Transmit Margin: Normal Operating > >>>>>>> Range, > >>>>>>> EnterModifiedCompliance- ComplianceSOS- > >>>>>>> Compliance De-emphasis: -6dB > >>>>>>> LnkSta2: Current De-emphasis Level: -6dB > >>>>>>> Capabilities: [e0] Vital Product Data > >>>>>>> Unknown small resource type 06, will not decode > >>>>>>> more. > >>>>>>> Kernel driver in use: ixgbe > >>>>>>> Kernel modules: ixgbe > >>>>>>> > >> > ========================================================== > >>>>>>> =============================================== > >>>>>>> > >>>>>>> Rgds, > >>>>>>> Nishit Shah. > >>>>>>> > >>>>>>> ---------------------------------------------------------------- > >>>>>>> - > >>>>>>> -- > >>>>>>> ----------- Introducing AppDynamics Lite, a free troubleshooting > >>>>>>> tool for Java/.NET > >>>>> Get > >>>>>>> 100% visibility into your production application - at no cost. > >>>>>>> Code-level diagnostics for performance bottlenecks with<2% > >>>>>>> overhead Download for free and get started troubleshooting in > >> minutes. > >>>>>>> http://p.sf.net/sfu/appdyn_d2d_ap1 > >>>>>>> _______________________________________________ > >>>>>>> E1000-devel mailing list > >>>>>>> [email protected] > >>>>>>> https://lists.sourceforge.net/lists/listinfo/e1000-devel > >>>>>>> To learn more about Intel® Ethernet, visit > >>>>>>> http://communities.intel.com/community/wired ------------------------------------------------------------------------------ AlienVault Unified Security Management (USM) platform delivers complete security visibility with the essential security capabilities. Easily and efficiently configure, manage, and operate all of your security controls from a single console and one unified framework. Download a free trial. http://p.sf.net/sfu/alienvault_d2d _______________________________________________ E1000-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
