Hey Nishit, Another engineer reminded me that IOMMU is never good for latency. You should also make sure that is disabled.
Thanks, -Don Skidmore <[email protected]> > -----Original Message----- > From: Skidmore, Donald C > Sent: Thursday, May 02, 2013 2:50 PM > To: Nishit Shah > Cc: [email protected] > Subject: RE: [E1000-devel] 82599 latency increase with rx_missed_errors > > Hey Nishit, > > I'm not sure what I can tell you. It looks like your device is loaded down > with > small packets we know this will overload the PCI bus, this will of course > adversely event latency. You could try something like DCB. Put your traffic > that needs low latency on one TC and the small packet traffic on another and > use FC on that TC. This of course would require switches that support DCB > and is assuming your small packet traffic and latency dependent are different > flows. If they are not then you need some other mechanism to throttle your > traffic before it saturates the bus. > > Thanks, > -Don Skidmore <[email protected]> > > > -----Original Message----- > > From: Nishit Shah [mailto:[email protected]] > > Sent: Thursday, May 02, 2013 8:52 AM > > To: Skidmore, Donald C > > Cc: [email protected] > > Subject: Re: [E1000-devel] 82599 latency increase with > > rx_missed_errors > > > > > > Hi Don, > > > > Thanks for all the explanations. It will be a great help. > > Can you suggest some pointers to look more into PCIe bus > > configurations/tuning ? > > > > Rgds, > > Nishit Shah. > > > > On 5/2/2013 3:41 AM, Skidmore, Donald C wrote: > > > Hey Nishit, > > > > > > I replied inline below. > > > > > > Thanks, > > > -Don Skidmore<[email protected]> > > > > > >> -----Original Message----- > > >> From: Nishit Shah [mailto:[email protected]] > > >> Sent: Tuesday, April 30, 2013 11:46 PM > > >> To: Skidmore, Donald C > > >> Cc: [email protected] > > >> Subject: Re: [E1000-devel] 82599 latency increase with > > >> rx_missed_errors > > >> > > >> > > >> Hi Don, > > >> > > >> On 5/1/2013 3:40 AM, Skidmore, Donald C wrote: > > >>> Hi Nishit, > > >>> > > >>> The rx_no_dma_resources means we are dropping packets because > we > > >> don't have any free descriptors in the RX Queue. While > > >> rx_missed_errors are due to insufficient space to store an ingress > > >> packet, so basically we ran out of buffers or bandwidth on the PCIe bus. > > >> > > >> Thanks for the explanation. Is there any way to verify whether we > > >> are running out of buffers or bandwidth on the PCIe bus ? > > > To a large extent these are transparent to the driver. You could > > > put on FC > > and if you still see that same number of rx_missed_errors this would > > infer that you are overloading your bus. > > > > > >> (ixgbe loading shows PCIe 5.0GT/s:Width x8) > > >> > > >> # dmesg | grep "0000:06:00" > > >> ixgbe 0000:06:00.0: PCI->APIC IRQ transform: INT C -> IRQ 18 > > >> ixgbe 0000:06:00.0: setting latency timer to 64 > > >> ixgbe 0000:06:00.0: irq 40 for MSI/MSI-X > > >> ixgbe 0000:06:00.0: irq 41 for MSI/MSI-X > > >> ixgbe 0000:06:00.0: irq 42 for MSI/MSI-X > > >> ixgbe 0000:06:00.0: (PCI Express:5.0GT/s:Width x8) > > >> 00:90:fb:45:f1:76 > > >> ixgbe 0000:06:00.0: eth0: MAC: 2, PHY: 14, SFP+: 5, PBA No: > > >> ixgbe 0000:06:00.0: eth0: Enabled Features: RxQ: 2 TxQ: 2 > > >> FdirHash > > RSS > > >> ixgbe 0000:06:00.0: eth0: Intel(R) 10 Gigabit Network > > >> Connection > > >> ixgbe 0000:06:00.1: PCI->APIC IRQ transform: INT D -> IRQ 19 > > >> ixgbe 0000:06:00.1: setting latency timer to 64 > > >> ixgbe 0000:06:00.1: irq 43 for MSI/MSI-X > > >> ixgbe 0000:06:00.1: irq 44 for MSI/MSI-X > > >> ixgbe 0000:06:00.1: irq 45 for MSI/MSI-X > > >> ixgbe 0000:06:00.1: (PCI Express:5.0GT/s:Width x8) > > >> 00:90:fb:45:f1:77 > > >> ixgbe 0000:06:00.1: eth1: MAC: 2, PHY: 14, SFP+: 6, PBA No: > > >> ixgbe 0000:06:00.1: eth1: Enabled Features: RxQ: 2 TxQ: 2 > > >> FdirHash > > RSS > > >> ixgbe 0000:06:00.1: eth1: Intel(R) 10 Gigabit Network > > >> Connection > > >> > > >> One another interesting observation. > > >> When we have changed the packet buffer from 512 KB to 128 KB > > >> (by changing rx_pb_size in ixgbe_82599.c), per packet latency is > > >> reduced from > > >> 500 microseconds to 100 microseconds for 64 bytes packets. > > >> Does it mean some kind of relation with size of packet buffer ? > > > This most likely shows that your small packet flow is overloading > > > the PCIe > > bus. Since shrinking the packet buffer would me it would take less > > time backfill while you wait on the saturated PCIe bus. So when you > > shrink the packet buffer you don't wait as long queued up for PCIe bus > > that can't handle your load. So I'm wouldn't think this would be a > > solution for you the real issue here is that the bus can't keep up with > > small > packets loads. > > > > > >>> > > >>> All that said when you see the rx_no_dma_resources errors is > > >>> their rate > > >> comparable with what you were seeing for rx_missed_errors? Both > > >> will lead to the same thing, dropped packets. > > >> > > >> I have find out the frame size from where we are getting > > >> rx_missed_errors and below is a rate at those sizes. > > >> > > >> frame size 110 bytes (avg. latency 45 microseconds) > > >> - no rx_missed_errors. > > >> - rx_no_dma_resources increase rate is 8200000/sec > > >> > > >> frame size 108 bytes (avg. latency 345 microseconds) > > >> - rx_missed_errors increase rate is 207000/sec > > >> - rx_no_dma_resources increase rate is 8300000/sec > > > I assume you crossed some boundary here with the lower frame size > > > you > > have more PCI transaction for the same data throughput. So once you > > crossed to 108 bytes you start running out of PCIe bandwidth. > > > > > >>> > > >>> Also what does 'lspci -vvv' show, I'm looking to see if you are > > >>> getting the full > > >> PCIe bandwidth. You could also try to turn on FC which should > > >> lower these types of overflow occurrences. > > >> > > >> Enabling flow control is even increasing the latency. Seems > > >> to be tester machine is not understanding the PAUSE frames and FC > > >> also clears the DROP_EN bit that is again increasing the latency. > > >> lspci -vvv output is attached with the mail. > > >> > > >>> Thanks, > > >>> -Don Skidmore<[email protected]> > > >> Rgds, > > >> Nishit Shah. > > >> > > >>>> -----Original Message----- > > >>>> From: Nishit Shah [mailto:[email protected]] > > >>>> Sent: Tuesday, April 30, 2013 9:07 AM > > >>>> To: [email protected] > > >>>> Subject: [E1000-devel] 82599 latency increase with > > >>>> rx_missed_errors > > >>>> > > >>>> > > >>>> Hi, > > >>>> > > >>>> We are measuring packet latencies at various packet sizes > > >>>> (64 bytes to > > >>>> 1518 bytes) with 82599 card with ixgbe driver 3.7.21. > > >>>> > > >>>> Setup: > > >>>> > > >>>> Spirent test center sender machine with > > >>>> 82599 > > >>>> (ixgbe 3.7.21 and vanilla 2.6.39.4) Spirent test center receiver > > >>>> > > >>>> 10 G<------------------------> 10G > > >>>> 10G<------------------------------> 10G > > >>>> > > >>>> When we don't have an increase in "rx_missed_errors" and > > >>>> "rx_no_dma_resources", we are getting per packet latency around > > >>>> 40-70 microseconds. ("rx_no_buffer_count" is not increasing) > > >>>> When we have an increase in "rx_no_dma_resources", we are > > >>>> still > > >> getting > > >>>> per packet latency around 40-70 microseconds. > > >>>> ("rx_no_buffer_count" is not increasing) > > >>>> When we have an increase in "rx_missed_errors", we are > > >>>> getting per packet latency around 500 microseconds. > > >>>> (rx_no_buffer_count is not > > >>>> increasing) > > >>>> > > >>>> Is there any specific reason for latency increase when > > >> "rx_missed_errors" > > >>>> are increased ? > > >>>> Is there a way to control it ? > > >>>> > > >>>> Below is a machine detail. > > >>>> > > >> > > > ========================================================== > > >>>> =============================================== > > >>>> Machine details. > > >>>> > > >>>> CPU: Dual Core Intel(R) Celeron(R) CPU G540 @ 2.50GHz > > >>>> Memory: 2 GB > > >>>> kernel: vanilla 2.6.39.4 > > >>>> Interface tuning parameters: > > >>>> Auto Negotiation is off (DROP_EN is set.) > > >>>> ethtool -G eth0 rx 64 tx 128 ; ethtool -G eth1 rx > > >>>> 64 tx 128 > > >>>> rx-usecs is set to 50. > > >>>> ethtool and lspci for bus information: > > >>>> > > >>>> # ethtool -i eth0 > > >>>> driver: ixgbe > > >>>> version: 3.7.21-NAPI > > >>>> firmware-version: 0x80000345 > > >>>> bus-info: 0000:06:00.0 > > >>>> # > > >>>> # ethtool -i eth1 > > >>>> driver: ixgbe > > >>>> version: 3.7.21-NAPI > > >>>> firmware-version: 0x80000345 > > >>>> bus-info: 0000:06:00.1 > > >>>> > > >>>> 06:00.0 Class 0200: Device 8086:10fb (rev 01) > > >>>> Subsystem: Device 15bb:30e0 > > >>>> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- > > >> VGASnoop- > > >>>> ParErr- Stepping- SERR- FastB2B- DisINTx+ > > >>>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast > > >>>> >TAbort-<TAbort-<MAbort->SERR-<PERR- INTx- > > >>>> Latency: 0, Cache Line Size: 64 bytes > > >>>> Interrupt: pin A routed to IRQ 18 > > >>>> Region 0: Memory at f7520000 (64-bit, > > >>>> non-prefetchable) > > >> [size=128K] > > >>>> Region 2: I/O ports at 8020 [size=32] > > >>>> Region 4: Memory at f7544000 (64-bit, > > >>>> non-prefetchable) > > >> [size=16K] > > >>>> Capabilities: [40] Power Management version 3 > > >>>> Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA > > >>>> PME(D0+,D1-,D2-,D3hot+,D3cold-) > > >>>> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 > > >>>> PME- > > >>>> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ > > >>>> 64bit+ > > >>>> Address: 0000000000000000 Data: 0000 > > >>>> Masking: 00000000 Pending: 00000000 > > >>>> Capabilities: [70] MSI-X: Enable+ Count=64 Masked- > > >>>> Vector table: BAR=4 offset=00000000 > > >>>> PBA: BAR=4 offset=00002000 > > >>>> Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > >>>> DevCap: MaxPayload 512 bytes, PhantFunc 0, > > >>>> Latency L0s<512ns, > > >> L1 > > >>>> <64us > > >>>> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ > > >>>> DevCtl: Report errors: Correctable- Non-Fatal- > > >>>> Fatal- > > >>>> Unsupported- > > >>>> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ > FLReset- > > >>>> MaxPayload 128 bytes, MaxReadReq 512 bytes > > >>>> DevSta: CorrErr+ UncorrErr- FatalErr- > > >>>> UnsuppReq+ > > >>>> AuxPwr- TransPend- > > >>>> LnkCap: Port #1, Speed 5GT/s, Width x8, ASPM > > >>>> L0s, Latency > > >> L0<2us, > > >>>> L1<32us > > >>>> ClockPM- Surprise- LLActRep- BwNot- > > >>>> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- > > >>>> Retrain- > > >>>> CommClk- > > >>>> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > > >>>> LnkSta: Speed 5GT/s, Width x8, TrErr- Train- > > >>>> SlotClk+ > > >>>> DLActive- BWMgmt- ABWMgmt- > > >>>> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ > > >>>> DevCtl2: Completion Timeout: 50us to 50ms, > > >>>> TimeoutDis- > > >>>> LnkCtl2: Target Link Speed: 5GT/s, > > >>>> EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB > > >>>> Transmit Margin: Normal Operating Range, > > >>>> EnterModifiedCompliance- ComplianceSOS- > > >>>> Compliance De-emphasis: -6dB > > >>>> LnkSta2: Current De-emphasis Level: -6dB > > >>>> Capabilities: [e0] Vital Product Data > > >>>> Unknown small resource type 06, will not decode > > >>>> more. > > >>>> Kernel driver in use: ixgbe > > >>>> Kernel modules: ixgbe > > >>>> > > >>>> 06:00.1 Class 0200: Device 8086:10fb (rev 01) > > >>>> Subsystem: Device 15bb:30e0 > > >>>> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- > > >> VGASnoop- > > >>>> ParErr- Stepping- SERR- FastB2B- DisINTx+ > > >>>> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast > > >>>> >TAbort-<TAbort-<MAbort->SERR-<PERR- INTx- > > >>>> Latency: 0, Cache Line Size: 64 bytes > > >>>> Interrupt: pin B routed to IRQ 19 > > >>>> Region 0: Memory at f7500000 (64-bit, > > >>>> non-prefetchable) > > >> [size=128K] > > >>>> Region 2: I/O ports at 8000 [size=32] > > >>>> Region 4: Memory at f7540000 (64-bit, > > >>>> non-prefetchable) > > >> [size=16K] > > >>>> Capabilities: [40] Power Management version 3 > > >>>> Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA > > >>>> PME(D0+,D1-,D2-,D3hot+,D3cold-) > > >>>> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 > > >>>> PME- > > >>>> Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ > > >>>> 64bit+ > > >>>> Address: 0000000000000000 Data: 0000 > > >>>> Masking: 00000000 Pending: 00000000 > > >>>> Capabilities: [70] MSI-X: Enable+ Count=64 Masked- > > >>>> Vector table: BAR=4 offset=00000000 > > >>>> PBA: BAR=4 offset=00002000 > > >>>> Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > >>>> DevCap: MaxPayload 512 bytes, PhantFunc 0, > > >>>> Latency L0s<512ns, > > >> L1 > > >>>> <64us > > >>>> ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ > > >>>> DevCtl: Report errors: Correctable- Non-Fatal- > > >>>> Fatal- > > >>>> Unsupported- > > >>>> RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ > FLReset- > > >>>> MaxPayload 128 bytes, MaxReadReq 512 bytes > > >>>> DevSta: CorrErr+ UncorrErr- FatalErr- > > >>>> UnsuppReq+ > > >>>> AuxPwr- TransPend- > > >>>> LnkCap: Port #1, Speed 5GT/s, Width x8, ASPM > > >>>> L0s, Latency > > >> L0<2us, > > >>>> L1<32us > > >>>> ClockPM- Surprise- LLActRep- BwNot- > > >>>> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- > > >>>> Retrain- > > >>>> CommClk- > > >>>> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > > >>>> LnkSta: Speed 5GT/s, Width x8, TrErr- Train- > > >>>> SlotClk+ > > >>>> DLActive- BWMgmt- ABWMgmt- > > >>>> DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ > > >>>> DevCtl2: Completion Timeout: 50us to 50ms, > > >>>> TimeoutDis- > > >>>> LnkCtl2: Target Link Speed: 2.5GT/s, > > >>>> EnterCompliance- SpeedDis- > > >> , > > >>>> Selectable De-emphasis: -6dB > > >>>> Transmit Margin: Normal Operating Range, > > >>>> EnterModifiedCompliance- ComplianceSOS- > > >>>> Compliance De-emphasis: -6dB > > >>>> LnkSta2: Current De-emphasis Level: -6dB > > >>>> Capabilities: [e0] Vital Product Data > > >>>> Unknown small resource type 06, will not decode > > >>>> more. > > >>>> Kernel driver in use: ixgbe > > >>>> Kernel modules: ixgbe > > >>>> > > >> > > > ========================================================== > > >>>> =============================================== > > >>>> > > >>>> Rgds, > > >>>> Nishit Shah. > > >>>> > > >>>> ----------------------------------------------------------------- > > >>>> -- > > >>>> ----------- Introducing AppDynamics Lite, a free troubleshooting > > >>>> tool for Java/.NET > > >> Get > > >>>> 100% visibility into your production application - at no cost. > > >>>> Code-level diagnostics for performance bottlenecks with<2% > > >>>> overhead Download for free and get started troubleshooting in > minutes. > > >>>> http://p.sf.net/sfu/appdyn_d2d_ap1 > > >>>> _______________________________________________ > > >>>> E1000-devel mailing list > > >>>> [email protected] > > >>>> https://lists.sourceforge.net/lists/listinfo/e1000-devel > > >>>> To learn more about Intel® Ethernet, visit > > >>>> http://communities.intel.com/community/wired ------------------------------------------------------------------------------ Get 100% visibility into Java/.NET code with AppDynamics Lite It's a free troubleshooting tool designed for production Get down to code-level detail for bottlenecks, with <2% overhead. Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap2 _______________________________________________ E1000-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
