On Tue, 2018-08-21 at 11:13 -0700, Alexander Duyck wrote: > On Tue, Aug 21, 2018 at 9:59 AM Nikita V. Shirokov < > tehn...@tehnerd.com> wrote: > > > > On Tue, Aug 21, 2018 at 08:58:15AM -0700, Alexander Duyck wrote: > > > On Mon, Aug 20, 2018 at 12:32 PM Nikita V. Shirokov < > > > tehn...@tehnerd.com> wrote: > > > > > > > > we are getting such errors: > > > > > > > > [ 408.737313] ixgbe 0000:03:00.0 eth0: Detected Tx Unit Hang > > > > (XDP) > > > > Tx Queue <46> > > > > TDH, TDT <0>, <2> > > > > next_to_use <2> > > > > next_to_clean <0> > > > > tx_buffer_info[next_to_clean] > > > > time_stamp <0> > > > > jiffies <1000197c0> > > > > [ 408.804438] ixgbe 0000:03:00.0 eth0: tx hang 1 detected on > > > > queue 46, resetting adapter > > > > [ 408.804440] ixgbe 0000:03:00.0 eth0: initiating reset due to > > > > tx timeout > > > > [ 408.817679] ixgbe 0000:03:00.0 eth0: Reset adapter > > > > [ 408.866091] ixgbe 0000:03:00.0 eth0: TXDCTL.ENABLE for one > > > > or more queues not cleared within the polling period > > > > [ 409.345289] ixgbe 0000:03:00.0 eth0: detected SFP+: 3 > > > > [ 409.497232] ixgbe 0000:03:00.0 eth0: NIC Link is Up 10 Gbps, > > > > Flow Control: RX/TX > > > > > > > > while running XDP prog on ixgbe nic. > > > > right now i'm seing this on bpfnext kernel > > > > (latest commit from Wed Aug 15 15:04:25 2018 -0700 ; > > > > 9a76aba02a37718242d7cdc294f0a3901928aa57) > > > > > > > > looks like this is the same issue as reported by Brenden in > > > > https://www.spinics.net/lists/netdev/msg439438.html > > > > > > > > -- > > > > Nikita V. Shirokov > > > > > > Could you provide some additional information about your setup. > > > Specifically useful would be "ethtool -i", "ethtool -l", and > > > lspci > > > -vvv info for your device. The total number of CPUs on the system > > > would be useful to know as well. In addition could you try > > > reproducing > > > > sure: > > > > ethtool -l eth0 > > Channel parameters for eth0: > > Pre-set maximums: > > RX: 0 > > TX: 0 > > Other: 1 > > Combined: 63 > > Current hardware settings: > > RX: 0 > > TX: 0 > > Other: 1 > > Combined: 48 > > > > # ethtool -i eth0 > > driver: ixgbe > > version: 5.1.0-k > > firmware-version: 0x800006f1 > > expansion-rom-version: > > bus-info: 0000:03:00.0 > > supports-statistics: yes > > supports-test: yes > > supports-eeprom-access: yes > > supports-register-dump: yes > > supports-priv-flags: yes > > > > > > # nproc > > 48 > > > > lspci: > > > > 03:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit > > SFI/SFP+ Network Connection (rev 01) > > Subsystem: Intel Corporation Device 000d > > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- > > VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ > > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast > > >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- > > Latency: 0, Cache Line Size: 32 bytes > > Interrupt: pin A routed to IRQ 30 > > NUMA node: 0 > > Region 0: Memory at c7d00000 (64-bit, non-prefetchable) > > [size=1M] > > Region 2: I/O ports at 6000 [size=32] > > Region 4: Memory at c7e80000 (64-bit, non-prefetchable) > > [size=16K] > > Expansion ROM at c7e00000 [disabled] [size=512K] > > Capabilities: [40] Power Management version 3 > > Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA > > PME(D0+,D1-,D2-,D3hot+,D3cold+) > > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 > > PME- > > Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ > > Address: 0000000000000000 Data: 0000 > > Masking: 00000000 Pending: 00000000 > > Capabilities: [70] MSI-X: Enable+ Count=64 Masked- > > Vector table: BAR=4 offset=00000000 > > PBA: BAR=4 offset=00002000 > > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > > DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency > > L0s <512ns, L1 <64us > > ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ > > FLReset+ SlotPowerLimit 0.000W > > DevCtl: Report errors: Correctable+ Non-Fatal+ > > Fatal+ Unsupported+ > > RlxdOrd- ExtTag- PhantFunc- AuxPwr- > > NoSnoop+ FLReset- > > MaxPayload 256 bytes, MaxReadReq 512 bytes > > DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ > > AuxPwr+ TransPend+ > > LnkCap: Port #2, Speed 5GT/s, Width x8, ASPM L0s, > > Exit Latency L0s unlimited, L1 <8us > > ClockPM- Surprise- LLActRep- BwNot- > > ASPMOptComp- > > LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- > > CommClk+ > > ExtSynch- ClockPM- AutWidDis- BWInt- > > AutBWInt- > > LnkSta: Speed 5GT/s, Width x8, TrErr- Train- > > SlotClk+ DLActive- BWMgmt- ABWMgmt- > > DevCap2: Completion Timeout: Range ABCD, > > TimeoutDis+, LTR-, OBFF Not Supported > > DevCtl2: Completion Timeout: 50us to 50ms, > > TimeoutDis-, LTR-, OBFF Disabled > > LnkCtl2: Target Link Speed: 5GT/s, > > EnterCompliance- SpeedDis- > > Transmit Margin: Normal Operating Range, > > EnterModifiedCompliance- ComplianceSOS- > > Compliance De-emphasis: -6dB > > LnkSta2: Current De-emphasis Level: -6dB, > > EqualizationComplete-, EqualizationPhase1- > > EqualizationPhase2-, EqualizationPhase3-, > > LinkEqualizationRequest- > > Capabilities: [100 v1] Advanced Error Reporting > > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- > > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- > > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > > UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- > > UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- > > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- > > NonFatalErr+ > > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- > > NonFatalErr+ > > AERCap: First Error Pointer: 00, GenCap+ CGenEn- > > ChkCap+ ChkEn- > > Capabilities: [140 v1] Device Serial Number 90-e2-ba-ff- > > ff-b6-b2-60 > > Capabilities: [150 v1] Alternative Routing-ID > > Interpretation (ARI) > > ARICap: MFVC- ACS-, Next Function: 0 > > ARICtl: MFVC- ACS-, Function Group: 0 > > Capabilities: [160 v1] Single Root I/O Virtualization (SR- > > IOV) > > IOVCap: Migration-, Interrupt Message Number: 000 > > IOVCtl: Enable- Migration- Interrupt- MSE- > > ARIHierarchy+ > > IOVSta: Migration- > > Initial VFs: 64, Total VFs: 64, Number of VFs: 0, > > Function Dependency Link: 00 > > VF offset: 128, stride: 2, Device ID: 10ed > > Supported Page Size: 00000553, System Page Size: > > 00000001 > > Region 0: Memory at 00000000c7c00000 (64-bit, > > prefetchable) > > Region 3: Memory at 00000000c7b00000 (64-bit, > > prefetchable) > > VF Migration: offset: 00000000, BIR: 0 > > Kernel driver in use: ixgbe > > > > > > > > > > workaround for now is to do the same, as Brenden did in his > > original > > finding: make sure that combined + xdp queues < max_tx_queues > > (e.g. w/ combined == 14 the issue goes away). > > > > > the issue with one of the sample XDP programs provided with the > > > kernel > > > such as the xdp2 which I believe uses the XDP_TX function. We > > > need to > > > try and create a similar setup in our own environment for > > > reproduction and debugging. > > > > will try but this could take a while, because i'm not sure that we > > have > > ixgbe in our test lab (and it would be hard to run such test in > > prod) > > > > > > > > Thanks. > > > > > > - Alex > > > > -- > > Nikita V. Shirokov > > So I have been reading the datasheet > ( > https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/82599-10-gbe-controller-datasheet.pdf > ) > and it looks like the assumption that Brenden came to in the earlier > referenced link is probably correct. From what I can tell there is a > limit of 64 queues in the base RSS mode of the device, so while it > supports more than 64 queues you can only make use of 64 as per table > 7-25. > > For now I think the workaround you are using is probably the only > viable solution. I myself don't have time to work on resolving this, > but I am sure on of the maintainers for ixgbe will be responding > shortly.
I have notified the 10GbE maintainers, and we are working to reproduce the issue currently. > > One possible solution we may want to look at would be to make use of > the 32 pool/VF mode in the MTQC register. That should enable us to > make use of all 128 queues but I am sure there would be other side > effects such as having to set the bits in the PFVFTE register in > order > to enable the extra Tx queues.
signature.asc
Description: This is a digitally signed message part
