Based on the info you provided I would say one possible red flag would be the flow control bits in the statistics. Specifically: > tx_flow_control_xon: 0 > rx_flow_control_xon: 164 > tx_flow_control_xoff: 0 > rx_flow_control_xoff: 164 > rx_csum_offload_errors: 1
The fact that you are getting rx_flow_control messages would indicate that the 10Gb port is being stopped by the link partner. One thing you could try in order to test this further is to disable flow control on the 82599 port. To do that you can run the following where ethX is the name of the ixgbe interface you are currently using: ethtool -A ethX tx off rx off autoneg off Also you may want to set the 82599 and 82574 to the same interrupt rate. Currently it looks like the 82599 is being limited to 4000 interrupts per second while the 82574 is being allowed up to 20,000. If you let the 82599 use its' default throttle rate setting that should be comparable to the dynamic throttling provided by the e1000 driver and should improve TCP performance by reducing latency. Thanks, Alex On 08/12/2013 01:20 PM, Alexey Stoyanov wrote: > 03:00.0 Ethernet controller: Intel Corporation 82599EB 10-Gigabit > SFI/SFP+ Network Connection (rev 01) > Subsystem: Intel Corporation Ethernet Server Adapter X520-2 > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- > ParErr- Stepping- SERR- FastB2B- DisINTx+ > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- > <TAbort- <MAbort- >SERR- <PERR- INTx- > Latency: 0, Cache Line Size: 256 bytes > Interrupt: pin A routed to IRQ 30 > Region 0: Memory at f8f80000 (64-bit, prefetchable) [size=512K] > Region 2: I/O ports at cc00 [size=32] > Region 4: Memory at f8f7c000 (64-bit, prefetchable) [size=16K] > Capabilities: [40] Power Management version 3 > Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA > PME(D0+,D1-,D2-,D3hot+,D3cold-) > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME- > Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ > Address: 0000000000000000 Data: 0000 > Masking: 00000000 Pending: 00000000 > Capabilities: [70] MSI-X: Enable+ Count=64 Masked- > Vector table: BAR=4 offset=00000000 > PBA: BAR=4 offset=00002000 > Capabilities: [a0] Express (v2) Endpoint, MSI 00 > DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s > <512ns, L1 <64us > ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ > DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ > Unsupported+ > RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- > MaxPayload 256 bytes, MaxReadReq 512 bytes > DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ > AuxPwr- TransPend- > LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, > Latency L0 <1us, L1 <8us > ClockPM- Surprise- LLActRep- BwNot- > LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- > CommClk+ > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > LnkSta: Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ > DLActive- BWMgmt- ABWMgmt- > DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ > DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- > LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- > SpeedDis-, Selectable De-emphasis: -6dB > Transmit Margin: Normal Operating Range, > EnterModifiedCompliance- ComplianceSOS- > Compliance De-emphasis: -6dB > LnkSta2: Current De-emphasis Level: -6dB, > EqualizationComplete-, EqualizationPhase1- > EqualizationPhase2-, EqualizationPhase3-, > LinkEqualizationRequest- > Capabilities: [100 v1] Advanced Error Reporting > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- > UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- > NonFatalErr+ > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- > NonFatalErr+ > AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ > ChkEn- > Capabilities: [140 v1] Device Serial Number 90-e2-ba-ff-ff-40-89-24 > Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI) > ARICap: MFVC- ACS-, Next Function: 1 > ARICtl: MFVC- ACS-, Function Group: 0 > Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV) > IOVCap: Migration-, Interrupt Message Number: 000 > IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+ > IOVSta: Migration- > Initial VFs: 64, Total VFs: 64, Number of VFs: 0, > Function Dependency Link: 00 > VF offset: 128, stride: 2, Device ID: 10ed > Supported Page Size: 00000553, System Page Size: 00000001 > Region 0: Memory at 00000000c0000000 (64-bit, > non-prefetchable) > Region 3: Memory at 00000000c0100000 (64-bit, > non-prefetchable) > VF Migration: offset: 00000000, BIR: 0 > Kernel driver in use: ixgbe > > 06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network > Connection > Subsystem: Super Micro Computer Inc Device 10d3 > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- > ParErr- Stepping- SERR- FastB2B- DisINTx+ > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- > <TAbort- <MAbort- >SERR- <PERR- INTx- > Latency: 0, Cache Line Size: 64 bytes > Interrupt: pin A routed to IRQ 16 > Region 0: Memory at fbce0000 (32-bit, non-prefetchable) [size=128K] > Region 2: I/O ports at dc00 [size=32] > Region 3: Memory at fbcdc000 (32-bit, non-prefetchable) [size=16K] > Capabilities: [c8] Power Management version 2 > Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA > PME(D0+,D1-,D2-,D3hot+,D3cold+) > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME- > Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+ > Address: 0000000000000000 Data: 0000 > Capabilities: [e0] Express (v1) Endpoint, MSI 00 > DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s > <512ns, L1 <64us > ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- > DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ > Unsupported+ > RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ > MaxPayload 128 bytes, MaxReadReq 512 bytes > DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- > AuxPwr+ TransPend- > LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, > Latency L0 <128ns, L1 <64us > ClockPM- Surprise- LLActRep- BwNot- > LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- > CommClk+ > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- > SlotClk+ DLActive- BWMgmt- ABWMgmt- > Capabilities: [a0] MSI-X: Enable+ Count=5 Masked- > Vector table: BAR=3 offset=00000000 > PBA: BAR=3 offset=00002000 > Capabilities: [100 v1] Advanced Error Reporting > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- > UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- > UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- > NonFatalErr- > CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- > NonFatalErr+ > AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- > ChkEn- > Kernel driver in use: e1000e > > NIC statistics: > rx_packets: 41739495 > tx_packets: 95312581 > rx_bytes: 4352172478 > tx_bytes: 141738144174 > rx_errors: 0 > tx_errors: 0 > rx_dropped: 0 > tx_dropped: 0 > multicast: 197 > collisions: 0 > rx_over_errors: 0 > rx_crc_errors: 0 > rx_frame_errors: 0 > rx_fifo_errors: 0 > rx_missed_errors: 0 > tx_aborted_errors: 0 > tx_carrier_errors: 0 > tx_fifo_errors: 0 > tx_heartbeat_errors: 0 > rx_pkts_nic: 41739495 > tx_pkts_nic: 95312583 > rx_bytes_nic: 4519130458 > tx_bytes_nic: 142121155687 > lsc_int: 1 > tx_busy: 0 > non_eop_descs: 0 > broadcast: 8660 > rx_no_buffer_count: 0 > tx_timeout_count: 0 > tx_restart_queue: 0 > rx_long_length_errors: 0 > rx_short_length_errors: 0 > tx_flow_control_xon: 0 > rx_flow_control_xon: 164 > tx_flow_control_xoff: 0 > rx_flow_control_xoff: 164 > rx_csum_offload_errors: 1 > alloc_rx_page_failed: 0 > alloc_rx_buff_failed: 0 > rx_no_dma_resources: 0 > hw_rsc_aggregated: 0 > hw_rsc_flushed: 0 > fdir_match: 0 > fdir_miss: 0 > fdir_overflow: 0 > fcoe_bad_fccrc: 0 > fcoe_last_errors: 0 > rx_fcoe_dropped: 0 > rx_fcoe_packets: 0 > rx_fcoe_dwords: 0 > fcoe_noddp: 0 > fcoe_noddp_ext_buff: 0 > tx_fcoe_packets: 0 > tx_fcoe_dwords: 0 > os2bmc_rx_by_bmc: 0 > os2bmc_tx_by_bmc: 0 > os2bmc_tx_by_host: 0 > os2bmc_rx_by_host: 0 > tx_queue_0_packets: 11049929 > tx_queue_0_bytes: 16457030063 > tx_queue_1_packets: 12414532 > tx_queue_1_bytes: 18387691955 > tx_queue_2_packets: 12102892 > tx_queue_2_bytes: 17910217923 > tx_queue_3_packets: 8355089 > tx_queue_3_bytes: 12373121374 > tx_queue_4_packets: 13821793 > tx_queue_4_bytes: 20670342980 > tx_queue_5_packets: 13797111 > tx_queue_5_bytes: 20651106177 > tx_queue_6_packets: 11812819 > tx_queue_6_bytes: 17520772018 > tx_queue_7_packets: 11958418 > tx_queue_7_bytes: 17767864712 > tx_queue_8_packets: 0 > tx_queue_8_bytes: 0 [...] > tx_queue_70_packets: 0 > tx_queue_70_bytes: 0 > rx_queue_0_packets: 5621055 > rx_queue_0_bytes: 388097014 > rx_queue_1_packets: 4505359 > rx_queue_1_bytes: 584427618 > rx_queue_2_packets: 5075008 > rx_queue_2_bytes: 351564327 > rx_queue_3_packets: 5720433 > rx_queue_3_bytes: 408234947 > rx_queue_4_packets: 5848976 > rx_queue_4_bytes: 438647350 > rx_queue_5_packets: 4019439 > rx_queue_5_bytes: 294072858 > rx_queue_6_packets: 5422820 > rx_queue_6_bytes: 1501980910 > rx_queue_7_packets: 5526405 > rx_queue_7_bytes: 385147454 > rx_queue_8_packets: 0 > rx_queue_8_bytes: 0 [...] > rx_queue_70_packets: 0 > rx_queue_70_bytes: 0 > tx_pb_0_pxon: 0 > tx_pb_0_pxoff: 0 > tx_pb_1_pxon: 0 > tx_pb_1_pxoff: 0 > tx_pb_2_pxon: 0 > tx_pb_2_pxoff: 0 > tx_pb_3_pxon: 0 > tx_pb_3_pxoff: 0 > tx_pb_4_pxon: 0 > tx_pb_4_pxoff: 0 > tx_pb_5_pxon: 0 > tx_pb_5_pxoff: 0 > tx_pb_6_pxon: 0 > tx_pb_6_pxoff: 0 > tx_pb_7_pxon: 0 > tx_pb_7_pxoff: 0 > rx_pb_0_pxon: 0 > rx_pb_0_pxoff: 0 > rx_pb_1_pxon: 0 > rx_pb_1_pxoff: 0 > rx_pb_2_pxon: 0 > rx_pb_2_pxoff: 0 > rx_pb_3_pxon: 0 > rx_pb_3_pxoff: 0 > rx_pb_4_pxon: 0 > rx_pb_4_pxoff: 0 > rx_pb_5_pxon: 0 > rx_pb_5_pxoff: 0 > rx_pb_6_pxon: 0 > rx_pb_6_pxoff: 0 > rx_pb_7_pxon: 0 > rx_pb_7_pxoff: 0 > > > for 82574 > NIC statistics: > rx_packets: 1756823 > tx_packets: 11083890 > rx_bytes: 117911917 > tx_bytes: 16765393107 > rx_broadcast: 36413 > tx_broadcast: 190 > rx_multicast: 842 > tx_multicast: 6 > rx_errors: 0 > tx_errors: 0 > tx_dropped: 0 > multicast: 842 > collisions: 0 > rx_length_errors: 0 > rx_over_errors: 0 > rx_crc_errors: 0 > rx_frame_errors: 0 > rx_no_buffer_count: 0 > rx_missed_errors: 0 > tx_aborted_errors: 0 > tx_carrier_errors: 0 > tx_fifo_errors: 0 > tx_heartbeat_errors: 0 > tx_window_errors: 0 > tx_abort_late_coll: 0 > tx_deferred_ok: 0 > tx_single_coll_ok: 0 > tx_multi_coll_ok: 0 > tx_timeout_count: 0 > tx_restart_queue: 0 > rx_long_length_errors: 0 > rx_short_length_errors: 0 > rx_align_errors: 0 > tx_tcp_seg_good: 261045 > tx_tcp_seg_failed: 0 > rx_flow_control_xon: 0 > rx_flow_control_xoff: 0 > tx_flow_control_xon: 0 > tx_flow_control_xoff: 0 > rx_long_byte_count: 117911917 > rx_csum_offload_good: 1727173 > rx_csum_offload_errors: 0 > rx_header_split: 0 > alloc_rx_buff_failed: 0 > tx_smbus: 0 > rx_smbus: 0 > dropped_smbus: 0 > rx_dma_failed: 0 > tx_dma_failed: 0 > uncorr_ecc_errors: 0 > corr_ecc_errors: 0 > > dmesg > [21484.607162] ixgbe: Interrupt Mode set to 2 > [21484.607165] ixgbe: Multiple Queue Support Enabled > [21484.607167] ixgbe: Direct Cache Access (DCA) set to 0 > [21484.607169] ixgbe: 0000:03:00.0: ixgbe_check_options: DCA is disabled > [21484.607170] ixgbe: Receive-Side Scaling (RSS) set to 8 > [21484.607172] ixgbe: Virtual Machine Device Queues (VMDQ) set to 0 > [21484.607174] ixgbe: I/O Virtualization (IOV) set to 0 > [21484.607175] ixgbe: L2 Loopback Enable set to 0 > [21484.607177] ixgbe: Interrupt Throttling Rate (ints/sec) set to 4000 > [21484.607179] ixgbe: Low Latency Interrupt TCP Port set to 5001 > [21484.607180] ixgbe: Low Latency Interrupt on Packet Size set to 1500 > [21484.607182] ixgbe: Low Latency Interrupt on TCP Push flag Enabled > [21484.607184] ixgbe: 0000:03:00.0: ixgbe_check_options: FCoE Offload > feature enabled > [21484.764194] ixgbe 0000:03:00.0: irq 77 for MSI/MSI-X > [21484.764206] ixgbe 0000:03:00.0: irq 78 for MSI/MSI-X > [21484.764217] ixgbe 0000:03:00.0: irq 79 for MSI/MSI-X > [21484.764227] ixgbe 0000:03:00.0: irq 80 for MSI/MSI-X > [21484.764237] ixgbe 0000:03:00.0: irq 81 for MSI/MSI-X > [21484.766332] ixgbe 0000:03:00.0: (PCI Express:5.0GT/s:Width x8) > 90:e2:ba:40:89:24 > [21484.766420] ixgbe 0000:03:00.0 eth1: MAC: 2, PHY: 15, SFP+: 5, PBA > No: E68793-006 > [21484.766424] ixgbe 0000:03:00.0 eth1: Enabled Features: RxQ: 8 TxQ: 8 > [21484.766459] ixgbe 0000:03:00.0 eth1: Intel(R) 10 Gigabit Network Connection > [21484.766716] ixgbe: Interrupt Mode set to 2 > [21484.766719] ixgbe: Multiple Queue Support Enabled > [21484.766721] ixgbe: Direct Cache Access (DCA) set to 0 > [21484.766723] ixgbe: 0000:03:00.1: ixgbe_check_options: DCA is disabled > [21484.766724] ixgbe: Receive-Side Scaling (RSS) set to 8 > [21484.766726] ixgbe: Virtual Machine Device Queues (VMDQ) set to 0 > [21484.766728] ixgbe: I/O Virtualization (IOV) set to 0 > [21484.766729] ixgbe: L2 Loopback Enable set to 0 > [21484.766731] ixgbe: Interrupt Throttling Rate (ints/sec) set to 4000 > [21484.766733] ixgbe: Low Latency Interrupt TCP Port set to 5001 > [21484.766735] ixgbe: Low Latency Interrupt on Packet Size set to 1500 > [21484.766736] ixgbe: Low Latency Interrupt on TCP Push flag Enabled > [21484.766738] ixgbe: 0000:03:00.1: ixgbe_check_options: FCoE Offload > feature enabled > [21485.002704] ixgbe 0000:03:00.0 eth1: detected SFP+: 5 > [21485.254336] ixgbe 0000:03:00.0 eth1: NIC Link is Up 10 Gbps, Flow > Control: RX/TX > [21485.907967] ixgbe 0000:03:00.1: irq 82 for MSI/MSI-X > [21485.907979] ixgbe 0000:03:00.1: irq 83 for MSI/MSI-X > [21485.907986] ixgbe 0000:03:00.1: irq 84 for MSI/MSI-X > [21485.907993] ixgbe 0000:03:00.1: irq 85 for MSI/MSI-X > [21485.908000] ixgbe 0000:03:00.1: irq 86 for MSI/MSI-X > [21485.910139] ixgbe 0000:03:00.1: (PCI Express:5.0GT/s:Width x8) > 90:e2:ba:40:89:25 > [21485.910226] ixgbe 0000:03:00.1 eth3: MAC: 2, PHY: 1, PBA No: E68793-006 > [21485.910230] ixgbe 0000:03:00.1 eth3: Enabled Features: RxQ: 8 TxQ: 8 > [21485.910268] ixgbe 0000:03:00.1 eth3: Intel(R) 10 Gigabit Network Connection > ------------------------------------------------------------------------------ Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with <2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk _______________________________________________ E1000-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
