One other thing that separates the 82574 and the 82599 is that 82599 is a multiqueue interface. Try loading the driver with RSS=1,1 to see if this issue might somehow be related to multiqueue.
Other than that the only other thing I can think of would be to start rate limiting the ixgbe port itself as the only other possibility I can think of that might be hurting the performance is the fact that the 82599 can produce traffic so much faster than the 82574 and this may be leading to packets being dropped somewhere. Thanks, Alex On 08/12/2013 02:50 PM, Alexey Stoyanov wrote: > One important thing that i not writed from start - this is real > internet, so this is not a LAN, but WAN > I have average 27 ms latency beetween hosts. > > --- yy.yy.74.11 ping statistics --- > 10 packets transmitted, 10 received, 0% packet loss, time 9012ms > rtt min/avg/max/mdev = 27.203/27.444/27.791/0.230 ms > > > i've reloaded ixgbe with InterruptThrottleRate=1,1 (default) > [31906.918256] ixgbe: Interrupt Mode set to 2 > [31906.918259] ixgbe: Multiple Queue Support Enabled > [31906.918264] ixgbe: Direct Cache Access (DCA) set to 0 > [31906.918271] ixgbe: 0000:03:00.0: ixgbe_check_options: DCA is disabled > [31906.918274] ixgbe: Receive-Side Scaling (RSS) set to 8 > [31906.918276] ixgbe: Virtual Machine Device Queues (VMDQ) set to 0 > [31906.918278] ixgbe: I/O Virtualization (IOV) set to 0 > [31906.918279] ixgbe: L2 Loopback Enable set to 0 > [31906.918281] ixgbe: 0000:03:00.0: ixgbe_check_options: dynamic > interrupt throttling enabled > [31906.918283] ixgbe: Low Latency Interrupt TCP Port set to 5001 > [31906.918285] ixgbe: Low Latency Interrupt on Packet Size set to 1500 > [31906.918287] ixgbe: Low Latency Interrupt on TCP Push flag Enabled > [31906.918289] ixgbe: 0000:03:00.0: ixgbe_check_options: FCoE Offload > feature enabled > [31907.083032] ixgbe 0000:03:00.0: irq 77 for MSI/MSI-X > [31907.083045] ixgbe 0000:03:00.0: irq 78 for MSI/MSI-X > [31907.083055] ixgbe 0000:03:00.0: irq 79 for MSI/MSI-X > [31907.083065] ixgbe 0000:03:00.0: irq 80 for MSI/MSI-X > [31907.083074] ixgbe 0000:03:00.0: irq 81 for MSI/MSI-X > [31907.085174] ixgbe 0000:03:00.0: (PCI Express:5.0GT/s:Width x8) > 90:e2:ba:40:89:24 > [31907.085262] ixgbe 0000:03:00.0 eth1: MAC: 2, PHY: 15, SFP+: 5, PBA > No: E68793-006 > [31907.085266] ixgbe 0000:03:00.0 eth1: Enabled Features: RxQ: 8 TxQ: 8 > [31907.085301] ixgbe 0000:03:00.0 eth1: Intel(R) 10 Gigabit Network Connection > [31907.085531] ixgbe: Interrupt Mode set to 2 > [31907.085534] ixgbe: Multiple Queue Support Enabled > [31907.085536] ixgbe: Direct Cache Access (DCA) set to 0 > [31907.085538] ixgbe: 0000:03:00.1: ixgbe_check_options: DCA is disabled > [31907.085540] ixgbe: Receive-Side Scaling (RSS) set to 8 > [31907.085542] ixgbe: Virtual Machine Device Queues (VMDQ) set to 0 > [31907.085544] ixgbe: I/O Virtualization (IOV) set to 0 > [31907.085545] ixgbe: L2 Loopback Enable set to 0 > [31907.085547] ixgbe: 0000:03:00.1: ixgbe_check_options: dynamic > interrupt throttling enabled > [31907.085549] ixgbe: Low Latency Interrupt TCP Port set to 5001 > [31907.085550] ixgbe: Low Latency Interrupt on Packet Size set to 1500 > [31907.085552] ixgbe: Low Latency Interrupt on TCP Push flag Enabled > [31907.085554] ixgbe: 0000:03:00.1: ixgbe_check_options: FCoE Offload > feature enabled > [31907.325658] ixgbe 0000:03:00.0 eth1: detected SFP+: 5 > [31907.784996] ixgbe 0000:03:00.0 eth1: NIC Link is Up 10 Gbps, Flow > Control: RX/TX > [31908.228991] ixgbe 0000:03:00.1: irq 82 for MSI/MSI-X > [31908.229002] ixgbe 0000:03:00.1: irq 83 for MSI/MSI-X > [31908.229007] ixgbe 0000:03:00.1: irq 84 for MSI/MSI-X > [31908.229012] ixgbe 0000:03:00.1: irq 85 for MSI/MSI-X > [31908.229016] ixgbe 0000:03:00.1: irq 86 for MSI/MSI-X > [31908.230732] ixgbe 0000:03:00.1: (PCI Express:5.0GT/s:Width x8) > 90:e2:ba:40:89:25 > [31908.230818] ixgbe 0000:03:00.1 eth3: MAC: 2, PHY: 1, PBA No: E68793-006 > [31908.230820] ixgbe 0000:03:00.1 eth3: Enabled Features: RxQ: 8 TxQ: 8 > [31908.230893] ixgbe 0000:03:00.1 eth3: Intel(R) 10 Gigabit Network Connection > [31913.504204] ixgbe 0000:03:00.0 eth1: detected SFP+: 5 > [31913.755868] ixgbe 0000:03:00.0 eth1: NIC Link is Up 10 Gbps, Flow > Control: None > > last line is when i executed disabling flow control > i disabled it both on switch and on card (actually i has disabled it > all long time ago, just tried today to turn it on, before i run out of > ideas how to resolve this issue) > > after i tried to do tests again, and got same (very close) result > > Optical 82599 > ------------------------------------------------------------ > Client connecting to yy.yy.74.11, TCP port 5001 > Binding to local address xxx.xxx.185.135 > TCP window size: 64.0 KByte (default) > ------------------------------------------------------------ > [ 3] local xxx.xxx.185.135 port 5001 connected with yy.yy.74.11 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-20.0 sec 207 MBytes 86.7 Mbits/sec > > Cooper 82574 > ------------------------------------------------------------ > Client connecting to yy.yy.74.11, TCP port 5001 > Binding to local address xxx.xxx.185.133 > TCP window size: 64.0 KByte (default) > ------------------------------------------------------------ > [ 3] local xxx.xxx.185.133 port 5001 connected with yy.yy.74.11 port 5001 > [ ID] Interval Transfer Bandwidth > [ 3] 0.0-20.1 sec 645 MBytes 270 Mbits/sec > > Again i got 82574 at least 3 times faster. > > This is ethtool -S eth1 report again after reload of driver. > NIC statistics: > rx_packets: 3474041 > tx_packets: 6528442 > rx_bytes: 450307293 > tx_bytes: 9525085597 > rx_errors: 0 > tx_errors: 0 > rx_dropped: 0 > tx_dropped: 0 > multicast: 22 > collisions: 0 > rx_over_errors: 0 > rx_crc_errors: 0 > rx_frame_errors: 0 > rx_fifo_errors: 0 > rx_missed_errors: 0 > tx_aborted_errors: 0 > tx_carrier_errors: 0 > tx_fifo_errors: 0 > tx_heartbeat_errors: 0 > rx_pkts_nic: 3472437 > tx_pkts_nic: 6525653 > rx_bytes_nic: 463886577 > tx_bytes_nic: 9547445047 > lsc_int: 3 > tx_busy: 0 > non_eop_descs: 0 > broadcast: 730 > rx_no_buffer_count: 0 > tx_timeout_count: 0 > tx_restart_queue: 0 > rx_long_length_errors: 0 > rx_short_length_errors: 0 > tx_flow_control_xon: 0 > rx_flow_control_xon: 0 > tx_flow_control_xoff: 0 > rx_flow_control_xoff: 0 > rx_csum_offload_errors: 0 > alloc_rx_page_failed: 0 > alloc_rx_buff_failed: 0 > rx_no_dma_resources: 0 > hw_rsc_aggregated: 0 > hw_rsc_flushed: 0 > fdir_match: 0 > fdir_miss: 0 > fdir_overflow: 0 > fcoe_bad_fccrc: 0 > fcoe_last_errors: 0 > rx_fcoe_dropped: 0 > rx_fcoe_packets: 0 > rx_fcoe_dwords: 0 > fcoe_noddp: 0 > fcoe_noddp_ext_buff: 0 > tx_fcoe_packets: 0 > tx_fcoe_dwords: 0 > os2bmc_rx_by_bmc: 0 > os2bmc_tx_by_bmc: 0 > os2bmc_tx_by_host: 0 > os2bmc_rx_by_host: 0 > tx_queue_0_packets: 752433 > tx_queue_0_bytes: 1106159204 > tx_queue_1_packets: 774829 > tx_queue_1_bytes: 1122892047 > tx_queue_2_packets: 837308 > tx_queue_2_bytes: 1224541683 > tx_queue_3_packets: 743260 > tx_queue_3_bytes: 1071268064 > tx_queue_4_packets: 775167 > tx_queue_4_bytes: 1137021910 > tx_queue_5_packets: 866713 > tx_queue_5_bytes: 1271941588 > tx_queue_6_packets: 1041911 > tx_queue_6_bytes: 1537638579 > tx_queue_7_packets: 736821 > tx_queue_7_bytes: 1053622522 > tx_queue_8_packets: 0 > tx_queue_8_bytes: 0 > tx_queue_9_packets: 0 > tx_queue_9_bytes: 0 > tx_queue_10_packets: 0 > tx_queue_10_bytes: 0 > tx_queue_11_packets: 0 > tx_queue_11_bytes: 0 > tx_queue_12_packets: 0 > tx_queue_12_bytes: 0 > tx_queue_13_packets: 0 > tx_queue_13_bytes: 0 > tx_queue_14_packets: 0 > tx_queue_14_bytes: 0 > tx_queue_15_packets: 0 > tx_queue_15_bytes: 0 > tx_queue_16_packets: 0 > tx_queue_16_bytes: 0 > tx_queue_17_packets: 0 > tx_queue_17_bytes: 0 > tx_queue_18_packets: 0 > tx_queue_18_bytes: 0 > tx_queue_19_packets: 0 > tx_queue_19_bytes: 0 > tx_queue_20_packets: 0 > tx_queue_20_bytes: 0 > tx_queue_21_packets: 0 > tx_queue_21_bytes: 0 > tx_queue_22_packets: 0 > tx_queue_22_bytes: 0 > tx_queue_23_packets: 0 > tx_queue_23_bytes: 0 > tx_queue_24_packets: 0 > tx_queue_24_bytes: 0 > tx_queue_25_packets: 0 > tx_queue_25_bytes: 0 > tx_queue_26_packets: 0 > tx_queue_26_bytes: 0 > tx_queue_27_packets: 0 > tx_queue_27_bytes: 0 > tx_queue_28_packets: 0 > tx_queue_28_bytes: 0 > tx_queue_29_packets: 0 > tx_queue_29_bytes: 0 > tx_queue_30_packets: 0 > tx_queue_30_bytes: 0 > tx_queue_31_packets: 0 > tx_queue_31_bytes: 0 > tx_queue_32_packets: 0 > tx_queue_32_bytes: 0 > tx_queue_33_packets: 0 > tx_queue_33_bytes: 0 > tx_queue_34_packets: 0 > tx_queue_34_bytes: 0 > tx_queue_35_packets: 0 > tx_queue_35_bytes: 0 > tx_queue_36_packets: 0 > tx_queue_36_bytes: 0 > tx_queue_37_packets: 0 > tx_queue_37_bytes: 0 > tx_queue_38_packets: 0 > tx_queue_38_bytes: 0 > tx_queue_39_packets: 0 > tx_queue_39_bytes: 0 > tx_queue_40_packets: 0 > tx_queue_40_bytes: 0 > tx_queue_41_packets: 0 > tx_queue_41_bytes: 0 > tx_queue_42_packets: 0 > tx_queue_42_bytes: 0 > tx_queue_43_packets: 0 > tx_queue_43_bytes: 0 > tx_queue_44_packets: 0 > tx_queue_44_bytes: 0 > tx_queue_45_packets: 0 > tx_queue_45_bytes: 0 > tx_queue_46_packets: 0 > tx_queue_46_bytes: 0 > tx_queue_47_packets: 0 > tx_queue_47_bytes: 0 > tx_queue_48_packets: 0 > tx_queue_48_bytes: 0 > tx_queue_49_packets: 0 > tx_queue_49_bytes: 0 > tx_queue_50_packets: 0 > tx_queue_50_bytes: 0 > tx_queue_51_packets: 0 > tx_queue_51_bytes: 0 > tx_queue_52_packets: 0 > tx_queue_52_bytes: 0 > tx_queue_53_packets: 0 > tx_queue_53_bytes: 0 > tx_queue_54_packets: 0 > tx_queue_54_bytes: 0 > tx_queue_55_packets: 0 > tx_queue_55_bytes: 0 > tx_queue_56_packets: 0 > tx_queue_56_bytes: 0 > tx_queue_57_packets: 0 > tx_queue_57_bytes: 0 > tx_queue_58_packets: 0 > tx_queue_58_bytes: 0 > tx_queue_59_packets: 0 > tx_queue_59_bytes: 0 > tx_queue_60_packets: 0 > tx_queue_60_bytes: 0 > tx_queue_61_packets: 0 > tx_queue_61_bytes: 0 > tx_queue_62_packets: 0 > tx_queue_62_bytes: 0 > tx_queue_63_packets: 0 > tx_queue_63_bytes: 0 > tx_queue_64_packets: 0 > tx_queue_64_bytes: 0 > tx_queue_65_packets: 0 > tx_queue_65_bytes: 0 > tx_queue_66_packets: 0 > tx_queue_66_bytes: 0 > tx_queue_67_packets: 0 > tx_queue_67_bytes: 0 > tx_queue_68_packets: 0 > tx_queue_68_bytes: 0 > tx_queue_69_packets: 0 > tx_queue_69_bytes: 0 > tx_queue_70_packets: 0 > tx_queue_70_bytes: 0 > rx_queue_0_packets: 463967 > rx_queue_0_bytes: 33270955 > rx_queue_1_packets: 346811 > rx_queue_1_bytes: 70602531 > rx_queue_2_packets: 445265 > rx_queue_2_bytes: 61583232 > rx_queue_3_packets: 426811 > rx_queue_3_bytes: 33946327 > rx_queue_4_packets: 473199 > rx_queue_4_bytes: 35975233 > rx_queue_5_packets: 316566 > rx_queue_5_bytes: 32463326 > rx_queue_6_packets: 529871 > rx_queue_6_bytes: 140641116 > rx_queue_7_packets: 471551 > rx_queue_7_bytes: 41824573 > rx_queue_8_packets: 0 > rx_queue_8_bytes: 0 > rx_queue_9_packets: 0 > rx_queue_9_bytes: 0 > rx_queue_10_packets: 0 > rx_queue_10_bytes: 0 > rx_queue_11_packets: 0 > rx_queue_11_bytes: 0 > rx_queue_12_packets: 0 > rx_queue_12_bytes: 0 > rx_queue_13_packets: 0 > rx_queue_13_bytes: 0 > rx_queue_14_packets: 0 > rx_queue_14_bytes: 0 > rx_queue_15_packets: 0 > rx_queue_15_bytes: 0 > rx_queue_16_packets: 0 > rx_queue_16_bytes: 0 > rx_queue_17_packets: 0 > rx_queue_17_bytes: 0 > rx_queue_18_packets: 0 > rx_queue_18_bytes: 0 > rx_queue_19_packets: 0 > rx_queue_19_bytes: 0 > rx_queue_20_packets: 0 > rx_queue_20_bytes: 0 > rx_queue_21_packets: 0 > rx_queue_21_bytes: 0 > rx_queue_22_packets: 0 > rx_queue_22_bytes: 0 > rx_queue_23_packets: 0 > rx_queue_23_bytes: 0 > rx_queue_24_packets: 0 > rx_queue_24_bytes: 0 > rx_queue_25_packets: 0 > rx_queue_25_bytes: 0 > rx_queue_26_packets: 0 > rx_queue_26_bytes: 0 > rx_queue_27_packets: 0 > rx_queue_27_bytes: 0 > rx_queue_28_packets: 0 > rx_queue_28_bytes: 0 > rx_queue_29_packets: 0 > rx_queue_29_bytes: 0 > rx_queue_30_packets: 0 > rx_queue_30_bytes: 0 > rx_queue_31_packets: 0 > rx_queue_31_bytes: 0 > rx_queue_32_packets: 0 > rx_queue_32_bytes: 0 > rx_queue_33_packets: 0 > rx_queue_33_bytes: 0 > rx_queue_34_packets: 0 > rx_queue_34_bytes: 0 > rx_queue_35_packets: 0 > rx_queue_35_bytes: 0 > rx_queue_36_packets: 0 > rx_queue_36_bytes: 0 > rx_queue_37_packets: 0 > rx_queue_37_bytes: 0 > rx_queue_38_packets: 0 > rx_queue_38_bytes: 0 > rx_queue_39_packets: 0 > rx_queue_39_bytes: 0 > rx_queue_40_packets: 0 > rx_queue_40_bytes: 0 > rx_queue_41_packets: 0 > rx_queue_41_bytes: 0 > rx_queue_42_packets: 0 > rx_queue_42_bytes: 0 > rx_queue_43_packets: 0 > rx_queue_43_bytes: 0 > rx_queue_44_packets: 0 > rx_queue_44_bytes: 0 > rx_queue_45_packets: 0 > rx_queue_45_bytes: 0 > rx_queue_46_packets: 0 > rx_queue_46_bytes: 0 > rx_queue_47_packets: 0 > rx_queue_47_bytes: 0 > rx_queue_48_packets: 0 > rx_queue_48_bytes: 0 > rx_queue_49_packets: 0 > rx_queue_49_bytes: 0 > rx_queue_50_packets: 0 > rx_queue_50_bytes: 0 > rx_queue_51_packets: 0 > rx_queue_51_bytes: 0 > rx_queue_52_packets: 0 > rx_queue_52_bytes: 0 > rx_queue_53_packets: 0 > rx_queue_53_bytes: 0 > rx_queue_54_packets: 0 > rx_queue_54_bytes: 0 > rx_queue_55_packets: 0 > rx_queue_55_bytes: 0 > rx_queue_56_packets: 0 > rx_queue_56_bytes: 0 > rx_queue_57_packets: 0 > rx_queue_57_bytes: 0 > rx_queue_58_packets: 0 > rx_queue_58_bytes: 0 > rx_queue_59_packets: 0 > rx_queue_59_bytes: 0 > rx_queue_60_packets: 0 > rx_queue_60_bytes: 0 > rx_queue_61_packets: 0 > rx_queue_61_bytes: 0 > rx_queue_62_packets: 0 > rx_queue_62_bytes: 0 > rx_queue_63_packets: 0 > rx_queue_63_bytes: 0 > rx_queue_64_packets: 0 > rx_queue_64_bytes: 0 > rx_queue_65_packets: 0 > rx_queue_65_bytes: 0 > rx_queue_66_packets: 0 > rx_queue_66_bytes: 0 > rx_queue_67_packets: 0 > rx_queue_67_bytes: 0 > rx_queue_68_packets: 0 > rx_queue_68_bytes: 0 > rx_queue_69_packets: 0 > rx_queue_69_bytes: 0 > rx_queue_70_packets: 0 > rx_queue_70_bytes: 0 > tx_pb_0_pxon: 0 > tx_pb_0_pxoff: 0 > tx_pb_1_pxon: 0 > tx_pb_1_pxoff: 0 > tx_pb_2_pxon: 0 > tx_pb_2_pxoff: 0 > tx_pb_3_pxon: 0 > tx_pb_3_pxoff: 0 > tx_pb_4_pxon: 0 > tx_pb_4_pxoff: 0 > tx_pb_5_pxon: 0 > tx_pb_5_pxoff: 0 > tx_pb_6_pxon: 0 > tx_pb_6_pxoff: 0 > tx_pb_7_pxon: 0 > tx_pb_7_pxoff: 0 > rx_pb_0_pxon: 0 > rx_pb_0_pxoff: 0 > rx_pb_1_pxon: 0 > rx_pb_0_pxoff: 0 > rx_pb_1_pxon: 0 > rx_pb_1_pxoff: 0 > rx_pb_2_pxon: 0 > rx_pb_2_pxoff: 0 > rx_pb_3_pxon: 0 > rx_pb_3_pxoff: 0 > rx_pb_4_pxon: 0 > rx_pb_4_pxoff: 0 > rx_pb_5_pxon: 0 > rx_pb_5_pxoff: 0 > rx_pb_6_pxon: 0 > rx_pb_6_pxoff: 0 > rx_pb_7_pxon: 0 > rx_pb_7_pxoff: 0 > > > 2013/8/13 Alexander Duyck <[email protected]>: >> Based on the info you provided I would say one possible red flag would >> be the flow control bits in the statistics. Specifically: >>> tx_flow_control_xon: 0 >>> rx_flow_control_xon: 164 >>> tx_flow_control_xoff: 0 >>> rx_flow_control_xoff: 164 >>> rx_csum_offload_errors: 1 >> The fact that you are getting rx_flow_control messages would indicate >> that the 10Gb port is being stopped by the link partner. One thing you >> could try in order to test this further is to disable flow control on >> the 82599 port. To do that you can run the following where ethX is the >> name of the ixgbe interface you are currently using: >> ethtool -A ethX tx off rx off autoneg off >> >> Also you may want to set the 82599 and 82574 to the same interrupt >> rate. Currently it looks like the 82599 is being limited to 4000 >> interrupts per second while the 82574 is being allowed up to 20,000. If >> you let the 82599 use its' default throttle rate setting that should be >> comparable to the dynamic throttling provided by the e1000 driver and >> should improve TCP performance by reducing latency. > > ------------------------------------------------------------------------------ Get 100% visibility into Java/.NET code with AppDynamics Lite! It's a free troubleshooting tool designed for production. Get down to code-level detail for bottlenecks, with <2% overhead. Download for free and get started troubleshooting in minutes. http://pubads.g.doubleclick.net/gampad/clk?id=48897031&iu=/4140/ostg.clktrk _______________________________________________ E1000-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
