On 04/03/2014 01:22 PM, Fujinaka, Todd wrote: > OK, I'm not running on all cylinders today. We also need software > configuration with version information and exact repro steps. Something we > can tell the lab for repro.
I don't know any off-the-shelf software that supports setting TCP MSS, but maybe iperf or similar can either do it now or can be modified to make it support that feature. If you don't have anything that reproduces this easily, let me know and I'll hack iperf or something similar and get a reproducible test case for you. Kernel was 3.9.11+ (including some local patches, but nothing that should impact this bug). We used our LANforge traffic generator to generate the traffic that caused the issues. Thanks, Ben > > Thanks. > > Todd Fujinaka > Software Application Engineer > Networking Division (ND) > Intel Corporation > [email protected] > (503) 712-4565 > > > -----Original Message----- > From: Fujinaka, Todd > Sent: Thursday, April 03, 2014 1:21 PM > To: Fujinaka, Todd; Ben Greear; e1000-devel list > Subject: RE: [E1000-devel] irq, watchdog errors with 82574L, TSO, and small > TCP MSS. > > Ben, > Can you tell us your exact hardware just in case that's required for this > issue to reproduce? > > Thanks. > > Todd Fujinaka > Software Application Engineer > Networking Division (ND) > Intel Corporation > [email protected] > (503) 712-4565 > > > -----Original Message----- > From: Fujinaka, Todd [mailto:[email protected]] > Sent: Thursday, April 03, 2014 12:51 PM > To: Ben Greear; e1000-devel list > Subject: Re: [E1000-devel] irq, watchdog errors with 82574L, TSO, and small > TCP MSS. > > I think we know what's going on and we're looking into this. > > Thanks. > > Todd Fujinaka > Software Application Engineer > Networking Division (ND) > Intel Corporation > [email protected] > (503) 712-4565 > > > -----Original Message----- > From: Ben Greear [mailto:[email protected]] > Sent: Monday, March 31, 2014 11:56 AM > To: e1000-devel list > Subject: [E1000-devel] irq, watchdog errors with 82574L, TSO, and small TCP > MSS. > > We have a test where we set the TCP MSS to a small value and then generate > traffic at high speeds. This causes the NIC to segment packets, giving us > some easy hardware-accel for creating lots of small packets on the wire. > Good for testing. > > With some on-board NICs (82474L), we see some problems when we do this unless > we disable TCP offload (TSO). MSS was set to 256 for this test. > > We do not see this problem with the i350 based ports on the add-in NIC. > > lspci: > > [root@localhost ~]# lspci > 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM > Controller (rev 09) > 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor > PCI Express Root Port (rev 09) > 00:16.0 Communication controller: Intel Corporation 6 Series/C200 Series > Chipset Family MEI Controller #1 (rev 04) > 00:16.1 Communication controller: Intel Corporation 6 Series/C200 Series > Chipset Family MEI Controller #2 (rev 04) > 00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family > USB Enhanced Host Controller #2 (rev 05) > 00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI > Express Root Port 1 (rev b5) > 00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI > Express Root Port 5 (rev b5) > 00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI > Express Root Port 6 (rev b5) > 00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI > Express Root Port 7 (rev b5) > 00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI > Express Root Port 8 (rev b5) > 00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family > USB Enhanced Host Controller #1 (rev 05) > 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5) > 00:1f.0 ISA bridge: Intel Corporation C204 Chipset Family LPC Controller (rev > 05) > 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset > Family SATA AHCI Controller (rev 05) > 00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus > Controller (rev 05) > 01:00.0 PCI bridge: PLX Technology, Inc. PEX 8617 16-lane, 4-Port PCI Express > Gen 2 (5.0 GT/s) Switch with P2P (rev ba) > 02:01.0 PCI bridge: PLX Technology, Inc. PEX 8617 16-lane, 4-Port PCI Express > Gen 2 (5.0 GT/s) Switch with P2P (rev ba) > 02:03.0 PCI bridge: PLX Technology, Inc. PEX 8617 16-lane, 4-Port PCI Express > Gen 2 (5.0 GT/s) Switch with P2P (rev ba) > 03:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network > Connection (rev 01) > 03:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network > Connection (rev 01) > 03:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network > Connection (rev 01) > 03:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network > Connection (rev 01) > 05:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network > Connection (rev 01) > 05:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network > Connection (rev 01) > 08:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network > Connection > 09:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network > Connection > 0a:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network > Connection > 0b:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network > Connection > 0c:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW > WPCM450 (rev 0a) [root@localhost ~]# > > > dmesg: > [23685.770208] e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow > Control: Rx/Tx [23685.793946] e1000e: eth2 NIC Link is Up 1000 Mbps Full > Duplex, Flow Control: Rx/Tx [23685.794452] IPv6: ADDRCONF(NETDEV_CHANGE): > eth2: link becomes ready [23801.390528] irq 101: nobody cared (try booting > with the "irqpoll" option) > [23801.396373] Pid: 42, comm: ksoftirqd/5 Tainted: G C O 3.9.11+ #136 > [23801.396374] Call Trace: > [23801.396376] <IRQ> [<ffffffff811065db>] __report_bad_irq+0x3a/0xc6 > [23801.396384] [<ffffffff811067d0>] note_interrupt+0x169/0x1e5 > [23801.396386] [<ffffffff81104761>] handle_irq_event_percpu+0x18f/0x1e5 > [23801.396388] [<ffffffff811047f8>] handle_irq_event+0x41/0x61 > [23801.396390] [<ffffffff81106fbc>] handle_edge_irq+0xa6/0xcb [23801.396393] > [<ffffffff81011d9f>] handle_irq+0x24/0x2d [23801.396397] > [<ffffffff815d38cd>] do_IRQ+0x4d/0xb4 [23801.396400] [<ffffffff815cb9ed>] > common_interrupt+0x6d/0x6d [23801.396400] <EOI> [<ffffffff81509b67>] ? > __netdev_alloc_frag+0xfb/0x112 [23801.396406] [<ffffffff8150adea>] > __netdev_alloc_skb+0x4c/0xfa [23801.396408] [<ffffffff81512f17>] ? > napi_gro_receive+0xe6/0xf1 [23801.396417] [<ffffffffa086ea7c>] > igb_poll+0x4a2/0xc92 [igb] [23801.396420] [<ffffffff810c0058>] ? > cpuacct_charge+0x67/0x6f [23801.396422] [<ffffffff810c6969>] ? > update_stats_wait_end+0x7d/0xd3 [23801.396424] [<ffffffff81512a43>] > net_rx_action+0xc2/0x267 [23801.396428] [<ffffffff8109da08>] > __do_softirq+0x114/0x254 [23801.396430] [<ffffffff8109db70>] > run_ksoftirqd+0x28/0x47 [23801.396433] [<ffffffff810bb40c>] > smpboot_thread_fn+0x217/0x21f [23801.396436] [<ffffff! ff810bb1f5 >] ? test_ti_thread_flag.clone.0+0x11/0x11 > [23801.396439] [<ffffffff810b496d>] kthread+0xb5/0xbd [23801.396442] > [<ffffffff810b48b8>] ? kthread_freezable_should_stop+0x60/0x60 > [23801.396444] [<ffffffff815d1bec>] ret_from_fork+0x7c/0xb0 [23801.396447] > [<ffffffff810b48b8>] ? kthread_freezable_should_stop+0x60/0x60 > [23801.396448] handlers: > [23801.397492] [<ffffffffa078b73e>] e1000_msix_other [e1000e] [23801.402016] > Disabling IRQ #101 [23803.545095] irq 89: nobody cared (try booting with the > "irqpoll" option) > [23803.550903] Pid: 0, comm: swapper/7 Tainted: G C O 3.9.11+ #136 > [23803.550904] Call Trace: > [23803.550905] <IRQ> [<ffffffff811065db>] __report_bad_irq+0x3a/0xc6 > [23803.550913] [<ffffffff811067d0>] note_interrupt+0x169/0x1e5 > [23803.550916] [<ffffffff81104761>] handle_irq_event_percpu+0x18f/0x1e5 > [23803.550918] [<ffffffff811047f8>] handle_irq_event+0x41/0x61 > [23803.550920] [<ffffffff81106fbc>] handle_edge_irq+0xa6/0xcb [23803.550924] > [<ffffffff81011d9f>] handle_irq+0x24/0x2d [23803.550928] > [<ffffffff815d38cd>] do_IRQ+0x4d/0xb4 [23803.550931] [<ffffffff815cb9ed>] > common_interrupt+0x6d/0x6d [23803.550932] <EOI> [<ffffffff814c8209>] ? > cpuidle_wrap_enter+0x46/0x78 [23803.550938] [<ffffffff814c8209>] ? > cpuidle_wrap_enter+0x46/0x78 [23803.550940] [<ffffffff814c81ff>] ? > cpuidle_wrap_enter+0x3c/0x78 [23803.550943] [<ffffffff814c824b>] > cpuidle_enter_tk+0x10/0x12 [23803.550945] [<ffffffff814c7d05>] > cpuidle_enter_state+0x17/0x3f [23803.550947] [<ffffffff814c8484>] > cpuidle_idle_call+0xba/0xfa [23803.550950] [<ffffffff810177dd>] > cpu_idle+0x65/0xb5 [23803.550953] [<ffffffff815c47b8>] > start_secondary+0x211/0x213 [23803.550954] handlers: > [23803.551992] [<ffffffffa078b73e>] e1000_msix_other [e1000e] [23803.556487] > Disabling IRQ #89 [23805.767276] NETDEV WATCHDOG: eth2 (e1000e): transmit > queue 0 timed out, trans_start: 4318501615, wd-timeout: 5000 jiffies: > 4318510992 tx-queues: 1 [23805.767285] ------------[ cut here ]------------ > [23805.767290] WARNING: at > /home/greearb/git/linux-3.9.dev.y/net/sched/sch_generic.c:261 > dev_watchdog+0x135/0x19b() [23805.767292] Hardware name: X9SCI/X9SCA > [23805.767293] Modules linked in: nf_nat_ipv4 nf_nat 8021q garp stp mrp llc > macvlan wanlink(O) pktgen fuse ip6table_filter ip6_tables ebtable_nat > ebtables igb e1000e coretemp acpi_cpufreq mperf intel_powerclamp kvm_intel > kvm hwmon dca iTCO_wdt iTCO_vendor_support microcode pcspkr serio_raw > i2c_i801 joydev lpc_ich ptp pps_core shpchp video uinput ipv6 usb_storage > mgag200 i2c_algo_bit drm_kms_helper ttm drm i2c_core [last unloaded: > iptable_nat] > [23805.767325] Pid: 0, comm: swapper/4 Tainted: G C O 3.9.11+ #136 > [23805.767326] Call Trace: > [23805.767327] <IRQ> [<ffffffff81096291>] warn_slowpath_common+0x85/0x9f > [23805.767336] [<ffffffff810962c5>] warn_slowpath_null+0x1a/0x1c > [23805.767338] [<ffffffff8152dfb4>] dev_watchdog+0x135/0x19b [23805.767341] > [<ffffffff8152de7f>] ? netif_tx_unlock+0x76/0x76 [23805.767344] > [<ffffffff810a4500>] call_timer_fn+0x57/0x12c [23805.767347] > [<ffffffff810a4855>] run_timer_softirq+0x197/0x1df [23805.767349] > [<ffffffff8152de7f>] ? netif_tx_unlock+0x76/0x76 [23805.767353] > [<ffffffff8109da08>] __do_softirq+0x114/0x254 [23805.767355] > [<ffffffff8109dbda>] irq_exit+0x4b/0xa8 [23805.767359] [<ffffffff815d39bf>] > smp_apic_timer_interrupt+0x8b/0x99 > [23805.767365] [<ffffffff815d289d>] apic_timer_interrupt+0x6d/0x80 > [23805.767367] <EOI> [<ffffffff815cb4a1>] ? > _raw_spin_unlock_irqrestore+0x31/0x3c > [23805.767374] [<ffffffff814c8209>] ? cpuidle_wrap_enter+0x46/0x78 > [23805.767377] [<ffffffff814c81ff>] ? cpuidle_wrap_enter+0x3c/0x78 > [23805.767380] [<ffffffff814c824b>] cpuidle_enter_tk+0x10/0x12 > [23805.767383] [<ffffffff814c7d05>] cpuidle_enter_state+0x17/0x3f > [23805.767387] [<ffffffff814c8484>] cpuidle_idle_call+0xba/0xfa > [23805.767391] [<ffffffff810177dd>] cpu_idle+0x65/0xb5 [23805.767396] > [<ffffffff815c47b8>] start_secondary+0x211/0x213 [23805.767400] ---[ end > trace 21ceb1647ffbb4c8 ]--- [23805.767412] e1000e 0000:0a:00.0 eth2: Reset > adapter unexpectedly [23805.783253] NETDEV WATCHDOG: eth3 (e1000e): transmit > queue 0 timed out, trans_start: 4318502992, wd-timeout: 5000 jiffies: > 4318511008 tx-queues: 1 [23805.783260] e1000e 0000:0b:00.0 eth3: Reset > adapter unexpectedly [23884.253486] IPv6: ADDRCONF(NETDEV_UP): eth2: link is > not ready [23884.253489] 8021q: adding VLAN 0 to HW filter on device eth2 > [23884.408039] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready ! [23884.408 044] 8021q: adding VLAN 0 to HW filter on device eth3 [23886.830934] e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx [23886.831369] IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready [23886.854655] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx [23886.855055] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready [23896.616030] NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_start: 4318596830, wd-timeout: 5000 jiffies: 4318601984 tx-queues: 1 [23896.616051] e1000e 0000:0b:00.0 eth3: Reset adapter unexpectedly [23897.431037] e1000e: eth2 NIC Link is Down [23900.022751] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx [23900.058162] e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx [23945.562851] NETDEV WATCHDOG: eth2 (e1000e): transmit queue 0 timed out, trans_start: 4318641649, wd-timeout: 5000 jiffies: 4318651008 tx-queues: 1 [23945.562853] NETDEV WATCHDOG: eth3! (e1000e): transmit queue 0 timed out, trans_start: 4318641850, wd-timeout: 5000 jiffies: 4318651008 tx-queues: 1 [23945.562863] e1000e 0000:0a:00.0 eth2: Reset adapter unexpectedly [23945.562866] e1000e 0000:0b:00.0 eth3: Reset adapter unexpectedly [23948.047189] e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx [23948.055451] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx [24802.212164] NETDEV WATCHDOG: eth3 (e1000e): transmit queue 0 timed out, trans_start: 4319501792, wd-timeout: 5000 jiffies: 4319509008 tx-queues: 1 [24802.212252] e1000e 0000:0b:00.0 eth3: Reset adapter unexpectedly [24803.039033] e1000e: eth2 NIC Link is Down [24805.520298] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx [24805.545601] e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx [25170.569269] irq 89: nobody cared (try booting with the "irqpoll" option) > [25170.575072] Pid: 0, comm: swapper/3 Tainted: G WC O 3.9.11+ #136 > [25170.575073] Call Trace: > [25170.575075] <IRQ> [<ffffffff811065db>] __report_bad_irq+0x3a/0xc6 > [25170.575085] [<ffffffff811067d0>] note_interrupt+0x169/0x1e5 > [25170.575088] [<ffffffff81104761>] handle_irq_event_percpu+0x18f/0x1e5 > [25170.575090] [<ffffffff811047f8>] handle_irq_event+0x41/0x61 > [25170.575092] [<ffffffff81106fbc>] handle_edge_irq+0xa6/0xcb [25170.575096] > [<ffffffff81011d9f>] handle_irq+0x24/0x2d [25170.575100] > [<ffffffff815d38cd>] do_IRQ+0x4d/0xb4 [25170.575103] [<ffffffff815cb9ed>] > common_interrupt+0x6d/0x6d [25170.575104] <EOI> [<ffffffff810e0e2a>] ? > tick_nohz_idle_exit+0x152/0x15b [25170.575111] [<ffffffff810e0dda>] ? > tick_nohz_idle_exit+0x102/0x15b [25170.575115] [<ffffffff8101781d>] > cpu_idle+0xa5/0xb5 [25170.575119] [<ffffffff815c47b8>] > start_secondary+0x211/0x213 [25170.575120] handlers: > [25170.576161] [<ffffffffa078b73e>] e1000_msix_other [e1000e] [25170.580714] > Disabling IRQ #89 [25170.623312] NETDEV WATCHDOG: eth2 (e1000e): transmit > queue 0 timed out, trans_start: 4319872046, wd-timeout: 5000 jiffies: > 4319878000 tx-queues: 1 [25170.623326] e1000e 0000:0a:00.0 eth2: Reset > adapter unexpectedly [25170.639287] NETDEV WATCHDOG: eth3 (e1000e): transmit > queue 0 timed out, trans_start: 4319871842, wd-timeout: 5000 jiffies: > 4319878016 tx-queues: 1 [25170.639298] e1000e 0000:0b:00.0 eth3: Reset > adapter unexpectedly [25173.147951] e1000e: eth3 NIC Link is Up 1000 Mbps > Full Duplex, Flow Control: Rx/Tx [25732.742950] IPv6: ADDRCONF(NETDEV_UP): > eth2: link is not ready [25732.742954] 8021q: adding VLAN 0 to HW filter on > device eth2 [25732.900780] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready > [25732.900784] 8021q: adding VLAN 0 to HW filter on device eth3 > [25735.327650] e1000e: eth3 NIC Link is Up 1000 Mbps Full Duplex, Flow > Control: Rx/Tx [25735.328136] IPv6: ADD! RCONF(NETD EV_CHANGE): eth3: link becomes ready [25735.335623] e1000e: eth2 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx [25735.336123] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready > > > Thanks, > Ben > > -- > Ben Greear <[email protected]> > Candela Technologies Inc http://www.candelatech.com > > > ------------------------------------------------------------------------------ > _______________________________________________ > E1000-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/e1000-devel > To learn more about Intel® Ethernet, visit > http://communities.intel.com/community/wired > > ------------------------------------------------------------------------------ > _______________________________________________ > E1000-devel mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/e1000-devel > To learn more about Intel® Ethernet, visit > http://communities.intel.com/community/wired > -- Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com ------------------------------------------------------------------------------ _______________________________________________ E1000-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/e1000-devel To learn more about Intel® Ethernet, visit http://communities.intel.com/community/wired
