Michel Benoit <michel.benoit <at> rte.se> writes:
>
> I found the source of problem.
>
> The simulator model for the 82567 was only setting the DD (descriptor
> done) bit in the tx descriptors that are written back when the RS
> (report status) bit was 1.
> The e1000e driver only cleans up tx descriptors that have the DD bit set.
> When using tcp segmentation offset the RS bit is only set by the
> driver in the last descriptor of the group of descriptors that make up
> the packet to be sent.
>
> The result was that the e1000e driver was not cleaning up the tx
> descriptors in the queue and the queue became full until netdev
> watchdog reset the controller.
>
> Thanks for your tips that helped me debug e1000e and netdev.
>
> Michel
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> E1000-devel mailing list
> E1000-devel <at> lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/e1000-devel
> To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
>
>
Hey Guys,
we have the same issue on our prod. loadbalancer.
Debian Wheezy 7.1, linux-image-3.2.0-4-amd64 resp. 3.2.46-1
Linux lb1 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux
Jul 4 19:58:03 lb1 kernel: [ 7662.861537] sched: RT throttling activated
Jul 4 19:58:20 lb1 kernel: [ 7681.598546] ------------[ cut here ]------------
Jul 4 19:58:20 lb1 kernel: [ 7681.598552] WARNING: at
/build/linux-s5x2oE/linux-3.2.46/net/sched/sch_generic.c:256
dev_watchdog+0xf2/0x151()
Jul 4 19:58:20 lb1 kernel: [ 7681.598553] Hardware name: X9SCL/X9SCM
Jul 4 19:58:20 lb1 kernel: [ 7681.598555] NETDEV WATCHDOG: eth0 (e1000e):
transmit queue 0 timed out
Jul 4 19:58:20 lb1 kernel: [ 7681.598556] Modules linked in:
nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack bonding loop coretemp
crc32c_intel ghash_clmulni_intel aesni_intel psmouse i2c_i801 iTCO_wdt
aes_x86_64 i2c_core aes_generic iTCO_vendor_support joydev snd_pcm
snd_page_alloc snd_timer snd cryptd soundcore evdev serio_raw pcspkr
acpi_cpufreq mperf video processor button ext3 mbcache jbd usbhid hid sg
sd_mod crc_t10dif sr_mod cdrom ahci libahci libata fan thermal thermal_sys
ehci_hcd 3w_sas usbcore scsi_mod e1000e usb_common [last unloaded:
scsi_wait_scan]
Jul 4 19:58:20 lb1 kernel: [ 7681.598579] Pid: 2138, comm: haproxy Not
tainted 3.2.0-4-amd64 #1 Debian 3.2.46-1
Jul 4 19:58:20 lb1 kernel: [ 7681.598581] Call Trace:
Jul 4 19:58:20 lb1 kernel: [ 7681.598582] <IRQ> [<ffffffff81046b75>] ?
warn_slowpath_common+0x78/0x8c
Jul 4 19:58:20 lb1 kernel: [ 7681.598587] [<ffffffff81046c21>] ?
warn_slowpath_fmt+0x45/0x4a
Jul 4 19:58:20 lb1 kernel: [ 7681.598589] [<ffffffff812a68c9>] ?
netif_tx_lock+0x40/0x75
Jul 4 19:58:20 lb1 kernel: [ 7681.598592] [<ffffffff812a6a39>] ?
dev_watchdog+0xf2/0x151
Jul 4 19:58:20 lb1 kernel: [ 7681.598594] [<ffffffff81052334>] ?
run_timer_softirq+0x19a/0x261
Jul 4 19:58:20 lb1 kernel: [ 7681.598596] [<ffffffff812a6947>] ?
netif_tx_unlock+0x49/0x49
Jul 4 19:58:20 lb1 kernel: [ 7681.598598] [<ffffffff8104c1ac>] ?
__do_softirq+0xb9/0x177
Jul 4 19:58:20 lb1 kernel: [ 7681.598601] [<ffffffff81355dac>] ?
call_softirq+0x1c/0x30
Jul 4 19:58:20 lb1 kernel: [ 7681.598604] [<ffffffff8100f8cd>] ?
do_softirq+0x3c/0x7b
Jul 4 19:58:20 lb1 kernel: [ 7681.598606] [<ffffffff8104c414>] ?
irq_exit+0x3c/0x99
Jul 4 19:58:20 lb1 kernel: [ 7681.598608] [<ffffffff810241c0>] ?
smp_apic_timer_interrupt+0x74/0x82
Jul 4 19:58:20 lb1 kernel: [ 7681.598610] [<ffffffff8135461e>] ?
apic_timer_interrupt+0x6e/0x80
Jul 4 19:58:20 lb1 kernel: [ 7681.598611] <EOI> [<ffffffff81353b52>] ?
system_call_fastpath+0x16/0x1b
Jul 4 19:58:20 lb1 kernel: [ 7681.598614] ---[ end trace 57642987e579c2c1 ]---
Jul 4 19:58:25 lb1 kernel: [ 7686.432886] e1000e: eth0 NIC Link is Up 1000
Mbps Full Duplex, Flow Control: None
Jul 4 22:42:56 lb1 kernel: [17536.618147] bonding: bond0: link status
definitely down for interface eth0, disabling it
Jul 4 22:42:56 lb1 kernel: [17536.618150] bonding: bond0: making interface
eth1 the new active one.
Jul 4 22:42:58 lb1 kernel: [17539.828987] bonding: bond0: link status
definitely up for interface eth0.
Jul 4 22:42:58 lb1 kernel: [17539.828990] bonding: bond0: making interface
eth0 the new active one.
Jul 4 22:43:01 lb1 kernel: [17543.717220] e1000e: eth0 NIC Link is Up 1000
Mbps Full Duplex, Flow Control: None
At around 20:00 till ~22:35 it had about ~1k connections, around 22:35 we
got ~4k connections. Between 19:58 and 23:07 it started to flap (Nagios)
until 22:45, then it was almost completely down/unavailable. We had to
reboot the system since a login through the shell was nearly impossible /
would have taken too long. This is a prod. LB.
~2k connections isn't that much actually. The 4k connections are (at least
around this time) somewhat unusual and I'll have to analyze it further but I
doubt that this is related to the actual problem. The connection count has
been taken from "netstat -ts|awk '$0 ~ /connections established/ {print $1}'".
I upgraded the e1000e drivers on lb1 now to 2.4.14-NAPI and I really hope it
helps.
I first thought it may be related to
http://article.gmane.org/gmane.linux.kernel/1233566 and those two patches
from Chris Boot but unfortunately it's not that, the Wheezy Kernel already
has those Patches included :(
Btw. this is a physical system, so nothing virtual.
Jeff, is there some workaround at least? Or has it been fixed in the
mentioned driver version already?
Our second system (lb2) is *exactly* the same, the same hardware as well as
OS and so forth and it was running fine for about 12-13 hours, until we
switched back to lb1, to test the new drivers.
lb1 had no "sched: RT throttling activated" message. About two hours after
switching back to lb1 again we got the message again "sched: RT throttling
activated" but right now it's running fine, no call trace.
We replaced our old lb's yesterday, they had different/older hardware than
this ones. So different board, NIC's and so on. We're considering to switch
back to the old machines.
Here are some more details, I hope it helps:
eth0:
# lspci -vvv -nn -d 8086:1502
00:19.0 Ethernet controller [0200]: Intel Corporation 82579LM Gigabit
Network Connection [8086:1502] (rev 05)
Subsystem: Super Micro Computer Inc Device [15d9:1502]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency: 0
Interrupt: pin A routed to IRQ 43
Region 0: Memory at dfb00000 (32-bit, non-prefetchable) [size=128K]
Region 1: Memory at dfb25000 (32-bit, non-prefetchable) [size=4K]
Region 2: I/O ports at f020 [size=32]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Address: 00000000fee00398 Data: 0000
Capabilities: [e0] PCI Advanced Features
AFCap: TP+ FLR+
AFCtrl: FLR-
AFStatus: TP-
Kernel driver in use: e1000e
eth1:
# lspci -vvv -nn -d 8086:10d3
04:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network
Connection [8086:10d3]
Subsystem: Super Micro Computer Inc Device [15d9:0000]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 16
Region 0: Memory at df900000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at d000 [size=32]
Region 3: Memory at df920000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [c8] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA
PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [e0] Express (v1) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns,
L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+
Unsupported+
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+
TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Latency
L0 <128ns,
L1 <64us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+
DLActive- BWMgmt-
ABWMgmt-
Capabilities: [a0] MSI-X: Enable+ Count=5 Masked-
Vector table: BAR=3 offset=00000000
PBA: BAR=3 offset=00002000
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt-
RxOF- MalfTLP-
ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt-
RxOF+ MalfTLP+
ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Capabilities: [140 v1] Device Serial Number 00-25-90-ff-ff-a8-02-5e
Kernel driver in use: e1000e
Jun 25 16:17:53 lb1 kernel: [ 1.605548] e1000e: Intel(R) PRO/1000 Network
Driver - 1.5.1-k
Jun 25 16:17:53 lb1 kernel: [ 1.605550] e1000e: Copyright(c) 1999 - 2011
Intel Corporation.
Jun 25 16:17:53 lb1 kernel: [ 1.912276] e1000e 0000:00:19.0: eth0: (PCI
Express:2.5GT/s:Width x1) 00:25:90:a8:02:5f
Jun 25 16:17:53 lb1 kernel: [ 1.912279] e1000e 0000:00:19.0: eth0:
Intel(R) PRO/1000 Network Connection
Jun 25 16:17:53 lb1 kernel: [ 1.912333] e1000e 0000:00:19.0: eth0: MAC:
10, PHY: 11, PBA No: FFFFFF-0FF
Jun 25 16:17:53 lb1 kernel: [ 1.912342] e1000e 0000:04:00.0: Disabling
ASPM L0s L1
Jun 25 16:17:53 lb1 kernel: [ 2.023924] e1000e 0000:04:00.0: eth1: (PCI
Express:2.5GT/s:Width x1) 00:25:90:a8:02:5e
Jun 25 16:17:53 lb1 kernel: [ 2.023927] e1000e 0000:04:00.0: eth1:
Intel(R) PRO/1000 Network Connection
Jun 25 16:17:53 lb1 kernel: [ 2.024016] e1000e 0000:04:00.0: eth1: MAC:
3, PHY: 8, PBA No: FFFFFF-0FF
Jun 25 16:17:55 lb1 kernel: [ 7.303013] e1000e: eth1 NIC Link is Up 1000
Mbps Full Duplex, Flow Control: None
Jun 25 16:17:55 lb1 kernel: [ 7.738759] e1000e: eth0 NIC Link is Up 1000
Mbps Full Duplex, Flow Control: None
------------------------------------------------------------------------------
This SF.net email is sponsored by Windows:
Build for Windows Store.
http://p.sf.net/sfu/windows-dev2dev
_______________________________________________
E1000-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/e1000-devel
To learn more about Intel® Ethernet, visit
http://communities.intel.com/community/wired
