Re: kernel crashes after soft lockups in xen domU

Wed, 05 Nov 2014 08:57:06 -0800 

And some more information ...

Am 2014-10-13 12:04, schrieb Jonas Meurer:

Am 2014-08-19 12:26, schrieb Jonas Meurer:
I encounter kernel crashes on an up-to-date Debian/Wheezy Xen domU with the stock kernel. The dom0 runs the same linux kernel and xen/4.1.4-3+deb7u1. 


the bug is still reproducible with the latest kernel and Xen packages 
from
Debian Wheezy. Unfortunately it seems like a corner case, somehow 
related

to the hardware setup. I'm unable to reproduce the crash on another Xen

system with same kernel an Xen versions but different CPU and 
motherboard.
The VM runs in production mode on the second Xen system since several 
weeks

without one single crash.


I've got a test system now where I'm able to reproduce the bug by 
putting the
VM (a webserver) under heavy load with the help of siege and pylot. The 
VM
crashes every time I put the webserver under heavy load, everytime with 
the

same backtrace.



I replaced the full system (all hardware components except the 
harddisks)

with a new one in the meantime - and still the bug is repoducible. I'll
try to describe the setup:

Two Xen virtualization servers (xen1 and xen2), mirroring one block
device with DRBD using a primary/secondary setup. The DRBD device
holds LVM with the LVs for one single virtual machine (the webserver).
This webserver runs an Apache2 daemon.

The first virtualization server (xen1, the one that's live) runs rock
stable, same for the webserver VM on top. No crashes, no exploding
load. The second virtualization server (xen2) runs well as long as
it's only secondary (i.e. no virtual machine started). As soon as I
switch the DRBD primary to xen2 and start the webserver VM there,
load on the webserver is unusual high, it becomes laggy and after
some hours (sometimes minutes) it crashes like described in earlier
mails.

Now I created a test-VM on xen2 that is not on top of the DRBD device
in order to factor out DRBD as reason. As already written, if I fire
some HTTP requests against the Apache daemon on the test-VM, the VM
crashes the same way.

I first replaced memory modules and CPU by similar ones without
results. Now I replaced the whole hardware (except harddisks) by
another one - still the same crashes.

So the question is: why does the VM run stable on xen1 while it
crashes all the time on xen2. If I compare xen1 and xen2, only
real difference is mainboard (Supermicro X8 on xen1; Supermicro
X9 on xen2) and CPU (Xeon L5939 on xen1; E5-2609 on xen2)

As a next step I'll put the harddisks into another X8/Xeon L5639
server system and try to reproduce the crashes there. My bet is
that this system will not crash anymore. In other words, I guess
that this very bug is only triggered with the X9 + E-2609
combination.


Can I do anything additional to help debugging the bug? Shall I report 
it

to Xen upstream or send it to lkml?


Still the same question. Shall I send the bugreport to upstream?
Unfortunately nobody from Debian Linux kernel and/or Xen team seems
to care :-/

Cheers,
 jonas


Regarding the hardware: the RAM was checked with memtest86+ and no 
errors
found and the CPU has been replaced by a new one (same model). Still, 
the

VM crash is reproducible.

The hardware on the crashing system is:
CPU: Intel Xeon E5-2609v2/4x2,5GHz
Motherboard: Supermicro X9SRI-F

For information, the hardware on non-crashing system is:
CPU: Intel XEON L5639/6x2,13 GHz
Motherboard: Supermicro X8STi


It seems like the crashes are related to a RT process, even though no

sched_fifo/rr processes are started on this system intentionally. 
Also, the
CPU usage is low all the time, no peaks at all. But the kernel 
reports:


kernel: [39101.461586] sched: RT throttling activated



This is still valid, even though I no longer think that it's related to 
a

RT process at all, as no sched_fifo/rr processes are started.


Usually, a few minutes later, soft lockups start to happen, and then 
the

system crashes:


The backtrace is slightly different now due to kernel and Xen updates:

[353013.384931] sched: RT throttling activated
[354008.100835] BUG: soft lockup - CPU#5 stuck for 22s! [apache2:24848]
[354008.100846] Modules linked in: evdev coretemp crc32c_intel
ghash_clmulni_intel snd_pcm snd_page_alloc aesni_intel snd_timer
aes_x86_64 snd aes_generic soundcore cryptd pcspkr ext4 crc16 jbd2
mbcache dm_mod xen_netfront xen_blkfront
[354008.100872] CPU 5
[354008.100874] Modules linked in: evdev coretemp crc32c_intel
ghash_clmulni_intel snd_pcm snd_page_alloc aesni_intel snd_timer
aes_x86_64 snd aes_generic soundcore cryptd pcspkr ext4 crc16 jbd2
mbcache dm_mod xen_netfront xen_blkfront
[354008.100894]
[354008.100898] Pid: 24848, comm: apache2 Not tainted 3.2.0-4-amd64 #1
Debian 3.2.60-1+deb7u3
[354008.100904] RIP: e030:[<ffffffff8100122a>]  [<ffffffff8100122a>]
hypercall_page+0x22a/0x1000
[354008.100914] RSP: e02b:ffff8802f0b41c00  EFLAGS: 00000246
[354008.100918] RAX: 0000000000040001 RBX: ffff8802ffff0200 RCX:
ffffffff8100122a
[354008.100922] RDX: ffffffffffffffc8 RSI: 0000000000000000 RDI:
0000000000000000
[354008.100927] RBP: 000000000000000e R08: 0000000000000200 R09:
dead000000100100
[354008.100931] R10: dead000000200200 R11: 0000000000000246 R12:
ffff88028e261da0
[354008.100935] R13: ffff8802ffffbe00 R14: ffffea00099905e8 R15:
000000000000000d
[354008.100944] FS:  00007f7b66cd2740(0000) GS:ffff8802ffd40000(0000)
knlGS:0000000000000000
[354008.100949] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[354008.100953] CR2: 00007f7b68c2b000 CR3: 00000002855b7000 CR4:
0000000000002660
[354008.100958] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[354008.100962] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[354008.100967] Process apache2 (pid: 24848, threadinfo
ffff8802f0b40000, task ffff8802ef49c8c0)
[354008.100972] Stack:
[354008.100974]  0000000000000000 0000000000000200 ffffffff81006790
ffffffff81006d22
[354008.100981]  0000000000000000 dead000000200200 dead000000100100
0000000000000200
[354008.100988]  ffff8802ffff0200 ffff8802ffff0200 ffffffffffffffc8
0000000000000200
[354008.100995] Call Trace:

[354008.101000]  [<ffffffff81006790>] ? 
xen_force_evtchn_callback+0x9/0xa

[354008.101006]  [<ffffffff81006d22>] ? check_events+0x12/0x20

[354008.101011]  [<ffffffff81006d0f>] ? 
xen_restore_fl_direct_reloc+0x4/0x4

[354008.101017]  [<ffffffff81071153>] ? arch_local_irq_restore+0x7/0x8

[354008.101024]  [<ffffffff8135049f>] ? 
_raw_spin_unlock_irqrestore+0xe/0xf

[354008.101031]  [<ffffffff810be895>] ? release_pages+0xf4/0x14d

[354008.101038]  [<ffffffff810de78b>] ? 
free_pages_and_swap_cache+0x48/0x60

[354008.101045]  [<ffffffff810cf527>] ? tlb_flush_mmu+0x37/0x50
[354008.101049]  [<ffffffff810cf54c>] ? tlb_finish_mmu+0xc/0x31
[354008.101054]  [<ffffffff810d5e79>] ? exit_mmap+0xc4/0xe9
[354008.101060]  [<ffffffff81044b82>] ? mmput+0x56/0xf8
[354008.101064]  [<ffffffff81049d07>] ? exit_mm+0x117/0x122
[354008.101069]  [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
[354008.101074]  [<ffffffff81350487>] ? _raw_spin_lock_irq+0xa/0x14
[354008.101078]  [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
[354008.101083]  [<ffffffff81049f57>] ? do_exit+0x245/0x713
[354008.101088]  [<ffffffff81039a92>] ? finish_task_switch+0x4e/0xb9
[354008.101093]  [<ffffffff8134f0e9>] ? __schedule+0x5f9/0x610
[354008.101098]  [<ffffffff810d5c58>] ? do_munmap+0x2da/0x2f3
[354008.101102]  [<ffffffff8104a6a5>] ? do_group_exit+0x74/0x9e
[354008.101107]  [<ffffffff8104a6de>] ? sys_exit_group+0xf/0xf
[354008.101112]  [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b
[354008.101115] Code: cc 51 41 53 b8 10 00 00 00 0f 05 41 5b 59 c3 cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 11 00
00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
cc cc
[354008.101157] Call Trace:

[354008.101160]  [<ffffffff81006790>] ? 
xen_force_evtchn_callback+0x9/0xa

[354008.101165]  [<ffffffff81006d22>] ? check_events+0x12/0x20

[354008.101170]  [<ffffffff81006d0f>] ? 
xen_restore_fl_direct_reloc+0x4/0x4

[354008.101175]  [<ffffffff81071153>] ? arch_local_irq_restore+0x7/0x8

[354008.101180]  [<ffffffff8135049f>] ? 
_raw_spin_unlock_irqrestore+0xe/0xf

[354008.101185]  [<ffffffff810be895>] ? release_pages+0xf4/0x14d

[354008.101190]  [<ffffffff810de78b>] ? 
free_pages_and_swap_cache+0x48/0x60

[354008.101195]  [<ffffffff810cf527>] ? tlb_flush_mmu+0x37/0x50
[354008.101199]  [<ffffffff810cf54c>] ? tlb_finish_mmu+0xc/0x31
[354008.101204]  [<ffffffff810d5e79>] ? exit_mmap+0xc4/0xe9
[354008.101209]  [<ffffffff81044b82>] ? mmput+0x56/0xf8
[354008.101213]  [<ffffffff81049d07>] ? exit_mm+0x117/0x122
[354008.101217]  [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
[354008.101221]  [<ffffffff81350487>] ? _raw_spin_lock_irq+0xa/0x14
[354008.101226]  [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
[354008.101230]  [<ffffffff81049f57>] ? do_exit+0x245/0x713
[354008.101234]  [<ffffffff81039a92>] ? finish_task_switch+0x4e/0xb9
[354008.101239]  [<ffffffff8134f0e9>] ? __schedule+0x5f9/0x610
[354008.101243]  [<ffffffff810d5c58>] ? do_munmap+0x2da/0x2f3
[354008.101247]  [<ffffffff8104a6a5>] ? do_group_exit+0x74/0x9e
[354008.101251]  [<ffffffff8104a6de>] ? sys_exit_group+0xf/0xf
[354008.101256]  [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b
[354036.127528] BUG: soft lockup - CPU#5 stuck for 22s! [apache2:24848]
[354036.127539] Modules linked in: evdev coretemp crc32c_intel
ghash_clmulni_intel snd_pcm snd_page_alloc aesni_intel snd_timer
aes_x86_64 snd aes_generic soundcore cryptd pcspkr ext4 crc16 jbd2
mbcache dm_mod xen_netfront xen_blkfront
[354036.127564] CPU 5
[354036.127567] Modules linked in: evdev coretemp crc32c_intel
ghash_clmulni_intel snd_pcm snd_page_alloc aesni_intel snd_timer
aes_x86_64 snd aes_generic soundcore cryptd pcspkr ext4 crc16 jbd2
mbcache dm_mod xen_netfront xen_blkfront
[354036.127587]
[354036.127590] Pid: 24848, comm: apache2 Not tainted 3.2.0-4-amd64 #1
Debian 3.2.60-1+deb7u3
[354036.127597] RIP: e030:[<ffffffff8100122a>]  [<ffffffff8100122a>]
hypercall_page+0x22a/0x1000
[354036.127606] RSP: e02b:ffff8802f0b41c00  EFLAGS: 00000246
[354036.127610] RAX: 0000000000040001 RBX: ffff8802ffff0200 RCX:
ffffffff8100122a
[354036.127615] RDX: ffffffffffffffc8 RSI: 0000000000000000 RDI:
0000000000000000
[354036.127619] RBP: 000000000000000e R08: 0000000000000200 R09:
dead000000100100
[354036.127623] R10: dead000000200200 R11: 0000000000000246 R12:
ffff880265e01cc0
[354036.127627] R13: ffff8802ffffbe00 R14: ffffea00099748c0 R15:
000000000000000d
[354036.127637] FS:  00007f7b66cd2740(0000) GS:ffff8802ffd40000(0000)
knlGS:0000000000000000
[354036.127642] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[354036.127646] CR2: 00007f7b68c2b000 CR3: 00000002855b7000 CR4:
0000000000002660
[354036.127650] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[354036.127655] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[354036.127660] Process apache2 (pid: 24848, threadinfo
ffff8802f0b40000, task ffff8802ef49c8c0)
[354036.127665] Stack:
[354036.127667]  0000000000000000 0000000000000200 ffffffff81006790
ffffffff81006d22
[354036.127674]  0000000000000000 dead000000200200 dead000000100100
0000000000000200
[354036.127680]  ffff8802ffff0200 ffff8802ffff0200 ffffffffffffffc8
0000000000000200
[354036.127687] Call Trace:

[354036.127693]  [<ffffffff81006790>] ? 
xen_force_evtchn_callback+0x9/0xa

[354036.127699]  [<ffffffff81006d22>] ? check_events+0x12/0x20

[354036.127704]  [<ffffffff81006d0f>] ? 
xen_restore_fl_direct_reloc+0x4/0x4

[354036.127710]  [<ffffffff81071153>] ? arch_local_irq_restore+0x7/0x8

[354036.127717]  [<ffffffff8135049f>] ? 
_raw_spin_unlock_irqrestore+0xe/0xf

[354036.127723]  [<ffffffff810be895>] ? release_pages+0xf4/0x14d

[354036.127730]  [<ffffffff810de78b>] ? 
free_pages_and_swap_cache+0x48/0x60

[354036.127736]  [<ffffffff810cf527>] ? tlb_flush_mmu+0x37/0x50
[354036.127741]  [<ffffffff810cf54c>] ? tlb_finish_mmu+0xc/0x31
[354036.127746]  [<ffffffff810d5e79>] ? exit_mmap+0xc4/0xe9
[354036.127752]  [<ffffffff81044b82>] ? mmput+0x56/0xf8
[354036.127756]  [<ffffffff81049d07>] ? exit_mm+0x117/0x122
[354036.127760]  [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
[354036.127765]  [<ffffffff81350487>] ? _raw_spin_lock_irq+0xa/0x14
[354036.127769]  [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
[354036.127773]  [<ffffffff81049f57>] ? do_exit+0x245/0x713
[354036.127779]  [<ffffffff81039a92>] ? finish_task_switch+0x4e/0xb9
[354036.127784]  [<ffffffff8134f0e9>] ? __schedule+0x5f9/0x610
[354036.127788]  [<ffffffff810d5c58>] ? do_munmap+0x2da/0x2f3
[354036.127793]  [<ffffffff8104a6a5>] ? do_group_exit+0x74/0x9e
[354036.127797]  [<ffffffff8104a6de>] ? sys_exit_group+0xf/0xf
[354036.127802]  [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b
[354036.127805] Code: cc 51 41 53 b8 10 00 00 00 0f 05 41 5b 59 c3 cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 11 00
00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
cc cc
[354036.127846] Call Trace:

[354036.127849]  [<ffffffff81006790>] ? 
xen_force_evtchn_callback+0x9/0xa

[354036.127854]  [<ffffffff81006d22>] ? check_events+0x12/0x20

[354036.127859]  [<ffffffff81006d0f>] ? 
xen_restore_fl_direct_reloc+0x4/0x4

[354036.127864]  [<ffffffff81071153>] ? arch_local_irq_restore+0x7/0x8

[354036.127868]  [<ffffffff8135049f>] ? 
_raw_spin_unlock_irqrestore+0xe/0xf

[354036.127873]  [<ffffffff810be895>] ? release_pages+0xf4/0x14d

[354036.127879]  [<ffffffff810de78b>] ? 
free_pages_and_swap_cache+0x48/0x60

[354036.127883]  [<ffffffff810cf527>] ? tlb_flush_mmu+0x37/0x50
[354036.127888]  [<ffffffff810cf54c>] ? tlb_finish_mmu+0xc/0x31
[354036.127893]  [<ffffffff810d5e79>] ? exit_mmap+0xc4/0xe9
[354036.127897]  [<ffffffff81044b82>] ? mmput+0x56/0xf8
[354036.127901]  [<ffffffff81049d07>] ? exit_mm+0x117/0x122
[354036.127905]  [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
[354036.127909]  [<ffffffff81350487>] ? _raw_spin_lock_irq+0xa/0x14
[354036.127914]  [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
[354036.127918]  [<ffffffff81049f57>] ? do_exit+0x245/0x713
[354036.127922]  [<ffffffff81039a92>] ? finish_task_switch+0x4e/0xb9
[354036.127927]  [<ffffffff8134f0e9>] ? __schedule+0x5f9/0x610
[354036.127931]  [<ffffffff810d5c58>] ? do_munmap+0x2da/0x2f3
[354036.127935]  [<ffffffff8104a6a5>] ? do_group_exit+0x74/0x9e
[354036.128001]  [<ffffffff8104a6de>] ? sys_exit_group+0xf/0xf
[354036.128007]  [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b
[354044.085361] BUG: soft lockup - CPU#3 stuck for 23s! [apache2:20278]
[354044.085370] Modules linked in: evdev coretemp crc32c_intel
ghash_clmulni_intel snd_pcm snd_page_alloc aesni_intel snd_timer
aes_x86_64 snd aes_generic soundcore cryptd pcspkr ext4 crc16 jbd2
mbcache dm_mod xen_netfront xen_blkfront
[354044.085395] CPU 3
[354044.085397] Modules linked in: evdev coretemp crc32c_intel
ghash_clmulni_intel snd_pcm snd_page_alloc aesni_intel snd_timer
aes_x86_64 snd aes_generic soundcore cryptd pcspkr ext4 crc16 jbd2
mbcache dm_mod xen_netfront xen_blkfront
[354044.085417]
[354044.085420] Pid: 20278, comm: apache2 Not tainted 3.2.0-4-amd64 #1
Debian 3.2.60-1+deb7u3
[354044.085426] RIP: e030:[<ffffffff8100122a>]  [<ffffffff8100122a>]
hypercall_page+0x22a/0x1000
[354044.085436] RSP: e02b:ffff8802ef743c00  EFLAGS: 00000246
[354044.085440] RAX: 0000000000040001 RBX: ffff8802ffff0200 RCX:
ffffffff8100122a
[354044.085444] RDX: ffffffffffffffc8 RSI: 0000000000000000 RDI:
0000000000000000
[354044.085449] RBP: 000000000000000e R08: 0000000000000200 R09:
dead000000100100
[354044.085453] R10: dead000000200200 R11: 0000000000000246 R12:
ffff880265c53320
[354044.085457] R13: ffff8802ffffbe00 R14: ffffea0009cec350 R15:
000000000000000d
[354044.085466] FS:  00007f7b66cd2740(0000) GS:ffff8802ffcc0000(0000)
knlGS:0000000000000000
[354044.085471] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[354044.085475] CR2: 00007f7b6906cff0 CR3: 00000002efbfb000 CR4:
0000000000002660
[354044.085479] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[354044.085484] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[354044.085489] Process apache2 (pid: 20278, threadinfo
ffff8802ef742000, task ffff8802efa5b880)
[354044.085493] Stack:
[354044.085495]  0000000000000000 0000000000000200 ffffffff81006790
ffffffff81006d22
[354044.085502]  0000000000000000 dead000000200200 dead000000100100
0000000000000200
[354044.085508]  ffff8802ffff0200 ffff8802ffff0200 ffffffffffffffc8
0000000000000200
[354044.085746] Call Trace:

[354044.085753]  [<ffffffff81006790>] ? 
xen_force_evtchn_callback+0x9/0xa

[354044.085758]  [<ffffffff81006d22>] ? check_events+0x12/0x20

[354044.085763]  [<ffffffff81006d0f>] ? 
xen_restore_fl_direct_reloc+0x4/0x4

[354044.085769]  [<ffffffff81071153>] ? arch_local_irq_restore+0x7/0x8

[354044.085776]  [<ffffffff8135049f>] ? 
_raw_spin_unlock_irqrestore+0xe/0xf

[354044.085783]  [<ffffffff810be895>] ? release_pages+0xf4/0x14d

[354044.085790]  [<ffffffff810de78b>] ? 
free_pages_and_swap_cache+0x48/0x60

[354044.085796]  [<ffffffff810cf527>] ? tlb_flush_mmu+0x37/0x50
[354044.085800]  [<ffffffff810cf54c>] ? tlb_finish_mmu+0xc/0x31
[354044.085806]  [<ffffffff810d5e79>] ? exit_mmap+0xc4/0xe9
[354044.085811]  [<ffffffff81044b82>] ? mmput+0x56/0xf8
[354044.085815]  [<ffffffff81049d07>] ? exit_mm+0x117/0x122
[354044.085820]  [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
[354044.085824]  [<ffffffff81350487>] ? _raw_spin_lock_irq+0xa/0x14
[354044.085829]  [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
[354044.085833]  [<ffffffff81049f57>] ? do_exit+0x245/0x713
[354044.085838]  [<ffffffff81039a92>] ? finish_task_switch+0x4e/0xb9
[354044.085843]  [<ffffffff8134f0e9>] ? __schedule+0x5f9/0x610
[354044.085848]  [<ffffffff810d5c58>] ? do_munmap+0x2da/0x2f3
[354044.085852]  [<ffffffff8104a6a5>] ? do_group_exit+0x74/0x9e
[354044.085856]  [<ffffffff8104a6de>] ? sys_exit_group+0xf/0xf
[354044.085861]  [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b
[354044.085865] Code: cc 51 41 53 b8 10 00 00 00 0f 05 41 5b 59 c3 cc
cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 51 41 53 b8 11 00
00 00 0f 05 <41> 5b 59 c3 cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc
cc cc
[354044.085905] Call Trace:

[354044.085909]  [<ffffffff81006790>] ? 
xen_force_evtchn_callback+0x9/0xa

[354044.085914]  [<ffffffff81006d22>] ? check_events+0x12/0x20

[354044.085919]  [<ffffffff81006d0f>] ? 
xen_restore_fl_direct_reloc+0x4/0x4

[354044.085924]  [<ffffffff81071153>] ? arch_local_irq_restore+0x7/0x8

[354044.085928]  [<ffffffff8135049f>] ? 
_raw_spin_unlock_irqrestore+0xe/0xf

[354044.085933]  [<ffffffff810be895>] ? release_pages+0xf4/0x14d

[354044.085939]  [<ffffffff810de78b>] ? 
free_pages_and_swap_cache+0x48/0x60

[354044.085944]  [<ffffffff810cf527>] ? tlb_flush_mmu+0x37/0x50
[354044.085948]  [<ffffffff810cf54c>] ? tlb_finish_mmu+0xc/0x31
[354044.085953]  [<ffffffff810d5e79>] ? exit_mmap+0xc4/0xe9
[354044.085957]  [<ffffffff81044b82>] ? mmput+0x56/0xf8
[354044.085961]  [<ffffffff81049d07>] ? exit_mm+0x117/0x122
[354044.085965]  [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
[354044.085970]  [<ffffffff81350487>] ? _raw_spin_lock_irq+0xa/0x14
[354044.085974]  [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
[354044.085978]  [<ffffffff81049f57>] ? do_exit+0x245/0x713
[354044.085983]  [<ffffffff81039a92>] ? finish_task_switch+0x4e/0xb9
[354044.085987]  [<ffffffff8134f0e9>] ? __schedule+0x5f9/0x610
[354044.085992]  [<ffffffff810d5c58>] ? do_munmap+0x2da/0x2f3
[354044.085996]  [<ffffffff8104a6a5>] ? do_group_exit+0x74/0x9e
[354044.086000]  [<ffffffff8104a6de>] ? sys_exit_group+0xf/0xf
[354044.086004]  [<ffffffff81355452>] ? system_call_fastpath+0x16/0x1b

[354047.219103] INFO: rcu_sched detected stall on CPU 5 (t=15039 
jiffies)

[354047.224054] INFO: rcu_sched detected stalls on CPUs/tasks: { 5}
(detected by 1, t=15039 jiffies)
[354047.224054] sending NMI to all CPUs:

[354047.224054] BUG: unable to handle kernel paging request at 
ffffffffff5fb310

[354047.224054] IP: [<ffffffff81027b9a>] native_apic_mem_write+0x2/0x9
[354047.224054] PGD 1607067 PUD 1608067 PMD 172d067 PTE 0
[354047.224054] Oops: 0002 [#1] SMP
[354047.224054] CPU 1
[354047.224054] Modules linked in: evdev coretemp crc32c_intel
ghash_clmulni_intel snd_pcm snd_page_alloc aesni_intel snd_timer
aes_x86_64 snd aes_generic soundcore cryptd pcspkr ext4 crc16 jbd2
mbcache dm_mod xen_netfront xen_blkfront
[354047.224054]
[354047.224054] Pid: 24843, comm: apache2 Not tainted 3.2.0-4-amd64 #1
Debian 3.2.60-1+deb7u3
[354047.224054] RIP: e030:[<ffffffff81027b9a>]  [<ffffffff81027b9a>]
native_apic_mem_write+0x2/0x9
[354047.224054] RSP: e02b:ffff8802ffc43c90  EFLAGS: 00010086
[354047.224054] RAX: 0000000000000000 RBX: ffffffff816800e0 RCX:
00000000000007c2
[354047.224054] RDX: 0000000000000000 RSI: 00000000ff000000 RDI:
0000000000000310
[354047.224054] RBP: 0000000000000002 R08: 0000000000000002 R09:
0000000000000000
[354047.224054] R10: 0000000000000000 R11: ffff8802f2863d40 R12:
0000000000000800
[354047.224054] R13: 00000000000000ff R14: ffffffff81624400 R15:
0000000000000000
[354047.224054] FS:  00007f7b66cd2740(0000) GS:ffff8802ffc40000(0000)
knlGS:0000000000000000
[354047.224054] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[354047.224054] CR2: ffffffffff5fb310 CR3: 00000002f06c2000 CR4:
0000000000002660
[354047.224054] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[354047.224054] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[354047.224054] Process apache2 (pid: 24843, threadinfo
ffff8802ef78a000, task ffff8802f1476800)
[354047.224054] Stack:
[354047.224054]  ffffffff81027dd3 0000000000000000 0000000000002710
ffffffff81622300
[354047.224054]  ffffffff81622400 0000000000000001 ffffffff81024d16
ffff8802ffc4edc0
[354047.224054]  ffffffff81096308 0000000000000048 ffffffff81624400
ffffffff81098d57
[354047.224054] Call Trace:
[354047.224054]  <IRQ>
[354047.224054]  [<ffffffff81027dd3>] ? _flat_send_IPI_mask+0x4b/0x78

[354047.224054]  [<ffffffff81024d16>] ? 
arch_trigger_all_cpu_backtrace+0x4d/0x7b

[354047.224054]  [<ffffffff81096308>] ? __rcu_pending+0x21a/0x358
[354047.224054]  [<ffffffff81098d57>] ? arch_local_irq_restore+0x7/0x8
[354047.224054]  [<ffffffff8106c35c>] ? tick_nohz_handler+0xd0/0xd0
[354047.224054]  [<ffffffff81096773>] ? rcu_check_callbacks+0x90/0xcc
[354047.224054]  [<ffffffff81052cbe>] ? update_process_times+0x31/0x63
[354047.224054]  [<ffffffff8106c3c6>] ? tick_sched_timer+0x6a/0x90
[354047.224054]  [<ffffffff81062632>] ? __run_hrtimer+0xac/0x135
[354047.224054]  [<ffffffff81062d1c>] ? hrtimer_interrupt+0xd7/0x1b1
[354047.224054]  [<ffffffff8103b087>] ? check_preempt_curr+0x52/0x5f
[354047.224054]  [<ffffffff810068b9>] ? xen_timer_interrupt+0x28/0xfc
[354047.224054]  [<ffffffff81244771>] ? get_cycles+0x5/0x8

[354047.224054]  [<ffffffff8124561e>] ? 
add_interrupt_randomness+0x38/0x155
[354047.224054]  [<ffffffff8109124d>] ? 
handle_irq_event_percpu+0x50/0x17d

[354047.224054]  [<ffffffff8121ca3a>] ? disable_pirq+0x2/0x2
[354047.224054]  [<ffffffff8121c624>] ? info_for_irq+0x7/0x17
[354047.224054]  [<ffffffff81093847>] ? handle_percpu_irq+0x3a/0x4f

[354047.224054]  [<ffffffff8121c866>] ? 
__xen_evtchn_do_upcall+0xd3/0x287

[354047.224054]  [<ffffffff8104b780>] ? __local_bh_enable+0x40/0x77
[354047.224054]  [<ffffffff813576ac>] ? call_softirq+0x1c/0x30
[354047.224054]  [<ffffffff81095239>] ? arch_local_irq_save+0x11/0x15
[354047.224054]  [<ffffffff8121dd98>] ? xen_evtchn_do_upcall+0x22/0x32

[354047.224054]  [<ffffffff813576fe>] ? 
xen_do_hypervisor_callback+0x1e/0x30

[354047.224054]  <EOI>
[354047.224054]  [<ffffffff810be219>] ? add_page_to_lru_list+0x64/0x64
[354047.224054]  [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
[354047.224054]  [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000

[354047.224054]  [<ffffffff81006790>] ? 
xen_force_evtchn_callback+0x9/0xa

[354047.224054]  [<ffffffff81006d22>] ? check_events+0x12/0x20

[354047.224054]  [<ffffffff81006d0f>] ? 
xen_restore_fl_direct_reloc+0x4/0x4

[354047.224054]  [<ffffffff81071153>] ? arch_local_irq_restore+0x7/0x8

[354047.224054]  [<ffffffff8135049f>] ? 
_raw_spin_unlock_irqrestore+0xe/0xf

[354047.224054]  [<ffffffff810be97d>] ? pagevec_lru_move_fn+0x8f/0xb5
[354047.224054]  [<ffffffff810beb8a>] ? __lru_cache_add+0x4a/0x51
[354047.224054]  [<ffffffff810d1537>] ? handle_pte_fault+0x224/0x79f
[354047.224054]  [<ffffffff810ceacb>] ? pmd_val+0x7/0x8
[354047.224054]  [<ffffffff810ceb49>] ? pte_offset_kernel+0x16/0x35
[354047.224054]  [<ffffffff813533ee>] ? do_page_fault+0x320/0x345
[354047.224054]  [<ffffffff81003223>] ? xen_end_context_switch+0xe/0x1c

[354047.224054]  [<ffffffff81003ba5>] ? 
xen_mc_issue.constprop.23+0x31/0x49

[354047.224054]  [<ffffffff8100d750>] ? __switch_to+0x1e5/0x258
[354047.224054]  [<ffffffff81035bd7>] ? arch_local_irq_enable+0x7/0x8
[354047.224054]  [<ffffffff81039a92>] ? finish_task_switch+0x4e/0xb9
[354047.224054]  [<ffffffff8134f0e9>] ? __schedule+0x5f9/0x610
[354047.224054]  [<ffffffff813509f5>] ? page_fault+0x25/0x30
[354047.224054] Code: 00 74 18 48 8d 74 24 0c bf 1b 00 00 00 e8 ab fb
ff ff f6 c4 04 0f 95 c0 0f b6 c0 48 83 c4 10 c3 90 ff 14 25 d8 57 61
81 c3 89 ff <89> b7 00 b0 5f ff c3 89 ff 8b 87 00 b0 5f ff c3 48 8b 07
25 ff
[354047.224054] RIP  [<ffffffff81027b9a>] native_apic_mem_write+0x2/0x9
[354047.224054]  RSP <ffff8802ffc43c90>
[354047.224054] CR2: ffffffffff5fb310
[354047.224054] ---[ end trace 94d691dcc7253fa7 ]---

[354047.224054] Kernel panic - not syncing: Fatal exception in 
interrupt

[354047.224054] Pid: 24843, comm: apache2 Tainted: G      D
3.2.0-4-amd64 #1 Debian 3.2.60-1+deb7u3
[354047.224054] Call Trace:
[354047.224054]  <IRQ>  [<ffffffff81349af6>] ? panic+0x95/0x1a2
[354047.224054]  [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8

[354047.224054]  [<ffffffff8135049f>] ? 
_raw_spin_unlock_irqrestore+0xe/0xf

[354047.224054]  [<ffffffff81351286>] ? oops_end+0xa9/0xb6
[354047.224054]  [<ffffffff8134943f>] ? no_context+0x1ff/0x20e
[354047.224054]  [<ffffffff81348ccd>] ? pmd_val+0x7/0x8
[354047.224054]  [<ffffffff81348d0c>] ? pte_offset_kernel+0x16/0x35
[354047.224054]  [<ffffffff81353284>] ? do_page_fault+0x1b6/0x345
[354047.224054]  [<ffffffff811b2c7f>] ? vsnprintf+0x7c/0x427
[354047.224054]  [<ffffffff811b2c7f>] ? vsnprintf+0x7c/0x427

[354047.224054]  [<ffffffff8102bb7c>] ? 
pvclock_clocksource_read+0x42/0xb2

[354047.224054]  [<ffffffff811b3070>] ? sprintf+0x46/0x4b
[354047.224054]  [<ffffffff8107115b>] ? arch_local_irq_disable+0x7/0x8
[354047.224054]  [<ffffffff8107116d>] ? arch_local_irq_save+0x11/0x17
[354047.224054]  [<ffffffff81071153>] ? arch_local_irq_restore+0x7/0x8

[354047.224054]  [<ffffffff8135049f>] ? 
_raw_spin_unlock_irqrestore+0xe/0xf

[354047.224054]  [<ffffffff810637e0>] ? down_trylock+0x20/0x29
[354047.224054]  [<ffffffff813509f5>] ? page_fault+0x25/0x30
[354047.224054]  [<ffffffff81027b9a>] ? native_apic_mem_write+0x2/0x9
[354047.224054]  [<ffffffff81027dd3>] ? _flat_send_IPI_mask+0x4b/0x78

[354047.224054]  [<ffffffff81024d16>] ? 
arch_trigger_all_cpu_backtrace+0x4d/0x7b

[354047.224054]  [<ffffffff81096308>] ? __rcu_pending+0x21a/0x358
[354047.224054]  [<ffffffff81098d57>] ? arch_local_irq_restore+0x7/0x8
[354047.224054]  [<ffffffff8106c35c>] ? tick_nohz_handler+0xd0/0xd0
[354047.224054]  [<ffffffff81096773>] ? rcu_check_callbacks+0x90/0xcc
[354047.224054]  [<ffffffff81052cbe>] ? update_process_times+0x31/0x63
[354047.224054]  [<ffffffff8106c3c6>] ? tick_sched_timer+0x6a/0x90
[354047.224054]  [<ffffffff81062632>] ? __run_hrtimer+0xac/0x135
[354047.224054]  [<ffffffff81062d1c>] ? hrtimer_interrupt+0xd7/0x1b1
[354047.224054]  [<ffffffff8103b087>] ? check_preempt_curr+0x52/0x5f
[354047.224054]  [<ffffffff810068b9>] ? xen_timer_interrupt+0x28/0xfc
[354047.224054]  [<ffffffff81244771>] ? get_cycles+0x5/0x8

[354047.224054]  [<ffffffff8124561e>] ? 
add_interrupt_randomness+0x38/0x155
[354047.224054]  [<ffffffff8109124d>] ? 
handle_irq_event_percpu+0x50/0x17d

[354047.224054]  [<ffffffff8121ca3a>] ? disable_pirq+0x2/0x2
[354047.224054]  [<ffffffff8121c624>] ? info_for_irq+0x7/0x17
[354047.224054]  [<ffffffff81093847>] ? handle_percpu_irq+0x3a/0x4f

[354047.224054]  [<ffffffff8121c866>] ? 
__xen_evtchn_do_upcall+0xd3/0x287

[354047.224054]  [<ffffffff8104b780>] ? __local_bh_enable+0x40/0x77
[354047.224054]  [<ffffffff813576ac>] ? call_softirq+0x1c/0x30
[354047.224054]  [<ffffffff81095239>] ? arch_local_irq_save+0x11/0x15
[354047.224054]  [<ffffffff8121dd98>] ? xen_evtchn_do_upcall+0x22/0x32

[354047.224054]  [<ffffffff813576fe>] ? 
xen_do_hypervisor_callback+0x1e/0x30
[354047.224054]  <EOI>  [<ffffffff810be219>] ? 
add_page_to_lru_list+0x64/0x64

[354047.224054]  [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000
[354047.224054]  [<ffffffff8100122a>] ? hypercall_page+0x22a/0x1000

[354047.224054]  [<ffffffff81006790>] ? 
xen_force_evtchn_callback+0x9/0xa

[354047.224054]  [<ffffffff81006d22>] ? check_events+0x12/0x20

[354047.224054]  [<ffffffff81006d0f>] ? 
xen_restore_fl_direct_reloc+0x4/0x4

[354047.224054]  [<ffffffff81071153>] ? arch_local_irq_restore+0x7/0x8

[354047.224054]  [<ffffffff8135049f>] ? 
_raw_spin_unlock_irqrestore+0xe/0xf

[354047.224054]  [<ffffffff810be97d>] ? pagevec_lru_move_fn+0x8f/0xb5
[354047.224054]  [<ffffffff810beb8a>] ? __lru_cache_add+0x4a/0x51
[354047.224054]  [<ffffffff810d1537>] ? handle_pte_fault+0x224/0x79f
[354047.224054]  [<ffffffff810ceacb>] ? pmd_val+0x7/0x8
[354047.224054]  [<ffffffff810ceb49>] ? pte_offset_kernel+0x16/0x35
[354047.224054]  [<ffffffff813533ee>] ? do_page_fault+0x320/0x345
[354047.224054]  [<ffffffff81003223>] ? xen_end_context_switch+0xe/0x1c

[354047.224054]  [<ffffffff81003ba5>] ? 
xen_mc_issue.constprop.23+0x31/0x49

[354047.224054]  [<ffffffff8100d750>] ? __switch_to+0x1e5/0x258
[354047.224054]  [<ffffffff81035bd7>] ? arch_local_irq_enable+0x7/0x8
[354047.224054]  [<ffffffff81039a92>] ? finish_task_switch+0x4e/0xb9
[354047.224054]  [<ffffffff8134f0e9>] ? __schedule+0x5f9/0x610
[354047.224054]  [<ffffffff813509f5>] ? page_fault+0x25/0x30

Cheers,
 jonas



--
To UNSUBSCRIBE, email to debian-kernel-requ...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Archive: 
https://lists.debian.org/0922f41f39664ecc9935605183802...@imap.steindlberger.de























					Reply via email to