Hello,
A few days ago, one of our routers (running Linux 4.13.9) crashed due
to a general protection fault in dst_destroy(). At the time, it had run
for several weeks without any problems, but then crashed three times in
a row within a few minutes - all due to a general protection fault at
dst_destroy()+0x35. Since then, it has run for several days without any
further problems, so I suspect that this was triggered by a traffic
pattern in the routed packets, but I don't have a way to reproduce it.
Disassembly shows that this is in the inlined dev_put(), which does
this_cpu_dec(*dev->pcpu_refcnt). As far as I can tell there haven't
been any fixes in this area since 4.13, and a Google search didn't find
anything recent, so I'm guessing this is not a known problem.
I have included the kernel output via serial console below as well as
gdb and objdump information. Please let me know, if I can provide any
additional information.
[2024260.461401] general protection fault: 0000 [#1] SMP
[2024260.467193] Modules linked in:
[2024260.470897] CPU: 15 PID: 0 Comm: swapper/15 Tainted: G W
4.13.9 #2
[2024260.479488] Hardware name: Dell Inc. PowerEdge R730/0H21J3, BIOS 2.5.5
08/16/2017
[2024260.488279] task: ffff88085b625cc0 task.stack: ffffc900000e4000
[2024260.495277] RIP: 0010:dst_destroy+0x35/0xa0
[2024260.500277] RSP: 0018:ffff88085f5c3f08 EFLAGS: 00010286
[2024260.506474] RAX: ffff88085ac0e880 RBX: ffff88082cf9fb00 RCX:
0000000000000020
[2024260.514868] RDX: ffff88082cf9fbc0 RSI: ffffffffffffffff RDI:
ffffffff816786c0
[2024260.523258] RBP: 0000000000000000 R08: ffffffffffffff00 R09:
0000000000000000
[2024260.531649] R10: 0000000000000000 R11: 0000000000000000 R12:
ffff88085f5da678
[2024260.540040] R13: 000000000000000a R14: ffff88085b625cc0 R15:
ffff88085b625cc0
[2024260.548431] FS: 0000000000000000(0000) GS:ffff88085f5c0000(0000)
knlGS:0000000000000000
[2024260.557924] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[2024260.564719] CR2: 00007fc800e48e88 CR3: 0000000001809000 CR4:
00000000001406e0
[2024260.573112] Call Trace:
[2024260.576113] <IRQ>
[2024260.578618] ? rcu_process_callbacks+0x18f/0x460
[2024260.584126] ? rebalance_domains+0xe2/0x290
[2024260.589128] ? __do_softirq+0x100/0x292
[2024260.593727] ? irq_exit+0x92/0xa0
[2024260.597729] ? smp_apic_timer_interrupt+0x39/0x50
[2024260.603328] ? apic_timer_interrupt+0x7c/0x90
[2024260.608528] </IRQ>
[2024260.611134] ? cpuidle_enter_state+0x14c/0x2b0
[2024260.616432] ? cpuidle_enter_state+0x128/0x2b0
[2024260.621731] ? do_idle+0xf9/0x190
[2024260.625733] ? cpu_startup_entry+0x5f/0x70
[2024260.630636] ? start_secondary+0x12a/0x130
[2024260.635536] ? secondary_startup_64+0x9f/0x9f
[2024260.640731] Code: f6 47 60 08 48 8b 6f 18 74 62 48 8b 43 20 48 8b 40 30 48
85 c0 74 05 48
89 df ff d0 48 8b 03 48 85 c0 74 0a 48 8b 80 e0 03 00 00 <65> ff 08 f6 43 60 80
74 26 48 8d bb
e0 00 00 00 e8 e6 7f 01 00
[2024260.662626] RIP: dst_destroy+0x35/0xa0 RSP: ffff88085f5c3f08
[2024260.669333] ---[ end trace 3c1827251806827c ]---
[2024260.724173] Kernel panic - not syncing: Fatal exception in interrupt
[2024261.102792] Kernel Offset: disabled
[2024261.156022] Rebooting in 60 seconds..
[2024321.167958] ACPI MEMORY or I/O RESET_REG.
[ 36.620034] general protection fault: 0000 [#1] SMP
[ 36.625637] Modules linked in:
[ 36.629141] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.13.9 #2
[ 36.635938] Hardware name: Dell Inc. PowerEdge R730/0H21J3, BIOS 2.5.5
08/16/2017
[ 36.644532] task: ffff88085b46a7c0 task.stack: ffffc9000007c000
[ 36.651333] RIP: 0010:dst_destroy+0x35/0xa0
[ 36.656133] RSP: 0018:ffff88085f283f08 EFLAGS: 00010286
[ 36.662133] RAX: 2e37307830203a65 RBX: ffff88082ac10000 RCX: 0000000000000020
[ 36.670326] RDX: ffff88082ac100c0 RSI: ffffffffffffffff RDI: ffffffff816786c0
[ 36.678521] RBP: 0000000000000000 R08: 0000000030e3e201 R09: 000000010080007a
[ 36.686714] R10: ffff88085f283e20 R11: ffffea0020c38e00 R12: ffff88085f29a678
[ 36.694906] R13: 000000000000000a R14: ffff88085b46a7c0 R15: ffff88085b46a7c0
[ 36.703102] FS: 0000000000000000(0000) GS:ffff88085f280000(0000)
knlGS:0000000000000000
[ 36.712395] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 36.718992] CR2: 000055568c725558 CR3: 0000000001809000 CR4: 00000000001406e0
[ 36.727184] Call Trace:
[ 36.729987] <IRQ>
[ 36.732287] ? rcu_process_callbacks+0x18f/0x460
[ 36.737588] ? rebalance_domains+0xe2/0x290
[ 36.742388] ? __do_softirq+0x100/0x292
[ 36.746790] ? irq_exit+0x92/0xa0
[ 36.750590] ? smp_apic_timer_interrupt+0x39/0x50
[ 36.755990] ? apic_timer_interrupt+0x7c/0x90
[ 36.760987] </IRQ>
[ 36.763392] ? poll_idle+0x46/0x7a
[ 36.767295] ? cpuidle_enter_state+0x102/0x2b0
[ 36.772396] ? do_idle+0xf9/0x190
[ 36.776197] ? cpu_startup_entry+0x5f/0x70
[ 36.780892] ? start_secondary+0x12a/0x130
[ 36.785592] ? secondary_startup_64+0x9f/0x9f
[ 36.790590] Code: f6 47 60 08 48 8b 6f 18 74 62 48 8b 43 20 48 8b 40 30 48
85 c0 74 05 48 89 df ff d0 48 8b 03 48 85 c0 74 0a 48 8b 80 e0 03 00 00 <65> ff
08 f6 43 60 80 74 26 48 8d bb e0 00 00 00 e8 e6 7f 01 00
[ 36.812257] RIP: dst_destroy+0x35/0xa0 RSP: ffff88085f283f08
[ 36.818754] BUG: unable to handle kernel paging request at 0000000000006f6c
[ 36.818867] ---[ end trace 414dfe768dd8d21f ]---
[ 36.869815] Kernel panic - not syncing: Fatal exception in interrupt
[ 36.886771] IP: kmem_cache_alloc+0x4a/0x130
[ 36.891570] PGD 0
[ 36.891570] P4D 0
[ 36.893874]
[ 36.897884] Oops: 0000 [#2] SMP
[ 36.901485] Modules linked in:
[ 36.904987] CPU: 15 PID: 0 Comm: swapper/15 Tainted: G D 4.13.9
#2
[ 36.913378] Hardware name: Dell Inc. PowerEdge R730/0H21J3, BIOS 2.5.5
08/16/2017
[ 36.921966] task: ffff88085b625cc0 task.stack: ffffc900000e4000
[ 36.935345] RIP: 0010:kmem_cache_alloc+0x4a/0x130
[ 36.947416] RSP: 0018:ffff88085f5c3bc0 EFLAGS: 00010206
[ 36.960096] RAX: 0000000000000000 RBX: ffffffff818bc2c0 RCX: 0000000000006c21
[ 36.974987] RDX: 0000000000006c20 RSI: 0000000001080020 RDI: ffff88085ae21900
[ 36.989832] RBP: 0000000000006f6c R08: 000000000001f590 R09: 0000000000000000
[ 37.004610] R10: ffff88085578b800 R11: 0000000100000000 R12: ffffffff814a5054
[ 37.019296] R13: 0000000001080020 R14: 00000000ffffffff R15: ffff88085ae21900
[ 37.033923] FS: 0000000000000000(0000) GS:ffff88085f5c0000(0000)
knlGS:0000000000000000
[ 37.049688] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 37.062841] CR2: 0000000000006f6c CR3: 0000000001809000 CR4: 00000000001406e0
[ 37.077577] Call Trace:
[ 37.086989] <IRQ>
[ 37.095745] ? dst_alloc+0x44/0x80
[ 37.106068] ? rt_dst_alloc+0x54/0xf0
[ 37.116613] ? ip_route_input_rcu+0x586/0x9d0
[ 37.127919] ? ip_finish_output2+0x132/0x2f0
[ 37.138982] ? ip_route_input_noref+0x14/0x20
[ 37.150099] ? ip_rcv_finish+0x63/0x330
[ 37.160505] ? ip_rcv+0x249/0x350
[ 37.170360] ? inet_del_offload+0x40/0x40
[ 37.180967] ? __netif_receive_skb_core+0x3f6/0x790
[ 37.192598] ? netif_receive_skb_internal+0x2d/0x3b0
[ 37.204364] ? napi_gro_receive+0xbc/0xe0
[ 37.215034] ? i40e_napi_poll+0x8f0/0x1670
[ 37.225819] ? net_rx_action+0x1d2/0x300
[ 37.236387] ? __do_softirq+0x100/0x292
[ 37.246814] ? irq_exit+0x92/0xa0
[ 37.256678] ? do_IRQ+0x4a/0xc0
[ 37.266294] ? common_interrupt+0x7c/0x7c
[ 37.276935] </IRQ>
[ 37.285357] ? cpuidle_enter_state+0x14c/0x2b0
[ 37.296490] ? cpuidle_enter_state+0x128/0x2b0
[ 37.307569] ? do_idle+0xf9/0x190
[ 37.317180] ? cpu_startup_entry+0x5f/0x70
[ 37.327524] ? start_secondary+0x12a/0x130
[ 37.337751] ? secondary_startup_64+0x9f/0x9f
[ 37.348158] Code: 01 01 00 00 49 8b 0f 65 48 8b 51 08 65 48 03 0d 75 0f e8
7e 48 8b 29 48 85 ed 0f 84 aa 00 00 00 49 63 47 20 48 8d 4a 01 4d 8b 07 <48> 8b
5c 05 00 48 89 e8 65 49 0f c7 08 0f 94 c0 84 c0 74 c5 49
[ 37.380985] RIP: kmem_cache_alloc+0x4a/0x130 RSP: ffff88085f5c3bc0
[ 37.393786] CR2: 0000000000006f6c
[ 37.403350] ---[ end trace 414dfe768dd8d220 ]---
[ 37.925925] Shutting down cpus with NMI
[ 38.138172] Kernel Offset: disabled
[ 38.185771] Rebooting in 60 seconds..
[ 98.203395] ACPI MEMORY or I/O RESET_REG.
[ 62.670029] general protection fault: 0000 [#1] SMP
[ 62.675630] Modules linked in:
[ 62.679134] CPU: 11 PID: 0 Comm: swapper/11 Not tainted 4.13.9 #2
[ 62.686128] Hardware name: Dell Inc. PowerEdge R730/0H21J3, BIOS 2.5.5
08/16/2017
[ 62.694719] task: ffff88085b6227c0 task.stack: ffffc900000c4000
[ 62.701517] RIP: 0010:dst_destroy+0x35/0xa0
[ 62.706317] RSP: 0018:ffff88085f4c3f08 EFLAGS: 00010286
[ 62.712313] RAX: 00090700000003ff RBX: ffff8807eec57000 RCX: 0000000000000020
[ 62.720497] RDX: ffff8807eec570c0 RSI: ffffffffffffffff RDI: ffffffff816786c0
[ 62.728689] RBP: 0000000000000000 R08: 00000000eee30801 R09: 0000000180800051
[ 62.736878] R10: ffff88085f4c3e20 R11: ffffea001fbb8c00 R12: ffff88085f4da678
[ 62.745063] R13: 000000000000000a R14: ffff88085b6227c0 R15: ffff88085b6227c0
[ 62.753254] FS: 0000000000000000(0000) GS:ffff88085f4c0000(0000)
knlGS:0000000000000000
[ 62.762543] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 62.769141] CR2: 000055671ac3c3d8 CR3: 0000000001809000 CR4: 00000000001406e0
[ 62.777334] Call Trace:
[ 62.780126] <IRQ>
[ 62.782434] ? rcu_process_callbacks+0x18f/0x460
[ 62.787733] ? rebalance_domains+0xe2/0x290
[ 62.792533] ? __do_softirq+0x100/0x292
[ 62.796933] ? irq_exit+0x92/0xa0
[ 62.800733] ? smp_apic_timer_interrupt+0x39/0x50
[ 62.806133] ? apic_timer_interrupt+0x7c/0x90
[ 62.811128] </IRQ>
[ 62.813532] ? poll_idle+0x43/0x7a
[ 62.817434] ? cpuidle_enter_state+0x102/0x2b0
[ 62.822532] ? do_idle+0xf9/0x190
[ 62.826333] ? cpu_startup_entry+0x5f/0x70
[ 62.831033] ? start_secondary+0x12a/0x130
[ 62.835733] ? secondary_startup_64+0x9f/0x9f
[ 62.840730] Code: f6 47 60 08 48 8b 6f 18 74 62 48 8b 43 20 48 8b 40 30 48
85 c0 74 05 48 89 df ff d0 48 8b 03 48 85 c0 74 0a 48 8b 80 e0 03 00 00 <65> ff
08 f6 43 60 80 74 26 48 8d bb e0 00 00 00 e8 e6 7f 01 00
[ 62.862386] RIP: dst_destroy+0x35/0xa0 RSP: ffff88085f4c3f08
[ 62.868893] ---[ end trace 9976b4b318e1acec ]---
[ 62.877621] Kernel panic - not syncing: Fatal exception in interrupt
[ 63.182165] Kernel Offset: disabled
[ 63.188730] Rebooting in 60 seconds..
[ 123.199492] ACPI MEMORY or I/O RESET_REG.4
(gdb) l *dst_destroy+0x35
0xffffffff814a5955 is in dst_destroy (./include/linux/netdevice.h:3342).
3337 *
3338 * Release reference to device to allow it to be freed.
3339 */
3340 static inline void dev_put(struct net_device *dev)
3341 {
3342 this_cpu_dec(*dev->pcpu_refcnt);
3343 }
3344
3345 /**
3346 * dev_hold - get reference to device
# objdump -r -S -l --disassemble net/core/dst.o
...
struct dst_entry *dst_destroy(struct dst_entry * dst)
{
400: e8 00 00 00 00 callq 405 <dst_destroy+0x5>
401: R_X86_64_PC32 __fentry__-0x4
405: 55 push %rbp
406: 53 push %rbx
407: 48 89 fb mov %rdi,%rbx
/usr/src/linux/net/core/dst.c:125
smp_rmb();
child = dst->child;
if (!(dst->flags & DST_NOCOUNT))
40a: f6 47 60 08 testb $0x8,0x60(%rdi)
/usr/src/linux/net/core/dst.c:123
{
struct dst_entry *child;
smp_rmb();
child = dst->child;
40e: 48 8b 6f 18 mov 0x18(%rdi),%rbp
/usr/src/linux/net/core/dst.c:125
if (!(dst->flags & DST_NOCOUNT))
412: 74 62 je 476 <dst_destroy+0x76>
/usr/src/linux/net/core/dst.c:128
dst_entries_add(dst->ops, -1);
if (dst->ops->destroy)
414: 48 8b 43 20 mov 0x20(%rbx),%rax
418: 48 8b 40 30 mov 0x30(%rax),%rax
41c: 48 85 c0 test %rax,%rax
41f: 74 05 je 426 <dst_destroy+0x26>
/usr/src/linux/net/core/dst.c:129
dst->ops->destroy(dst);
421: 48 89 df mov %rbx,%rdi
424: ff d0 callq *%rax
/usr/src/linux/net/core/dst.c:130
if (dst->dev)
426: 48 8b 03 mov (%rbx),%rax
429: 48 85 c0 test %rax,%rax
42c: 74 0a je 438 <dst_destroy+0x38>
42e: 48 8b 80 e0 03 00 00 mov 0x3e0(%rax),%rax
dev_put():
/usr/src/linux/./include/linux/netdevice.h:3342
435: 65 ff 08 decl %gs:(%rax)
dst_destroy():
/usr/src/linux/net/core/dst.c:135
dev_put(dst->dev);
lwtstate_put(dst->lwtstate);
if (dst->flags & DST_METADATA)
438: f6 43 60 80 testb $0x80,0x60(%rbx)
43c: 74 26 je 464 <dst_destroy+0x64>
metadata_dst_free():
/usr/src/linux/net/core/dst.c:302
EXPORT_SYMBOL_GPL(metadata_dst_alloc);
void metadata_dst_free(struct metadata_dst *md_dst)
{
#ifdef CONFIG_DST_CACHE
dst_cache_destroy(&md_dst->u.tun_info.dst_cache);
43e: 48 8d bb e0 00 00 00 lea 0xe0(%rbx),%rdi
445: e8 00 00 00 00 callq 44a <dst_destroy+0x4a>
446: R_X86_64_PC32 dst_cache_destroy-0x4
/usr/src/linux/net/core/dst.c:304
#endif
kfree(md_dst);
44a: 48 89 df mov %rbx,%rdi
44d: e8 00 00 00 00 callq 452 <dst_destroy+0x52>
44e: R_X86_64_PC32 kfree-0x4
dst_destroy():
/usr/src/linux/net/core/dst.c:141
metadata_dst_free((struct metadata_dst *)dst);
else
kmem_cache_free(dst->ops->kmem_cachep, dst);
dst = child;
if (dst)
452: 48 85 ed test %rbp,%rbp
455: 74 08 je 45f <dst_destroy+0x5f>
/usr/src/linux/net/core/dst.c:142
dst_release_immediate(dst);
457: 48 89 ef mov %rbp,%rdi
45a: e8 00 00 00 00 callq 45f <dst_destroy+0x5f>
45b: R_X86_64_PC32 dst_release_immediate-0x4
/usr/src/linux/net/core/dst.c:144
return NULL;
}
45f: 31 c0 xor %eax,%eax
461: 5b pop %rbx
462: 5d pop %rbp
463: c3 retq
/usr/src/linux/net/core/dst.c:138
lwtstate_put(dst->lwtstate);
Regards,
Anders
