Hi,
I have an amd64 system that I am using as a router/firewall for my home
network running OpenBSD 6.6-stable with the latest syspatches. I have
been running it without problems in an IPv4-only configuration.
Recently, I decided I wanted to experiment with running a dual stack
IPv4 and IPv6 network. I think I have successfully added IPv6 to my
configuration which seemed to be functioning correctly to the best of my
knowledge, but I have started to encounter kernel panics that crash to
the ddb debugger after about 12 hours of uptime. I believe the panics
might be specifically occurring when the DHCPv6 lease expires or renews
on my external interface and the IPv6 routing table is updated. The
panics go away when I reverted back to my IPv4-only configuration so I
believe the panics are specific to IPv6. I have been troubleshooting and
researching and for the past week or so and I thought I would check to
see if some more experienced OpenBSD users might have some advice on how
to proceed with troubleshooting.
At the end of this email, I have included the relevant section from
/var/run/dmesg.boot related to one instance of the panic. It includes
the boot messages, panic message, trace, and the output of ps I ran in
ddb after the panic.
I did find a similar bug report in the mail archive for openbsd-bugs
that looks like it was unfortunately not resolved:
https://marc.info/?l=openbsd-bugs&m=152587044611044&w=2
Some more info about my configuration that I think may be relevant:
My ISP uses DHCPv6 prefix delegation to distribute IPv6 addresses so I
used dhcpcd from ports installed using pkg_add. I used rad for router
advertisements on my internal interfaces and I am using SLAAC to
auto-configure IPv6 addresses.
My system has 6 Intel NICs, em[0-5], and I also have 4 VLANs vlan[0-3]
on em5 that I am using for my wireless AP. em0 is the external interface
connected to my cable modem. em[1-5] are my internal interfaces. em[1-3]
are each connected to separate computers, em4 is unconnected, and em5
and vlan[0-3] are connected to my wireless AP.
The only other observation I had that might be related to the panic is
regarding the "ndp info overwritten" and "cannot forward src" lines in
my dmesg. The "ndp info overwritten" lines correspond to interfaces that
were connected to devices/hosts that powered on at the time of the panic
while I think that the "cannot forward src" messages correspond to em2
and em3 which were connected to computers that were powered off at the
time of the panic, but were previously powered on to test if they were
receiving IPv6 addresses. I am wondering if the computers being powered
off at the time of the DHCPv6 lease update is triggering the panic and I
think I might test this by only enabling IPv6 on em5 and vlan[0-3] which
are connected to my wireless AP is always on.
I would be happy to provide any other relevant info that I may have
omitted. Thanks for reading.
Ben
######
OpenBSD 6.6 (GENERIC.MP) #3: Thu Nov 21 03:20:01 MST 2019
r...@syspatch-66-amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4192743424 (3998MB)
avail mem = 4052963328 (3865MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x8d317000 (85 entries)
bios0: vendor American Megatrends Inc. version "5.12" date 07/08/2019
bios0: Protectli FW6
acpi0 at bios0: ACPI 6.1
acpi0: sleep states S0 S5
acpi0: tables DSDT FACP APIC FPDT MCFG SSDT FIDT SSDT HPET SSDT SSDT
UEFI SSDT LPIT WSMT SSDT SSDT SSDT SSDT DBGP DBG2 BGRT DMAR ASF!
acpi0: wakeup devices PS2K(S0) PS2M(S0) RP09(S0) PXSX(S0) RP10(S0)
PXSX(S0) RP11(S0) PXSX(S0) RP12(S0) PXSX(S0) RP13(S0) PXSX(S0) RP01(S0)
PXSX(S0) RP02(S0) PXSX(S0) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Celeron(R) CPU 3865U @ 1.80GHz, 1696.58 MHz, 06-8e-09
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,SMEP,ERMS,INVPCID,MPX,RDSEED,SMAP,CLFLUSHOPT,PT,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 24MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1.1.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Celeron(R) CPU 3865U @ 1.80GHz, 1696.06 MHz, 06-8e-09
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,SMEP,ERMS,INVPCID,MPX,RDSEED,SMAP,CLFLUSHOPT,PT,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 120 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 23999999 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (PEG0)
acpiprt2 at acpi0: bus -1 (PEG1)
acpiprt3 at acpi0: bus -1 (PEG2)
acpiprt4 at acpi0: bus -1 (RP09)
acpiprt5 at acpi0: bus -1 (RP10)
acpiprt6 at acpi0: bus -1 (RP11)
acpiprt7 at acpi0: bus -1 (RP12)
acpiprt8 at acpi0: bus -1 (RP13)
acpiprt9 at acpi0: bus 1 (RP01)
acpiprt10 at acpi0: bus 2 (RP02)
acpiprt11 at acpi0: bus 3 (RP03)
acpiprt12 at acpi0: bus 4 (RP04)
acpiprt13 at acpi0: bus 5 (RP05)
acpiprt14 at acpi0: bus 6 (RP06)
acpiprt15 at acpi0: bus -1 (RP07)
acpiprt16 at acpi0: bus -1 (RP08)
acpiprt17 at acpi0: bus -1 (RP17)
acpiprt18 at acpi0: bus -1 (RP18)
acpiprt19 at acpi0: bus -1 (RP19)
acpiprt20 at acpi0: bus -1 (RP20)
acpiprt21 at acpi0: bus -1 (RP21)
acpiprt22 at acpi0: bus -1 (RP22)
acpiprt23 at acpi0: bus -1 (RP23)
acpiprt24 at acpi0: bus -1 (RP24)
acpiprt25 at acpi0: bus -1 (RP14)
acpiprt26 at acpi0: bus -1 (RP15)
acpiprt27 at acpi0: bus -1 (RP16)
acpiec0 at acpi0: not present
acpicpu0 at acpi0: C3(200@1034 mwait.1@0x60), C2(200@151 mwait.1@0x33),
C1(1000@1 mwait.1), PSS
acpicpu1 at acpi0: C3(200@1034 mwait.1@0x60), C2(200@151 mwait.1@0x33),
C1(1000@1 mwait.1), PSS
acpipwrres0 at acpi0: WRST
acpipwrres1 at acpi0: WRST
acpipwrres2 at acpi0: WRST
acpipwrres3 at acpi0: WRST
acpipwrres4 at acpi0: WRST
acpipwrres5 at acpi0: WRST
acpipwrres6 at acpi0: WRST
acpipwrres7 at acpi0: WRST
acpipwrres8 at acpi0: WRST
acpipwrres9 at acpi0: WRST
acpipwrres10 at acpi0: WRST
acpipwrres11 at acpi0: WRST
acpipwrres12 at acpi0: WRST
acpipwrres13 at acpi0: WRST
acpipwrres14 at acpi0: WRST
acpipwrres15 at acpi0: WRST
acpipwrres16 at acpi0: WRST
acpipwrres17 at acpi0: WRST
acpipwrres18 at acpi0: WRST
acpipwrres19 at acpi0: WRST
acpipwrres20 at acpi0: FN00, resource for FAN0
acpipwrres21 at acpi0: FN01, resource for FAN1
acpipwrres22 at acpi0: FN02, resource for FAN2
acpipwrres23 at acpi0: FN03, resource for FAN3
acpipwrres24 at acpi0: FN04, resource for FAN4
acpitz0 at acpi0: critical temperature is 119 degC
acpitz1 at acpi0: critical temperature is 119 degC
acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
acpicmos0 at acpi0
acpibtn0 at acpi0: SLPB
"INT33A1" at acpi0 not configured
acpibtn1 at acpi0: PWRB
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
acpivideo0 at acpi0: GFX0
acpivout0 at acpivideo0: DD1F
cpu0: using VERW MDS workaround (except on vmm entry)
cpu0: Enhanced SpeedStep 1696 MHz: speeds: 1801, 1800, 1700, 1600, 1500,
1400, 1300, 1200, 1100, 1000, 900, 800, 700, 600, 500, 400 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Core 7G Host" rev 0x02
inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics 610" rev 0x02
drm0 at inteldrm0
inteldrm0: msi
xhci0 at pci0 dev 20 function 0 "Intel 100 Series xHCI" rev 0x21: msi,
xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev
3.00/1.00 addr 1
"Intel 100 Series MEI" rev 0x21 at pci0 dev 22 function 0 not configured
ahci0 at pci0 dev 23 function 0 "Intel 100 Series AHCI" rev 0x21: msi,
AHCI 1.3.1
ahci0: port 0: 6.0Gb/s
ahci0: PHY offline on port 1
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, TS64GMSA230S, R081> naa.57c3548176158942
sd0: 61057MB, 512 bytes/sector, 125045424 sectors, thin
ppb0 at pci0 dev 28 function 0 "Intel 100 Series PCIE" rev 0xf1: msi
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address
00:e0:67:18:19:42
ppb1 at pci0 dev 28 function 1 "Intel 100 Series PCIE" rev 0xf1: msi
pci2 at ppb1 bus 2
em1 at pci2 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address
00:e0:67:18:19:43
ppb2 at pci0 dev 28 function 2 "Intel 100 Series PCIE" rev 0xf1: msi
pci3 at ppb2 bus 3
em2 at pci3 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address
00:e0:67:18:19:44
ppb3 at pci0 dev 28 function 3 "Intel 100 Series PCIE" rev 0xf1: msi
pci4 at ppb3 bus 4
em3 at pci4 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address
00:e0:67:18:19:45
ppb4 at pci0 dev 28 function 4 "Intel 100 Series PCIE" rev 0xf1: msi
pci5 at ppb4 bus 5
em4 at pci5 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address
00:e0:67:18:19:46
ppb5 at pci0 dev 28 function 5 "Intel 100 Series PCIE" rev 0xf1: msi
pci6 at ppb5 bus 6
em5 at pci6 dev 0 function 0 "Intel 82583V" rev 0x00: msi, address
00:e0:67:18:19:47
pcib0 at pci0 dev 31 function 0 "Intel 200 Series LPC" rev 0x21
"Intel 100 Series PMC" rev 0x21 at pci0 dev 31 function 2 not configured
ichiic0 at pci0 dev 31 function 4 "Intel 100 Series SMBus" rev 0x21:
apic 2 int 16
iic0 at ichiic0
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0 mux 1
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
it0 at isa0 port 0x2e/2: IT8772F rev 1, EC port 0xa40
vmm0 at mainbus0: VMX/EPT
efifb at mainbus0 not configured
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (fc4381a8357fd1b0.a) swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
inteldrm0: 1024x768, 32bpp
wsdisplay0 at inteldrm0 mux 1
pckbd_enable: command error
wsdisplay0: screen 0-5 added (std, vt100 emulation)
ndp info overwritten for fe80:c::2e0:67ff:fe18:1947 by 00:e0:67:18:19:45
on vlan3
ndp info overwritten for fe80:6::2e0:67ff:fe18:1947 by 00:e0:67:18:19:45
on em5
ndp info overwritten for fe80:b::2e0:67ff:fe18:1947 by 00:e0:67:18:19:45
on vlan2
ndp info overwritten for fe80:a::2e0:67ff:fe18:1947 by 00:e0:67:18:19:45
on vlan1
ndp info overwritten for fe80:9::2e0:67ff:fe18:1947 by 00:e0:67:18:19:45
on vlan0
ndp info overwritten for fe80:2::2e0:67ff:fe18:1943 by 00:e0:67:18:19:45
on em1
cannot forward src fe80:a::5660:9ff:fedd:cca8, dst 2001:4860:4860::6464,
nxt 17, rcvif 10, outif 1
cannot forward src fe80:a::5660:9ff:fedd:cca8, dst 2001:4860:4860::6464,
nxt 17, rcvif 10, outif 1
panic: kernel diagnostic assertion "!ISSET(rt->rt_flags, RTF_LOCAL)"
failed: file "/usr/src/sys/netinet6/nd6.c", line 727
Stopped at db_enter+0x10: popq %rbp
TID PID UID PRFLAGS PFLAGS CPU COMMAND
* 12172 56020 0 0x14000 0x40000200 0K softclock
db_enter() at db_enter+0x10
panic() at panic+0x128
__assert(ffffffff81ca1bdf,ffffffff81cac63d,2d7,ffffffff81c877a4) at
__assert+0x2b
nd6_free(fffffd815eadb710) at nd6_free+0x12f
nd6_llinfo_timer(fffffd815eadb710) at nd6_llinfo_timer+0x19e
nd6_timer(ffffffff81ff7d50) at nd6_timer+0x64
softclock_thread(ffff8000ffffe000) at softclock_thread+0xfb
end trace frame: 0x0, count: 8
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports. Insufficient info makes it difficult to find and fix bugs.
ddb{0}> ps
PID TID PPID UID S FLAGS WAIT COMMAND
71101 370210 81160 94 3 0x100092 kqread rad
36663 233150 81160 94 3 0x100092 kqread rad
81160 34274 1 0 3 0x100080 kqread rad
8607 115278 1 0 3 0x100083 ttyin getty
92088 384510 1 0 3 0x100083 ttyin getty
99883 499812 1 0 3 0x100083 ttyin getty
26478 124491 1 0 3 0x100083 ttyin getty
69363 154114 1 0 3 0x100083 ttyin getty
86060 293146 1 0 3 0x100083 ttyin getty
37066 4797 1 0 3 0x100098 poll cron
54502 514469 1 0 3 0x80 kqread apmd
77136 522236 1 0 3 0x80 kqread dhcpcd
65621 515668 1 629 3 0x90 poll avahi-daemon
6344 170264 1 572 3 0x90 poll dbus-daemon
72501 157373 76033 95 3 0x100092 kqread smtpd
46629 436753 76033 103 3 0x100092 kqread smtpd
56768 98112 76033 95 3 0x100092 kqread smtpd
8204 106837 76033 95 3 0x100092 kqread smtpd
62327 125479 76033 95 3 0x100092 kqread smtpd
34358 470816 76033 95 3 0x100092 kqread smtpd
76033 7544 1 0 3 0x100080 kqread smtpd
61841 392491 1 77 3 0x100090 poll dhcpd
8345 285284 1 0 3 0x80 select sshd
66534 22989 1 0 3 0x100080 poll ntpd
6894 217071 30254 83 3 0x100092 poll ntpd
30254 338166 1 83 3 0x100092 poll ntpd
82791 382379 1 53 3 0x90 kqread unbound
57461 8270 54858 74 3 0x100092 bpf pflogd
54858 522464 1 0 3 0x80 netio pflogd
26929 242333 57577 73 3 0x100090 kqread syslogd
57577 493915 1 0 3 0x100082 netio syslogd
60733 317951 1 77 3 0x100090 poll dhclient
3305 473407 1 0 3 0x80 poll dhclient
51834 334199 35318 115 3 0x100092 kqread slaacd
47429 282699 35318 115 3 0x100092 kqread slaacd
35318 517851 1 0 3 0x100080 kqread slaacd
99085 420269 0 0 3 0x14280 schto i915/signal:6
33798 484525 0 0 3 0x14280 schto i915/signal:2
25861 170194 0 0 3 0x14280 schto i915/signal:1
76675 337056 0 0 3 0x14280 schto i915/signal:0
94463 253283 0 0 3 0x14200 bored i915-userptr-acq
89440 298121 0 0 3 0x14200 bored i915_modeset
43276 342976 0 0 3 0x14200 bored i915-dp
33409 16254 0 0 3 0x14200 bored i915
33089 228516 0 0 3 0x14200 pgzero zerothread
86710 349325 0 0 3 0x14200 aiodoned aiodoned
50212 84906 0 0 3 0x14200 syncer update
83345 246040 0 0 3 0x14200 cleaner cleaner
97789 396968 0 0 3 0x14200 reaper reaper
17359 153556 0 0 3 0x14200 pgdaemon pagedaemon
9948 234580 0 0 3 0x14200 bored crynlk
81030 332928 0 0 3 0x14200 bored crypto
51819 191268 0 0 3 0x14200 usbtsk usbtask
32163 214038 0 0 3 0x14200 usbatsk usbatsk
59871 370750 0 0 3 0x14200 bored drmtskl
80945 102084 0 0 3 0x14200 bored drmlwq
85907 159437 0 0 3 0x14200 bored drmlwq
3277 260402 0 0 3 0x14200 bored drmlwq
40689 82336 0 0 3 0x14200 bored drmubwq
71293 514738 0 0 3 0x14200 bored drmubwq
15998 134631 0 0 3 0x14200 bored drmubwq
71439 227948 0 0 3 0x14200 bored drmubwq
65395 170820 0 0 3 0x14200 bored drmwq
12666 420844 0 0 3 0x14200 bored drmwq
91628 417254 0 0 3 0x14200 bored drmwq
50050 283919 0 0 3 0x14200 bored drmwq
48545 214698 0 0 3 0x40014200 acpi0 acpi0
89352 316255 0 0 7 0x40014200 idle1
45452 317644 0 0 3 0x14200 bored sensors
42779 145809 0 0 3 0x14200 bored softnet
82778 378842 0 0 3 0x14200 bored systqmp
86243 366424 0 0 3 0x14200 bored systq
*56020 12172 0 0 7 0x40014200 softclock
86139 86337 0 0 3 0x40014200 idle0
38007 388788 0 0 3 0x14200 bored smr
1 518657 0 0 3 0x82 wait init
0 0 -1 0 3 0x10200 scheduler swapper
