I am unfortunately still experiencing this, but managed to get some
extra info that might be helpful. I am still trying to get a ddb
trace, the problem is I need to have a monitor connected otherwise I
cannot get a trace after the fact (the machine is normally headless),
and even when that is the case I can't get ddb to respond at all - the
machine is completely hung. I will keep trying, but I did manage to
figure out how to reproduce the issue.
I have a suspicion this is could be related to pf. Ports 22, 80 and
443 are exposed to the outside world over a Wireguard tunnel. pf is
handling this with the following ruleset (this is the complete
/etc/pf.conf).
ext_if='re0'
set skip on lo
set limit table-entries 400000
table <pfbadhost> persist file "/etc/pf-badhost.txt"
table <ssh-annoyances> persist
block return # block stateless traffic
pass # establish keep-state
# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010
# Port build user does not need network
block return out log proto {tcp udp} user _pbuild
block in quick from <ssh-annoyances>
block in quick from <pfbadhost>
block out quick to <pfbadhost>
pass in on wg0 proto tcp from any to <redacted> port 80 reply-to
<wireguard-endpoint-redacted> keep state
pass in on wg0 proto tcp from any to <redacted> port 443 reply-to
<wireguard-endpoint-redacted> keep state
pass in on wg0 proto tcp from any to <redacted> port 22 reply-to
<wireguard-endpoint-redacted> flags S/SA keep state \
(max-src-conn 5, max-src-conn-rate 5/3, overload <ssh-annoyances> flush global)
The machine will crash reliably (sometimes within minutes, but usually
after a couple of hours) with the wireguard tunnel enabled. However,
the machine did run for 5 days at a time without the Wireguard tunnel
up and running (accessing only from the LAN side), and crashed within
about an hour of me brining up the Wireguard tunnel again.
Does anyone have suggestions? If need be I can bring the machine up to
running current, and/or compile a custom kernel with any debug options
enabled that might be helpful.
Thanks.
On Sun, 22 Oct 2023 at 21:30, Ashton Fagg <ash...@fagg.id.au> wrote:
>
> >Synopsis: amd64 system kernel panicking multiple times per hour - appears
> >networking related
> >Category: bug
> >Environment:
> System : OpenBSD 7.4
> Details : OpenBSD 7.4 (GENERIC.MP) #1397: Tue Oct 10 09:02:37 MDT 2023
> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> Architecture: OpenBSD.amd64
> Machine : amd64
> >Description:
>
> System kernel panics with the following error:
>
> panic sbdrop
> db_enter() at db_enter+0x14
> panic(ffffffff820d627c) at panic+0xc3
> sbdrop(fffffd83b292f990,fffffd83b292fab8, 7f) at sbdrop+0x23f
> tcp_input(fff8000180fb488, fff8000180fb494, 6, 2)) at tcp_input+0x2416
> ip_deliver(fff8000180fb488, fff8000180fb494, 6, 2) at ip_deliver+0x113
> ipintr() at ipintr+0x69
> if_netisr(0) at if_netisr+0xc0
> taskq_thread(ffff80000002f200) at taskq_thread+0x100
>
> Screen picture here:
>
> https://ibb.co/Lrrqd3S
>
> Unfortunately ddb would not respond so I was unable to capture the
> full trace for the other 3 CPUs. I have rebooted the machine and will
> try again next time this happens.
>
> >How-To-Repeat:
>
> Currently this happens again within about 20 mins of bringing the
> machine back online. This machine has been running fine on 7.3, but
> after upgrading to 7.4 yesterday this has been happening consistently.
>
> >Fix:
>
> Unsure.
>
> dmesg:
> OpenBSD 7.4 (GENERIC.MP) #1397: Tue Oct 10 09:02:37 MDT 2023
> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 14903021568 (14212MB)
> avail mem = 14431617024 (13763MB)
> random: good seed from bootblocks
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 3.3 @ 0xe6cc0 (32 entries)
> bios0: vendor American Megatrends Inc. version "P4.50" date 11/04/2020
> bios0: ASRock B450 Pro4
> efi0 at bios0: UEFI 2.7
> efi0: American Megatrends rev 0x50011
> acpi0 at bios0: ACPI 6.0
> acpi0: sleep states S0 S3 S4 S5
> acpi0: tables DSDT FACP SSDT SSDT SSDT FIDT MCFG AAFT HPET BGRT SSDT
> CRAT CDIT SSDT SSDT WSMT APIC SSDT FPDT
> acpi0: wakeup devices GPP0(S4) GPP2(S4) GPP3(S4) GPP4(S4) GPP5(S4)
> GPP6(S4) GP17(S4) XHC0(S4) XHC1(S4) GP18(S4) GPP1(S4) PTXH(S4)
> acpitimer0 at acpi0: 3579545 Hz, 32 bits
> acpimcfg0 at acpi0
> acpimcfg0: addr 0xf0000000, bus 0-127
> acpihpet0 at acpi0: 14318180 Hz
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: AMD Ryzen 3 3200G with Radeon Vega Graphics, 3600.01 MHz,
> 17-18-01, patch 08108109
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,HWPSTATE,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
> cpu0: 32KB 64b/line 8-way D-cache, 64KB 64b/line 4-way I-cache, 512KB
> 64b/line 8-way L2 cache, 4MB 64b/line 16-way L3 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
> cpu0: apic clock running at 25MHz
> cpu0: mwait min=64, max=64, C-substates=1.1, IBE
> cpu1 at mainbus0: apid 1 (application processor)
> cpu1: AMD Ryzen 3 3200G with Radeon Vega Graphics, 3600.00 MHz,
> 17-18-01, patch 08108109
> cpu1:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,HWPSTATE,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
> cpu1: 32KB 64b/line 8-way D-cache, 64KB 64b/line 4-way I-cache, 512KB
> 64b/line 8-way L2 cache, 4MB 64b/line 16-way L3 cache
> cpu1: smt 0, core 1, package 0
> cpu2 at mainbus0: apid 2 (application processor)
> cpu2: AMD Ryzen 3 3200G with Radeon Vega Graphics, 3600.00 MHz,
> 17-18-01, patch 08108109
> cpu2:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,HWPSTATE,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
> cpu2: 32KB 64b/line 8-way D-cache, 64KB 64b/line 4-way I-cache, 512KB
> 64b/line 8-way L2 cache, 4MB 64b/line 16-way L3 cache
> cpu2: smt 0, core 2, package 0
> cpu3 at mainbus0: apid 3 (application processor)
> cpu3: AMD Ryzen 3 3200G with Radeon Vega Graphics, 3600.00 MHz,
> 17-18-01, patch 08108109
> cpu3:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,SKINIT,TCE,TOPEXT,CPCTR,DBKP,PCTRL3,MWAITX,HWPSTATE,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,RDSEED,ADX,SMAP,CLFLUSHOPT,SHA,IBPB,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
> cpu3: 32KB 64b/line 8-way D-cache, 64KB 64b/line 4-way I-cache, 512KB
> 64b/line 8-way L2 cache, 4MB 64b/line 16-way L3 cache
> cpu3: smt 0, core 3, package 0
> ioapic0 at mainbus0: apid 5 pa 0xfec00000, version 21, 24 pins
> ioapic1 at mainbus0: apid 6 pa 0xfec01000, version 21, 32 pins
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus -1 (GPP0)
> acpiprt2 at acpi0: bus -1 (GPP2)
> acpiprt3 at acpi0: bus -1 (GPP3)
> acpiprt4 at acpi0: bus -1 (GPP4)
> acpiprt5 at acpi0: bus -1 (GPP5)
> acpiprt6 at acpi0: bus -1 (GPP6)
> acpiprt7 at acpi0: bus 9 (GP17)
> acpiprt8 at acpi0: bus -1 (GP18)
> acpiprt9 at acpi0: bus 1 (GPP1)
> acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
> acpicmos0 at acpi0
> com0 at acpi0 UAR1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
> acpibtn0 at acpi0: PWRB
> amdgpio0 at acpi0 GPIO uid 0 addr 0xfed81500/0x400 irq 7, 184 pins
> "AMDIF030" at acpi0 not configured
> "PNP0C14" at acpi0 not configured
> acpicpu0 at acpi0: C1(@1 halt!), PSS
> acpicpu1 at acpi0: C1(@1 halt!), PSS
> acpicpu2 at acpi0: C1(@1 halt!), PSS
> acpicpu3 at acpi0: C1(@1 halt!), PSS
> acpivideo0 at acpi0: VGA_
> acpivideo1 at acpi0: VGA_
> acpivout0 at acpivideo1: LCD_
> cpu0: 3600 MHz: speeds: 3600 2300 1400 MHz
> pci0 at mainbus0 bus 0
> ksmn0 at pci0 dev 0 function 0 "AMD 17h/1xh Root Complex" rev 0x00
> pchb0 at pci0 dev 1 function 0 "AMD 17h PCIE" rev 0x00
> ppb0 at pci0 dev 1 function 2 "AMD 17h/1xh PCIE" rev 0x00: msi
> pci1 at ppb0 bus 1
> xhci0 at pci1 dev 0 function 0 vendor "AMD", unknown product 0x43d5
> rev 0x01: msix, xHCI 1.10
> usb0 at xhci0: USB revision 3.0
> uhub0 at usb0 configuration 1 interface 0 "AMD xHCI root hub" rev
> 3.00/1.00 addr 1
> ahci0 at pci1 dev 0 function 1 "AMD 400 Series AHCI" rev 0x01: msi, AHCI 1.3.1
> ahci0: port busy after first PMP probe FIS
> ahci0: port busy after first PMP probe FIS
> ahci0: port 1: 6.0Gb/s
> scsibus1 at ahci0: 32 targets
> sd0 at scsibus1 targ 1 lun 0: <ATA, PNY CS900 1TB SS, CS90>
> naa.5f8db4c45200b6ff
> sd0: 953869MB, 512 bytes/sector, 1953525168 sectors, thin
> ppb1 at pci1 dev 0 function 2 "AMD 400 Series PCIE" rev 0x01
> pci2 at ppb1 bus 2
> ppb2 at pci2 dev 0 function 0 "AMD 400 Series PCIE" rev 0x01: msi
> pci3 at ppb2 bus 3
> ppb3 at pci2 dev 1 function 0 "AMD 400 Series PCIE" rev 0x01: msi
> pci4 at ppb3 bus 4
> ppb4 at pci2 dev 4 function 0 "AMD 400 Series PCIE" rev 0x01: msi
> pci5 at ppb4 bus 5
> ppb5 at pci2 dev 5 function 0 "AMD 400 Series PCIE" rev 0x01: msi
> pci6 at ppb5 bus 6
> ppb6 at pci2 dev 6 function 0 "AMD 400 Series PCIE" rev 0x01: msi
> pci7 at ppb6 bus 7
> ahci1 at pci7 dev 0 function 0 "ASMedia ASM1061 AHCI" rev 0x02: msi, AHCI 1.2
> ahci1: port 0: 6.0Gb/s
> ahci1: port 1: 6.0Gb/s
> scsibus2 at ahci1: 32 targets
> sd1 at scsibus2 targ 0 lun 0: <ATA, CT1000BX500SSD1, M6CR>
> naa.500a0751e4e92e9e
> sd1: 953869MB, 512 bytes/sector, 1953525168 sectors, thin
> sd2 at scsibus2 targ 1 lun 0: <ATA, CT1000BX500SSD1, M6CR>
> naa.500a0751e4e92f88
> sd2: 953869MB, 512 bytes/sector, 1953525168 sectors, thin
> ppb7 at pci2 dev 7 function 0 "AMD 400 Series PCIE" rev 0x01: msi
> pci8 at ppb7 bus 8
> re0 at pci8 dev 0 function 0 "Realtek 8168" rev 0x15: RTL8168H/8111H
> (0x5400), msi, address a8:a1:59:45:89:37
> rgephy0 at re0 phy 7: RTL8251 PHY, rev. 0
> pchb1 at pci0 dev 8 function 0 "AMD 17h PCIE" rev 0x00
> ppb8 at pci0 dev 8 function 1 "AMD 17h/1xh PCIE" rev 0x00
> pci9 at ppb8 bus 9
> amdgpu0 at pci9 dev 0 function 0 "ATI Picasso" rev 0xc9
> drm0 at amdgpu0
> amdgpu0: msi
> azalia0 at pci9 dev 0 function 1 "ATI Radeon Vega HD Audio" rev 0x00: msi
> azalia0: no supported codecs
> ccp0 at pci9 dev 0 function 2 "AMD 17h/1xh Crypto" rev 0x00
> xhci1 at pci9 dev 0 function 3 "AMD 17h/1xh xHCI" rev 0x00: msix, xHCI 1.10
> usb1 at xhci1: USB revision 3.0
> uhub1 at usb1 configuration 1 interface 0 "AMD xHCI root hub" rev
> 3.00/1.00 addr 1
> xhci2 at pci9 dev 0 function 4 "AMD 17h/1xh xHCI" rev 0x00: msix, xHCI 1.10
> usb2 at xhci2: USB revision 3.0
> uhub2 at usb2 configuration 1 interface 0 "AMD xHCI root hub" rev
> 3.00/1.00 addr 1
> azalia1 at pci9 dev 0 function 6 "AMD 17h/1xh HD Audio" rev 0x00: apic 6 int
> 30
> azalia1: codecs: Realtek ALC892
> audio0 at azalia1
> piixpm0 at pci0 dev 20 function 0 "AMD FCH SMBus" rev 0x61: SMI
> iic0 at piixpm0
> spdmem0 at iic0 addr 0x50: 8GB DDR4 SDRAM PC4-17000
> spdmem1 at iic0 addr 0x51: 8GB DDR4 SDRAM PC4-17000
> iic1 at piixpm0
> pcib0 at pci0 dev 20 function 3 "AMD FCH LPC" rev 0x51
> pchb2 at pci0 dev 24 function 0 "AMD 17h/1xh Data Fabric" rev 0x00
> pchb3 at pci0 dev 24 function 1 "AMD 17h/1xh Data Fabric" rev 0x00
> pchb4 at pci0 dev 24 function 2 "AMD 17h/1xh Data Fabric" rev 0x00
> pchb5 at pci0 dev 24 function 3 "AMD 17h/1xh Data Fabric" rev 0x00
> pchb6 at pci0 dev 24 function 4 "AMD 17h/1xh Data Fabric" rev 0x00
> pchb7 at pci0 dev 24 function 5 "AMD 17h/1xh Data Fabric" rev 0x00
> pchb8 at pci0 dev 24 function 6 "AMD 17h/1xh Data Fabric" rev 0x00
> pchb9 at pci0 dev 24 function 7 "AMD 17h/1xh Data Fabric" rev 0x00
> isa0 at pcib0
> isadma0 at isa0
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pckbd0 at pckbc0 (kbd slot)
> wskbd0 at pckbd0: console keyboard
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> wbsio0 at isa0 port 0x2e/2: NCT6779D rev 0x62
> lm1 at wbsio0 port 0x290/8: NCT6779D
> vmm0 at mainbus0: SVM/RVI
> efifb at mainbus0 not configured
> uhidev0 at uhub0 port 8 configuration 1 interface 0 "Lenovo Lenovo
> Traditional USB Keyboard" rev 2.00/1.00 addr 2
> uhidev0: iclass 3/1
> ukbd0 at uhidev0: 8 variable keys, 6 key codes
> wskbd1 at ukbd0 mux 1
> vscsi0 at root
> scsibus3 at vscsi0: 256 targets
> softraid0 at root
> scsibus4 at softraid0: 256 targets
> root on sd0a (167ad635c8d3c660.a) swap on sd0b dump on sd0b
> WARNING: / was not properly unmounted
> amdgpu0: PICASSO GC 9.1.0 8 CU rev 0x01
> amdgpu0: 1024x768, 32bpp
> wsdisplay0 at amdgpu0 mux 1: console (std, vt100 emulation), using wskbd0
> wskbd1: connecting to wsdisplay0
> wsdisplay0: screen 1-5 added (std, vt100 emulation)
>
> usbdevs:
> Controller /dev/usb0:
> addr 01: 1022:0000 AMD, xHCI root hub
> super speed, self powered, config 1, rev 1.00
> driver: uhub0
> addr 02: 17ef:6099 Lenovo, Lenovo Traditional USB Keyboard
> low speed, power 100 mA, config 1, rev 1.00
> driver: uhidev0
> Controller /dev/usb1:
> addr 01: 1022:0000 AMD, xHCI root hub
> super speed, self powered, config 1, rev 1.00
> driver: uhub1
> Controller /dev/usb2:
> addr 01: 1022:0000 AMD, xHCI root hub
> super speed, self powered, config 1, rev 1.00
> driver: uhub2
