On Thu, Nov 9, 2023 at 10:12 PM Alexandr Nedvedicky <sas...@fastmail.net> wrote:
>
> Hello Johan,
>
> On Sat, Nov 04, 2023 at 10:01:06AM -0400, Johan Huldtgren wrote:
> > hello,
> >
> > On 2023-11-03 19:10, Alexandr Nedvedicky wrote:
> > > Hello Johan,
> > >
> > >
> > > On Fri, Nov 03, 2023 at 12:27:53PM -0400, Johan Huldtgren wrote:
> > > </snip>
> > > >
> > > > so this box just has the default (from when it was installed) ruleset.
> > > >
> > > > $ doas cat /etc/pf.conf
> > > > # $OpenBSD: pf.conf,v 1.55 2017/12/03 20:40:04 sthen Exp $
> > > > #
> > > > # See pf.conf(5) and /etc/examples/pf.conf
> > > >
> > > > set skip on lo
> > > > set state-defaults pflow
> > > >
> > > > block return # block stateless traffic
> > > > pass # establish keep-state
> > > >
> > > > # By default, do not permit remote connections to X11
> > > > block return in on ! lo0 proto tcp to port 6000:6010
> > > >
> > > > # Port build user does not need network
> > > > block return out log proto {tcp udp} user _pbuild
> > > >
> > >
> > > So that's surprising then... Looks like you are very lucky
> > > to hit the ASSERT. I'm surprised we have not seen it earlier.
> > >
> > > Diff below makes sure pf_test() function does not overwrite
> > > timeout member in pf_state structure when timeout is set
> > > to PFTM_UNLINKED already. We also modify/update timeout member
> > > under protection of state mutex (pf_state::mtx).
> > >
> > >
> > > Can you test the diff below? It applies to current as well to 7.4
> >
> > I've rebuilt with your diff, as the panic was seemingly random I'm not
> > sure how I can test, but I'll let this system run with your patch and
> > report any issues should I see them. If you have any specific things
> > you'd like me to try don't hesitate to let me know. dmesg below for
> > complteness sake.
> >
> > thanks again,
> >
>
> I'm afraid there is nothing more to do than keep an eye on your
> system. I think what really increased a chance here is the number
> of CPUs your box has.
>
> It is OK if you can come back with report early in December to let
> us know if it helps or if there are more similar issues (which I'm
> sure there are still some left).
>
>
> thanks and
> regards
> sashan
>
I encountered similar panic last week after 3 weeks of uptime with
7.3-current (GENERIC.MP) #1346: Sun Aug 20 12:39:24 MDT 2023
dera...@amd64.openbsd.org:/usr/src/sys/arch/adm64/compile/GENERIC.MP:
ddb{2}> show panic
*cpu2: kernel diagnostic assertion "st->timeout == PFTM_UNLINKED" failed: file "
/usr/src/sys/net/pf.c", line 1844
ddb{2}> trace
db_enter() at db_enter+0x14
panic(ffffffff820a61d9) at panic+0xc3
__assert(ffffffff82122832,ffffffff8207fadb,734,ffffffff8207fb0c) at __assert+0x
29
pf_free_state(fffffd804fa00038) at pf_free_state+0x1ef
pf_purge_expired_states(40,40) at pf_purge_expired_states+0x288
pf_purge_states(0) at pf_purge_states+0x20
taskq_thread(ffffffff82476340) at taskq_thread+0x100
end trace frame: 0x0, count: -7
ddb{2}> mach ddbcpu 0
Stopped at x86_ipi_db+0x16: leave
x86_ipi_db(ffffffff82486ff0) at x86_ipi_db+0x16
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x27
_kernel_lock() at _kernel_lock+0xb6
softintr_dispatch(0) at softintr_dispatch+0x4d
Xsoftclock() at Xsoftclock+0x27
acpicpu_idle() at acpicpu_idle+0x11f
sched_idle(ffffffff82486ff0) at sched_idle_0x282
end trace fram: 0x0, count: 7
ddb{0}> trace
x86_ipi_db(ffffffff82486ff0) at x86_ipi_db+0x16
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x27
_kernel_lock() at _kernel_lock+0xb6
softintr_dispatch(0) at softintr_dispatch+0x4d
Xsoftclock() at Xsoftclock+0x27
acpicpu_idle() at acpicpu_idle+0x11f
sched_idle(ffffffff82486ff0) at sched_idle+0x282
end trace frame: 0x0, count: -8
ddb{0}> mach ddbcpu 1
Stopped at x86_ipi_db+0x16: leave
x86_ipi_db(ffff80001fa88ff0) at x86_ipi_db+0x16
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x27
msleep_nsec(ffff80000002d000,ffff80001fc5e6f0,b75a161d98142751,ffff80000002d018
,ffff80000002d000) at msleep_nsec+0xf7
taskq_thread(ffff80000002d000) at taskq_thread+0xeb
end trace frame: 0x0, count: 10
ddb{1}> trace
x86_ipi_db(ffff80001fa88ff0) at x86_ipi_db+0x16
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x27
msleep_nsec(ffff80000002d000,ffff80001fc5e6f0,b75a161d98142751,ffff80000002d018
,ffff80000002d000) at msleep_nsec+0xf7
taskq_thread(ffff80000002d000) at taskq_thread+0xeb
end trace frame: 0x0, count: -5
ddb{1}> mach ddbcpu 2
Stopped at db_enter+0x14: popq %rbp
db_enter() at db_enter+0x14
panic(ffffffff820a61d9) at panic+0xc3
__assert(ffffffff82122832,ffffffff8207fadb,734,ffffffff8207fb0c) at __assert+0x
29
pf_free_state(fffffd804fa00038 at pf_free_state+0x1ef
pf_purge_expired_states(40,40) at pf_purge_expired_states+0x288
pf_purge_states(0) at pf_purge_states+0x20
taskq_thread(ffffffff82476340) at taskq_thread+0x100
end trace fram: 0x0, count: 8
ddb{2}> trace
db_enter() at db_enter+0x14
panic(ffffffff820a61d9) at panic+0xc3
__assert(ffffffff82122832,ffffffff8207fadb,734,ffffffff8207fb0c) at __assert+0x
29
pf_free_state(fffffd804fa00038) at pf_free_state+0x1ef
pf_purge_expired_states(40,40) at pf_purge_expired_states+0x288
pf_purge_states(0) at pf_purge_states+0x20
taskq_thread(ffffffff82476340) at taskq_thread+0x100
end trace frame: 0x0, count: -7
ddb{2}> mach ddbcpu 3
Stopped at x86_ipi_db+0x16: leave
x86_ipi_db(ffff80001fa9aff0) at x86_ipi_db+0x16
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x27
acpicpu_idle() at acpicpu_idle+0x11f
sched_idle(ffff80001fa9aff0) at sched_idle+0x282
end trace frame: 0x0, count: 10
ddb{3}> trace
x86_ipi_db(ffff80001fa9aff0) at x86_ipi_db+0x16
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x27
acpicpu_idle() at acpicpu_idle_0x11f
sched_idle(ffff80001fa9aff0) at sched_idle+0x282
end trace frame: 0x0, count: -5
ddb{3}> ps
PID TID PPID UID S FLAGS WAIT
COMMAND
49828 109313 40635 1000 3 0x100083 ttyin less
99913 426164 56093 1000 3 0x100083 kqread ssh
56093 378608 14826 1000 3 0x10008b sigsusp ksh
14826 482819 14407 1000 3 0x98 kqread sshd
14407 331907 89469 0 3 0x92 kqread sshd
40635 234336 65443 1000 3 0x10008b sigsusp ksh
65443 171362 59936 1000 3 0x98 kqread sshd
59936 439467 89469 0 3 0x92 kqread sshd
35176 10066 1 77 3 0x1100090 kqread dhcpd
2538 230844 1 0 3 0x100083 ttyin
getty
77260 469097 1 0 3 0x100098 kqread cron
16355 442093 1 99 3 0x1100090 kqread sndiod
29505 476138 1 110 3 0x100090 kqread sndiod
33217 269070 40338 95 3 0x1100092 kqread smtpd
19183 200874 40388 103 3 0x1100092 kqread smtpd
61831 368489 40338 95 3 0x1100092 kqread smtpd
20543 238490 40338 95 3 0x100092 kqread smtpd
8533 128283 40338 95 3 0x1100092 kqread smtpd
65920 228242 40338 95 3 0x1100092 kqread smtpd
40338 383245 1 0 3 0x100080 kqread smtpd
89469 524248 1 0 3 x88 kqread
sshd
--db_more--
93438 491280 1820 101 3 0x1100092 kqread iked
24368 222283 1820 101 3 0x1100092 kqread iked
423 228599 1820 101 3 0x1100092 kqread iked
1820 498383 1 0 3 0x100080 kqread iked
37653 112388 1 0 3 0x100080 kqread ntpd
75733 231182 40443 83 3 0x100092 kqread ntpd
40443 20635 1 83 3 0x1100092 kqread ntpd
92154 158057 42420 74 3 0x1100092 bpf pflogd
42420 163145 1 0 3 0x80 netio
pflogd
47619 327457 11043 73 3 0x1100090 kqread syslogd
11043 279185 1 0 3 0x100082 netio
syslogd
69525 374523 1 0 3 0x100080 kqread
resolvd
47238 79075 99439 77 3 0x100092 kqread
dhcpleased
78825 148940 99439 77 3 0x100092 kqread dhcpleased
99439 483578 1 0 3 0x80 kqread
dhcpleased
47318 455993 10057 115 3 0x100092 kqread slaacd
93212 127040 10057 115 3 0x100092 kqread slaacd
10057 442056 1 0 3 0x100080 kqread
slaacd
76557 428351 0 0 3 0x14200 bored smr
26845 430752 0 0 3 0x14200 pgzero
zerothread
16801 41909 0 0 3 0x14200 aiodoned
aiodoned
3476 118302 0 0 3 0x14200 syncer
update
72113 349466 0 0 3 0x14200 cleaner
cleaner
--db_more--
92145 112504 0 0 3 0x14200 reaper
reaper
41287 90567 0 0 3 0x14200 pgdaemon
pagedaemon
18443 295853 0 0 3 0x14200 mmctsk
sdmmc0
26526 513876 0 0 3 0x14200 usbtsk
usbtask
32084 69700 0 0 3 0x14200 usbatsk
usbatsk
97604 396731 0 0 3 0x40014200 acpi0
acpi0
*42829 165462 0 0 7 0x40014200
idle3
64433 158507 0 0 3 0x40014200
idle2
15343 67888 0 0 3 0x40014200
idle1
26746 37883 0 0 3 0x14200 bored
sensors
28923 282355 0 0 3 0x14200 bored
softnet3
99424 257684 0 0 3 0x14200 bored
softnet2
53191 229437 0 0 3 0x14200 bored
softnet1
56816 400661 0 0 7 0x14200
softnet0
77230 43264 0 0 7 0x14200
systqmp
87215 96496 0 0 3 0x14200 bored
systq
18207 79492 0 0 3 0x40014200 bored
softclock
90993 33193 0 0 7 0x40014200
idle0
1 154578 0 0 3 0x82
wait init
0 0 -1 0 3 0x10200
scheduler swapper
ddb{3}>
ddb{2}> ps
[same results as ddb{3} ps except for asterisk is before PID 77230
instead of PID 42829]
dmesg:
OpenBSD 7.3-current (GENERIC.MP) #1346: Sun Aug 20 12:39:24 MDT 2023
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 1996476416 (1903MB)
avail mem = 1916317696 (1827MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus at root
bios0 at mainbus0: SMBIOS rev.2.8 @ 0x7ee9e020 (12 entries)
bios0: vendor coreboot version "v4.11.0.2" date 12/30/2019
bios0: PC Engines apu4
acpi0 at bios0: ACPI 4.0
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP SSDT MCFG APIC HEST SSDT SSDT HPET
acpi0: wakeup devices PWRB(S4) PBR4(S4) PBR5(S4) PBR6(S4) PBR7(S4) PBR8(S4) U0H1
(S3) U0H2(S3) U0H3(S3) U0H4(S3) U0H5(S3) U0H6(S3) XHC0(S4)
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimcfg0 at acpi0
acpimcfg0: addr 0xf8000000, bus 0-64
acpimadt0 at acpi0 addr 0xfee0000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD GX-412TC SOC, 998.21 MHz, 16-30-01, patch 07030105
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 2-way I-cache, 2MB
64b/line 16-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: AMD GX-412TC SOC, 998.21 MHz, 16-30-01, patch 07030105
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 2-way I-cache, 2MB
64b/line 16-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: AMD GX-412TC SOC, 998.21 MHz, 16-30-01, patch 07030105
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 2-way I-cache, 2MB
64b/line 16-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: AMD GX-412TC SOC, 998.21 MHz, 16-30-01, patch 07030105
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu3: 32KB 64b/line 8-way D-cache, 32KB 64b/line 2-way I-cache, 2MB
64b/line 16-way L2 cache
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 21, 24 pins
ioapic1 at mainbus0: apid 5 pa 0xfec20000, version 21, 32 pins, remapped
acpihpet0 at acpi0: 14318180 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PBR4)
acpiprt2 at acpi0: bus 2 (PBR5)
acpiprt3 at acpi0: bus 3 (PBR6)
acpiprt4 at acpi0: bus 4 (PBR7)
acpiprt5 at acpi0: bus 5 (PBR8)
acpibtn0 at acpi0: PWRB(wakeup)
acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
acpicmos0 at acpi0
com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at acpi0 COM2 addr 0x2f8/0x8 irq 3: ns16550a, 16 byte fifo
amdgpio0 at acpi0 GPIO uid 0 addr 0xfed81500/0x300 irq 7, 184 pins
"PRP0001" at acpi0 not configured
"PRP0001" at acpi0 not configured
"PRP0001" at acpi0 not configured
"PRP0001" at acpi0 not configured
"PRP0001" at acpi0 not configured
"PRP0001" at acpi0 not configured
"BOOT0000" at acpi0 not configured
acpicpu0 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpicpu1 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpicpu2 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpicpu3 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
cpu0: 988 MHz: speeds: 1000 800 600 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "AMD 16h Root Complex" rev 0x00
vendor "AMD", unknown product 0x1567 (class system subclass IOMMU, rev
0x00) at pci0 function 2 not configured
pchb1 at pci0 dev 2 function 0 "AMD 16h Host" rev 0x00
ppb0 at pci0 dev 2 function 1 "AMD 16h PCIE" rev 0x00: msi
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 ""Intel I211" rev 0x03: msi, address
xx:xx:xx:xx:xx:xx
ppb1 at pci0 dev 2 function 2 "AMD 16h PCIE" rev 0x00: msi
pci2 at ppb1 bus 2
em1 at pci2 dev 0 function 0 "Intel I211" rev 0x03: msi, address
xx:xx:xx:xx:xx:xx
ppb2 at pci0 dev 2 function 3 "AMD 16h PCIE" rev 0x00: msi
pci3 at ppb2 bus 3
em2 at pci3 dev 0 function 0 "Intel I211" rev 0x03: msi, address
xx:xx:xx:xx:xx:xx
ppb3 at pci0 dev 2 function 4 "AMD 16h PCIE" rev 0x00: msi
pci4 at ppb3 bus 4
em3 at pci4 dev 0 function 0 "Intel I211" rev 0x03: msi, address
xx:xx:xx:xx:xx:xx
ppb4 at pci0 dev 2 function 5 "AMD 16h PCIE" rev 0x00: msi
pci5 at ppb4 bus 5
athn0 at pci5 dev 0 function 0 "Atheros AR928X" rev 0x01: apic 5 int 16
athn0: AR9280 rev 2 (2T2R), ROM rev 22, address xx:xx:xx:xx:xx:xx
ccp0 at pci0 dev 8 function 0 "AMD 16h Crypto" rev 0x00
xhci0 at pci0 dev 16 function 0 "AMD Bolton xHCI" rev 0x11: msix, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "AMD xHCI root hub" rev
3.00/1.00 addr 1
ahci0 at pci0 dev 17 function 0 "AMD Hudson-2 SATA" rev 0x40: apic 4
int 19, AHCI 1.3
ahci0: port 0: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, SATA SSD, SBFM>
t10.ATA_SATA_SSD_FFXXXXXXXXXXXXXXXXXX
sd0: 28626MB, 512 bytes/sector, 58626288 sectors, thin
ehci0 at pci0 dev 19 function 0 "AMD Hudson-2 USB2" rev 0x39: apic 4 int 18
usb1 at ehci0: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "AMD EHCI root hub" rev
2.00/1.00 addr 1
piixpm0 at pci0 dev 20 function 0 "AMD Hudson-2 SMBus" rev 0x42: SMI
iic0 at piixpm0
iic1 at piixpm0
pcib0 at pci0 dev 20 function 3 "AMD Hudson-2 LPC" rev 0x11
sdhc0 at pci dev 20 function 7 "AMD Bolton SD/MMC" rev 0x01: apic 4 int 16
sdhc0: SDHC 2.0, 50 MHz base clock
sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed, dma
pchb2 at pci0 dev 24 function 0 "AMD 16h Link Cfg" rev 0x00
pchb3 at pci0 dev 24 function 1 "AMD 16h Address Map" rev 0x00
pchb4 at pci0 dev 24 function 2 "AMD 16h DRAM Cfg" rev 0x00
km0 at pci0 dev 24 function 3 "AMD 16h Misc Cfg" rev 0x00
pchb5 at pci0 dev 24 function 4 "AMD 16h CPU Power" rev 0x00
pchb6 at pci0 dev 24 function 5 "AMD 16h Misc Cfg" rev 0x00
isa0 at pcib0
isadma0 at isa0
com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
intr_establish: pic ioapic0 pin 7: can't share type 3 with 2
wbsio0 at isa0 port 0x2e/2: NCT5104D rev 0x53
vmm0 at mainbus0: SVM/RVI
uplcom0 at uhub0 port 3 configuration 1 interface 0 "Prolific
Technology Inc. USB-Serial Controller D" rev 1.10/4.00 addr 2
ucom0 at uplcom0
uhub2 at uhub1 port 1 configuration 1 interface 0 "Advanced Micro
Devices Hub" rev 2.00/0.18 addr 2
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (axxxxxxxxxxxxxxx.a) swap on sd0b dump on sd0b
- Ted
