Hi,
I have a couple identical servers that provide a few services (not FW or
gateway -- http, ftp, etc.). Figured they would make a great CARP pair,
so if the primary broke, the secondary would take over immediately.
It would also make maintenance windows shorter...make changes on secondary
machine, test, reboot primary to force the secondary to become master.
The two machines should be equals. I have no preference on running on
one machine or the other. IF nothing breaks, I'd prefer that the one
that is serving keep serving until I tell it otherwise. Both machines
should have no issue with performance with the tasks they have, lots of
proc, lots of RAM, nvme disk, etc.
Here's the problem I've seen: I have my two machines flipping state
randomly(?). This bothers me because that means it is breaking people's
downloads. Longest period betweek flips was less than two weeks.
So ... I cranked up the carp logging to 5 and then 7 to see what it had
to say about why...and it had almost nothing to say.
Here is the info from messages from both machines for the most recent
flip. Past ones look basically the same.
Node 2:
/var/log $ zgrep carp0 messages
May 9 21:51:23 node2 /bsd: carp0: state transition: BACKUP -> MASTER
May 9 21:51:25 node2 /bsd: carp0: state transition: MASTER -> BACKUP
May 11 16:36:04 node2 /bsd: carp0: state transition: BACKUP -> MASTER
Node 1:
/var/log $ zgrep carp messages
May 9 21:51:25 node1 /bsd: carp0: state transition: MASTER -> BACKUP
May 9 21:51:28 node1 /bsd: carp0: state transition: BACKUP -> MASTER
May 11 16:36:07 node1 /bsd: carp0: state transition: MASTER -> BACKUP
hostname.carp0 from both machines:
inet a.b.c.240 255.255.255.0 128.100.17.255 vhid 1 carpdev ixl0 pass censored
inet alias a.b.c.241 255.255.255.255 128.100.17.255
inet alias a.b.c.243 255.255.255.255 128.100.17.255
inet alias a.b.c.246 255.255.255.255 128.100.17.255
verified identical (before slight anonymizing) on both systems.
hostname.ixl0 on node1:
inet a.b.c.248/24
hostname.ixl0 on node2:
inet a.b.c.247 0xffffff00
pf.conf includes this before any other "quick" statements:
pass quick inet proto carp all
Is there something I'm missing? Incorrect expectations on my part?
Nick.
dmesg:
OpenBSD 7.3-current (GENERIC.MP) #1175: Wed May 3 08:19:33 MDT 2023
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 50078154752 (47758MB)
avail mem = 48540807168 (46292MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.2 @ 0x6f3c3000 (84 entries)
bios0: vendor American Megatrends Inc. version "3.4" date 10/30/2020
bios0: Supermicro X11SPW-TF
efi0 at bios0: UEFI 2.7
efi0: American Megatrends rev 0x5000e
acpi0 at bios0: ACPI 6.2
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP FPDT FIDT SPMI UEFI SSDT MCFG HPET APIC MIGT MSCT PCAT
PCCT RASF SLIT SRAT SVOS WDDT OEM4 OEM1 SSDT OEM3 SSDT SSDT DMAR HEST BERT ERST
EINJ WSMT
acpi0: wakeup devices XHCI(S4) RP17(S4) PXSX(S4) RP18(S4) PXSX(S4) RP19(S4)
PXSX(S4) RP20(S4) PXSX(S4) RP01(S4) PXSX(S4) RP02(S4) PXSX(S4) RP03(S4)
PXSX(S4) RP04(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimcfg0 at acpi0
acpimcfg0: addr 0x80000000, bus 0-255
acpihpet0 at acpi0: 23999999 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) Bronze 3204 CPU @ 1.90GHz, 1900.06 MHz, 06-55-07
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PT,AVX512CD,AVX512BW,AVX512VL,PKU,WAITPKG,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 1MB 64b/line
16-way L2 cache, 8MB 64b/line 11-way L3 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 25MHz
cpu0: mwait min=64, max=64, C-substates=0.2.0.2, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) Bronze 3204 CPU @ 1.90GHz, 1900.09 MHz, 06-55-07
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PT,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 1MB 64b/line
16-way L2 cache, 8MB 64b/line 11-way L3 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Xeon(R) Bronze 3204 CPU @ 1.90GHz, 1900.16 MHz, 06-55-07
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PT,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu2: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 1MB 64b/line
16-way L2 cache, 8MB 64b/line 11-way L3 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) Bronze 3204 CPU @ 1.90GHz, 1900.34 MHz, 06-55-07
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PT,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu3: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 1MB 64b/line
16-way L2 cache, 8MB 64b/line 11-way L3 cache
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 8 (application processor)
cpu4: Intel(R) Xeon(R) Bronze 3204 CPU @ 1.90GHz, 1900.23 MHz, 06-55-07
cpu4:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PT,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu4: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 1MB 64b/line
16-way L2 cache, 8MB 64b/line 11-way L3 cache
cpu4: smt 0, core 4, package 0
cpu5 at mainbus0: apid 10 (application processor)
cpu5: Intel(R) Xeon(R) Bronze 3204 CPU @ 1.90GHz, 1900.22 MHz, 06-55-07
cpu5:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,MPX,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,PT,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu5: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 1MB 64b/line
16-way L2 cache, 8MB 64b/line 11-way L3 cache
cpu5: smt 0, core 5, package 0
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0: apid 9 pa 0xfec01000, version 20, 8 pins
ioapic2 at mainbus0: apid 10 pa 0xfec08000, version 20, 8 pins
ioapic3 at mainbus0: apid 11 pa 0xfec10000, version 20, 8 pins
ioapic4 at mainbus0: apid 12 pa 0xfec18000, version 20, 8 pins
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpimadt0: unknown apic structure type 7f
acpiprt0 at acpi0: bus 0 (PC00)
acpiprt1 at acpi0: bus 2 (RP06)
acpiprt2 at acpi0: bus 3 (VB00)
acpiprt3 at acpi0: bus -1 (RP09)
acpiprt4 at acpi0: bus -1 (RP11)
acpiprt5 at acpi0: bus 22 (PC01)
acpiprt6 at acpi0: bus -1 (BR1A)
acpiprt7 at acpi0: bus -1 (BR1B)
acpiprt8 at acpi0: bus 23 (BR1C)
acpiprt9 at acpi0: bus 24 (EPCU)
acpiprt10 at acpi0: bus -1 (VSP0)
acpiprt11 at acpi0: bus -1 (VSP1)
acpiprt12 at acpi0: bus -1 (VSP2)
acpiprt13 at acpi0: bus 25 (VSP3)
acpiprt14 at acpi0: bus 100 (PC02)
acpiprt15 at acpi0: bus -1 (BR2A)
acpiprt16 at acpi0: bus -1 (BR2B)
acpiprt17 at acpi0: bus -1 (BR2C)
acpiprt18 at acpi0: bus -1 (BR2D)
acpiprt19 at acpi0: bus 178 (PC03)
acpiprt20 at acpi0: bus -1 (BR3A)
acpiprt21 at acpi0: bus -1 (BR3B)
acpiprt22 at acpi0: bus -1 (BR3C)
acpiprt23 at acpi0: bus -1 (BR3D)
"ACPI0004" at acpi0 not configured
acpicpu0 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1), PSS
acpicpu1 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1), PSS
acpicpu2 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1), PSS
acpicpu3 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1), PSS
acpicpu4 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1), PSS
acpicpu5 at acpi0: C2(350@41 mwait.3@0x20), C1(1000@1 mwait.1), PSS
"PNP0C33" at acpi0 not configured
"PNP0C14" at acpi0 not configured
acpipci0 at acpi0 PC00: 0x00000010 0x00000011 0x00000000
"PNP0003" at acpi0 not configured
acpicmos0 at acpi0
com0 at acpi0 UAR1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at acpi0 UAR2 addr 0x2f8/0x8 irq 3: ns16550a, 16 byte fifo
"IPI0001" at acpi0 not configured
acpipci1 at acpi0 PC01: 0x00000010 0x00000011 0x00000000
acpipci2 at acpi0 PC02: 0x00000010 0x00000011 0x00000000
acpipci3 at acpi0 PC03: 0x00000010 0x00000011 0x00000000
"INTL0000" at acpi0 not configured
"ACPI000D" at acpi0 not configured
ipmi at mainbus0 not configured
cpu0: Enhanced SpeedStep 1900 MHz: speeds: 1900, 1800, 1700, 1600, 1500, 1400,
1300, 1200, 1100, 1000, 900, 800 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Xeon Scalable Host" rev 0x07
"Intel Xeon Scalable CBDMA" rev 0x07 at pci0 dev 4 function 0 not configured
"Intel Xeon Scalable CBDMA" rev 0x07 at pci0 dev 4 function 1 not configured
"Intel Xeon Scalable CBDMA" rev 0x07 at pci0 dev 4 function 2 not configured
"Intel Xeon Scalable CBDMA" rev 0x07 at pci0 dev 4 function 3 not configured
"Intel Xeon Scalable CBDMA" rev 0x07 at pci0 dev 4 function 4 not configured
"Intel Xeon Scalable CBDMA" rev 0x07 at pci0 dev 4 function 5 not configured
"Intel Xeon Scalable CBDMA" rev 0x07 at pci0 dev 4 function 6 not configured
"Intel Xeon Scalable CBDMA" rev 0x07 at pci0 dev 4 function 7 not configured
"Intel Xeon Scalable VT-d" rev 0x07 at pci0 dev 5 function 0 not configured
"Intel Xeon Scalable RAS" rev 0x07 at pci0 dev 5 function 2 not configured
"Intel Xeon Scalable I/O APIC" rev 0x07 at pci0 dev 5 function 4 not configured
"Intel Xeon Scalable Ubox" rev 0x07 at pci0 dev 8 function 0 not configured
"Intel Xeon Scalable Ubox" rev 0x07 at pci0 dev 8 function 1 not configured
"Intel Xeon Scalable Ubox" rev 0x07 at pci0 dev 8 function 2 not configured
"Intel C620 MROM" rev 0x09 at pci0 dev 17 function 0 not configured
vendor "Intel", unknown product 0xa1ed (class undefined unknown subclass 0x00,
rev 0x09) at pci0 dev 17 function 1 not configured
ahci0 at pci0 dev 17 function 5 "Intel C620 AHCI" rev 0x09: msi, AHCI 1.3.1
scsibus1 at ahci0: 32 targets
xhci0 at pci0 dev 20 function 0 "Intel C620 xHCI" rev 0x09: msi, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00
addr 1
"Intel C620 Thermal" rev 0x09 at pci0 dev 20 function 2 not configured
"Intel C620 MEI" rev 0x09 at pci0 dev 22 function 0 not configured
"Intel C620 MEI" rev 0x09 at pci0 dev 22 function 1 not configured
"Intel C620 MEI" rev 0x09 at pci0 dev 22 function 4 not configured
ahci1 at pci0 dev 23 function 0 "Intel C620 AHCI" rev 0x09: msi, AHCI 1.3.1
ahci1: port 0: 6.0Gb/s
ahci1: port 1: 6.0Gb/s
scsibus2 at ahci1: 32 targets
sd0 at scsibus2 targ 0 lun 0: <ATA, INTEL SSDSC2KB03, XCV1> naa.55cd2e415275c70f
sd0: 3662830MB, 512 bytes/sector, 7501476528 sectors, thin
sd1 at scsibus2 targ 1 lun 0: <ATA, INTEL SSDSC2KB03, XCV1> naa.55cd2e415275c794
sd1: 3662830MB, 512 bytes/sector, 7501476528 sectors, thin
ppb0 at pci0 dev 28 function 0 "Intel C620 PCIE" rev 0xf9: msi
pci1 at ppb0 bus 1
ppb1 at pci0 dev 28 function 5 vendor "Intel", unknown product 0xa195 rev 0xf9
pci2 at ppb1 bus 2
ppb2 at pci2 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x04
pci3 at ppb2 bus 3
"ASPEED Technology AST2000" rev 0x41 at pci3 dev 0 function 0 not configured
pcib0 at pci0 dev 31 function 0 vendor "Intel", unknown product 0xa1c2 rev 0x09
"Intel C620 Power" rev 0x09 at pci0 dev 31 function 2 not configured
ichiic0 at pci0 dev 31 function 4 "Intel C620 SMBus" rev 0x09: apic 8 int 16
iic0 at ichiic0
"Intel C620 SPI" rev 0x09 at pci0 dev 31 function 5 not configured
isa0 at pcib0
isadma0 at isa0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
pci4 at mainbus0 bus 22
ppb3 at pci4 dev 2 function 0 "Intel Xeon Scalable PCIE" rev 0x07: msi
pci5 at ppb3 bus 23
ppb4 at pci5 dev 0 function 0 vendor "Intel", unknown product 0x37c0 rev 0x09
pci6 at ppb4 bus 24
ppb5 at pci6 dev 3 function 0 vendor "Intel", unknown product 0x37c5 rev 0x09
pci7 at ppb5 bus 25
ixl0 at pci7 dev 0 function 0 "Intel X722 10GBASE-T" rev 0x09: port 0, FW
3.1.57069 API 1.5, msix, 4 queues, address 3c:ec:ef:1b:83:12
ixl1 at pci7 dev 0 function 1 "Intel X722 10GBASE-T" rev 0x09: port 1, FW
3.1.57069 API 1.5, msix, 4 queues, address 3c:ec:ef:1b:83:13
"Intel Xeon Scalable VT-d" rev 0x07 at pci4 dev 5 function 0 not configured
"Intel Xeon Scalable RAS" rev 0x07 at pci4 dev 5 function 2 not configured
"Intel Xeon Scalable IOxAPIC" rev 0x07 at pci4 dev 5 function 4 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 8 function 0 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 8 function 1 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 8 function 2 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 8 function 3 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 8 function 4 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 8 function 5 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 8 function 6 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 8 function 7 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 9 function 0 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 9 function 1 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 14 function 0 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 14 function 1 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 14 function 2 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 14 function 3 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 14 function 4 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 14 function 5 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 14 function 6 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 14 function 7 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 15 function 0 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 15 function 1 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 29 function 0 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 29 function 1 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 29 function 2 not configured
"Intel Xeon Scalable CHA" rev 0x07 at pci4 dev 29 function 3 not configured
"Intel Xeon Scalable PCU" rev 0x07 at pci4 dev 30 function 0 not configured
"Intel Xeon Scalable PCU" rev 0x07 at pci4 dev 30 function 1 not configured
"Intel Xeon Scalable PCU" rev 0x07 at pci4 dev 30 function 2 not configured
"Intel Xeon Scalable PCU" rev 0x07 at pci4 dev 30 function 3 not configured
"Intel Xeon Scalable PCU" rev 0x07 at pci4 dev 30 function 4 not configured
"Intel Xeon Scalable PCU" rev 0x07 at pci4 dev 30 function 5 not configured
"Intel Xeon Scalable PCU" rev 0x07 at pci4 dev 30 function 6 not configured
pci8 at mainbus0 bus 100
"Intel Xeon Scalable VT-d" rev 0x07 at pci8 dev 5 function 0 not configured
"Intel Xeon Scalable RAS" rev 0x07 at pci8 dev 5 function 2 not configured
"Intel Xeon Scalable IOxAPIC" rev 0x07 at pci8 dev 5 function 4 not configured
"Intel Xeon Scalable IMC" rev 0x07 at pci8 dev 8 function 0 not configured
"Intel Xeon Scalable IMC" rev 0x07 at pci8 dev 9 function 0 not configured
"Intel Xeon Scalable IMC" rev 0x07 at pci8 dev 10 function 0 not configured
"Intel Xeon Scalable IMC" rev 0x07 at pci8 dev 10 function 1 not configured
"Intel Xeon Scalable IMC" rev 0x07 at pci8 dev 10 function 2 not configured
"Intel Xeon Scalable IMC" rev 0x07 at pci8 dev 10 function 3 not configured
"Intel Xeon Scalable IMC" rev 0x07 at pci8 dev 10 function 4 not configured
"Intel Xeon Scalable LM" rev 0x07 at pci8 dev 10 function 5 not configured
"Intel Xeon Scalable LMS" rev 0x07 at pci8 dev 10 function 6 not configured
"Intel Xeon Scalable LMDP" rev 0x07 at pci8 dev 10 function 7 not configured
"Intel Xeon Scalable DECS" rev 0x07 at pci8 dev 11 function 0 not configured
"Intel Xeon Scalable LM" rev 0x07 at pci8 dev 11 function 1 not configured
"Intel Xeon Scalable LMS" rev 0x07 at pci8 dev 11 function 2 not configured
"Intel Xeon Scalable LMDP" rev 0x07 at pci8 dev 11 function 3 not configured
"Intel Xeon Scalable IMC" rev 0x07 at pci8 dev 12 function 0 not configured
"Intel Xeon Scalable IMC" rev 0x07 at pci8 dev 12 function 1 not configured
"Intel Xeon Scalable IMC" rev 0x07 at pci8 dev 12 function 2 not configured
"Intel Xeon Scalable IMC" rev 0x07 at pci8 dev 12 function 3 not configured
"Intel Xeon Scalable IMC" rev 0x07 at pci8 dev 12 function 4 not configured
"Intel Xeon Scalable LM" rev 0x07 at pci8 dev 12 function 5 not configured
"Intel Xeon Scalable LMS" rev 0x07 at pci8 dev 12 function 6 not configured
"Intel Xeon Scalable LMDP" rev 0x07 at pci8 dev 12 function 7 not configured
"Intel Xeon Scalable DECS" rev 0x07 at pci8 dev 13 function 0 not configured
"Intel Xeon Scalable LM" rev 0x07 at pci8 dev 13 function 1 not configured
"Intel Xeon Scalable LMS" rev 0x07 at pci8 dev 13 function 2 not configured
"Intel Xeon Scalable LMDP" rev 0x07 at pci8 dev 13 function 3 not configured
pci9 at mainbus0 bus 178
"Intel Xeon Scalable VT-d" rev 0x07 at pci9 dev 5 function 0 not configured
"Intel Xeon Scalable RAS" rev 0x07 at pci9 dev 5 function 2 not configured
"Intel Xeon Scalable IOxAPIC" rev 0x07 at pci9 dev 5 function 4 not configured
"Intel Xeon Scalable KTI" rev 0x07 at pci9 dev 14 function 0 not configured
"Intel Xeon Scalable UPI" rev 0x07 at pci9 dev 14 function 1 not configured
"Intel Xeon Scalable KTI" rev 0x07 at pci9 dev 15 function 0 not configured
"Intel Xeon Scalable UPI" rev 0x07 at pci9 dev 15 function 1 not configured
"Intel Xeon Scalable M3KTI" rev 0x07 at pci9 dev 18 function 0 not configured
"Intel Xeon Scalable M3KTI" rev 0x07 at pci9 dev 18 function 1 not configured
"Intel Xeon Scalable M3KTI" rev 0x07 at pci9 dev 18 function 2 not configured
"Intel Xeon Scalable M2PCI" rev 0x07 at pci9 dev 21 function 0 not configured
"Intel Xeon Scalable M2PCIe" rev 0x07 at pci9 dev 21 function 1 not configured
"Intel Xeon Scalable M2PCI" rev 0x07 at pci9 dev 22 function 0 not configured
"Intel Xeon Scalable M2PCIe" rev 0x07 at pci9 dev 22 function 1 not configured
"Intel Xeon Scalable M2PCI" rev 0x07 at pci9 dev 22 function 4 not configured
"Intel Xeon Scalable M2PCIe" rev 0x07 at pci9 dev 22 function 5 not configured
vmm0 at mainbus0: VMX/EPT
efifb0 at mainbus0: 1024x768, 32bpp
wsdisplay0 at efifb0 mux 1
wsdisplay0: screen 0-5 added (std, vt100 emulation)
uhub1 at uhub0 port 7 configuration 1 interface 0 "ATEN International product
0x7000" rev 2.00/0.00 addr 2
uhidev0 at uhub1 port 1 configuration 1 interface 0 "ATEN International product
0x2419" rev 1.10/1.00 addr 3
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd0 at ukbd0 mux 1
wskbd0: connecting to wsdisplay0
uhidev1 at uhub1 port 1 configuration 1 interface 1 "ATEN International product
0x2419" rev 1.10/1.00 addr 3
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
umass0 at uhub0 port 14 configuration 1 interface 0 "Kingston DataTraveler 2.0"
rev 2.00/1.00 addr 4
umass0: using SCSI over Bulk-Only
scsibus3 at umass0: 2 targets, initiator 0
sd2 at scsibus3 targ 1 lun 0: <Kingston, DataTraveler 2.0, 1.00> removable
serial.09306544C190193DE778
sd2: 14782MB, 512 bytes/sector, 30274560 sectors
vscsi0 at root
scsibus4 at vscsi0: 256 targets
softraid0 at root
scsibus5 at softraid0: 256 targets
sd3 at scsibus5 targ 1 lun 0: <OPENBSD, SR RAID 1, 006>
sd3: 3584022MB, 512 bytes/sector, 7340079098 sectors
root on sd3a (62fa5c6be4ad1ded.a) swap on sd3b dump on sd3b
carp0: state transition: INIT -> BACKUP (at boot)
carp0: state transition: BACKUP -> INIT ( " " )
carp0: state transition: INIT -> BACKUP ( " " )
carp0: state transition: BACKUP -> INIT ( " " )
carp0: state transition: INIT -> BACKUP ( " " )
carp0: state transition: BACKUP -> INIT ( " " )
carp0: state transition: INIT -> BACKUP ( " " )
carp0: state transition: BACKUP -> MASTER (today)
carp0: state transition: MASTER -> BACKUP
carp0: state transition: BACKUP -> MASTER
