Is this still an issue?
We run several OpenBSD 7.5 VMs at Hetzner and I am somewhat concerned about the
machines not coming back up after a reboot.
So I created a new CX22 host, installed OpenBSD 7.5 and it seems to work fine.
dmesg:
OpenBSD 7.5 (GENERIC.MP) #82: Wed Mar 20 15:48:40 MDT 2024
dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4177379328 (3983MB)
avail mem = 4029792256 (3843MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5210 (11 entries)
bios0: vendor Hetzner version "20171111" date 11/11/2017
bios0: Hetzner vServer
acpi0 at bios0: ACPI 3.0
acpi0: sleep states S5
acpi0: tables DSDT FACP APIC HPET MCFG WAET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel Xeon Processor (Skylake, IBRS, no TSX), 2100.74 MHz, 06-55-04
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLWB,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,SSBD,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line
16-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel Xeon Processor (Skylake, IBRS, no TSX), 2133.33 MHz, 06-55-04
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,CLWB,AVX512CD,AVX512BW,AVX512VL,PKU,MD_CLEAR,IBRS,IBPB,SSBD,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line
16-way L2 cache, 16MB 64b/line 16-way L3 cache
cpu1: smt 0, core 1, package 0
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpimcfg0 at acpi0
acpimcfg0: addr 0xb0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PCI0)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo
acpicmos0 at acpi0
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
"ACPI0010" at acpi0 not configured
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
cpu0: using VERW MDS workaround
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82G33 Host" rev 0x00
vga1 at pci0 dev 1 function 0 "Qumranet Virtio 1.x GPU" rev 0x01
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ppb0 at pci0 dev 2 function 0 "Red Hat PCIE" rev 0x00: apic 0 int 22
pci1 at ppb0 bus 1
virtio0 at pci1 dev 0 function 0 "Qumranet Virtio 1.x Network" rev 0x01
vio0 at virtio0: address 96:00:03:b0:6b:55
virtio0: msix per-VQ
ppb1 at pci0 dev 2 function 1 "Red Hat PCIE" rev 0x00: apic 0 int 22
pci2 at ppb1 bus 2
xhci0 at pci2 dev 0 function 0 "Red Hat xHCI" rev 0x01: msix, xHCI 0.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Red Hat xHCI root hub" rev 3.00/1.00
addr 1
ppb2 at pci0 dev 2 function 2 "Red Hat PCIE" rev 0x00: apic 0 int 22
pci3 at ppb2 bus 3
virtio1 at pci3 dev 0 function 0 "Qumranet Virtio 1.x Console" rev 0x01
virtio1: no matching child driver; not configured
ppb3 at pci0 dev 2 function 3 "Red Hat PCIE" rev 0x00: apic 0 int 22
pci4 at ppb3 bus 4
virtio2 at pci4 dev 0 function 0 "Qumranet Virtio 1.x Memory Balloon" rev 0x01
viomb0 at virtio2
virtio2: apic 0 int 22
ppb4 at pci0 dev 2 function 4 "Red Hat PCIE" rev 0x00: apic 0 int 22
pci5 at ppb4 bus 5
virtio3 at pci5 dev 0 function 0 "Qumranet Virtio 1.x RNG" rev 0x01
viornd0 at virtio3
virtio3: apic 0 int 22
ppb5 at pci0 dev 2 function 5 "Red Hat PCIE" rev 0x00: apic 0 int 22
pci6 at ppb5 bus 6
virtio4 at pci6 dev 0 function 0 "Qumranet Virtio 1.x SCSI" rev 0x01
vioscsi0 at virtio4: qsize 256
scsibus1 at vioscsi0: 255 targets
sd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU HARDDISK, 2.5+>
sd0: 39064MB, 512 bytes/sector, 80003072 sectors, thin
virtio4: msix per-VQ
ppb6 at pci0 dev 2 function 6 "Red Hat PCIE" rev 0x00: apic 0 int 22
pci7 at ppb6 bus 7
ppb7 at pci0 dev 2 function 7 "Red Hat PCIE" rev 0x00: apic 0 int 22
pci8 at ppb7 bus 8
ppb8 at pci0 dev 3 function 0 "Red Hat PCIE" rev 0x00: apic 0 int 23
pci9 at ppb8 bus 9
pcib0 at pci0 dev 31 function 0 "Intel 82801IB LPC" rev 0x02
ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: msi, AHCI 1.0
ahci0: port 0: 1.5Gb/s
scsibus2 at ahci0: 32 targets
cd0 at scsibus2 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable
ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 0 int 16
iic0 at ichiic0
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
uhidev0 at uhub0 port 5 configuration 1 interface 0 "QEMU QEMU USB Tablet" rev
2.00/0.00 addr 2
uhidev0: iclass 3/0
ums0 at uhidev0: 3 buttons, Z dir
wsmouse1 at ums0 mux 0
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (cb7ffb503da3a7aa.a) swap on sd0b dump on sd0b
That doesn’t prove anything other than that the problem does not seem to be
universal for all VMs at Hetzner.
Thanks!
Mike
> Am 06.09.2024 um 22:32 schrieb Stefan Fritsch <s...@openbsd.org>:
>
> Hi,
>
> I accidentally replied only to the sender. Here is the reply to the list.
>
> On Fri, 6 Sep 2024, Thomas Siegmund wrote:
>> We at SEPPmail have been using OpenBSD as the operating system for our
>> SEPPmail mail gateway since
>> the company was founded.
>> For two years we have been developing and operating a cloud service for mail
>> processing,
>> encryption/decryption, spam filtering and so on.
>>
>> Yesterday we had an incident at Hetzner, a German data center operator. They
>> upgraded their QEMU
>> version (we assume to 8.2.3, but they are not willing to tell us the exact
>> version) and this caused
>> all OpenBSD 7.5 appliances to fail to boot (short boot up to initialize the
>> Virtio network interface
>> or the Vioscsi driver and then the appliance just shuts down).
>> We found out that this was caused by the Virtio drivers if_vio and vioscsi.
>> By undoing one of your
>> commits
>> https://github.com/openbsd/src/commit/cdd248411fe303b936d5a056fde97097bd7015f0
>> "virtio: Set
>> DRIVER_OK earlier” we were able to get our service up and running again.
>> Hetzner themselves found out that it could be a problem related to the QEMU
>> commit
>> https://gitlab.com/qemu-project/qemu/-/commit/fcbb086ae590e910614fe5b8bf76e264f71ef304.
>>
>> Since I am not familiar with the OpenBSD kernel and the virtio system, I am
>> not able to figure out
>> what is really causing the crash/shutdown.
>> Maybe you can figure it out and this information will be helpful.
>
> I have seen a qemu crash related to virtio with openbsd on some qemu 9.1
> rc, I think rc2. This commit fixed it:
> https://gitlab.com/qemu-project/qemu/-/commit/a8e63ff289d137197ad7a701a587cc432872d798
>
> I have also tried an older version and that had the same issue, I believe
> that was some 9.0.2 package from debian. The fix is in qemu 9.1 release. I
> think it likely that this is your issue. Maybe you can ask Hetzner if they
> can backport the fix to their build?
>
> I don't think there is a bug in OpenBSD in this. The OpenBSD commit you
> mentioned fixes various problems, both with qemu and with other
> hypervisors. We definitely don't want to revert that commit.
>
> Cheers,
> Stefan
--
Mike Fischer
fisc...@lavielle.com
