Hi all! I have this OpenBSD 7.4 qemu/kvm VM managed by libvirt on an Ubuntu 22.04 host.
I started to notice this month that it started to act weird, it seems like the clock stops every night. I couldn't pinpoint exactly what caused the change in behavior, the host had two package updates that raised suspicion: 2024-01-11 06:51:04 upgrade linux-image-generic-hwe-22.04:amd64 6.2.0.39.40~22.04.16 6.5.0.14.14~22.04.7 2024-01-12 09:10:36 upgrade libvirt-daemon:amd64 8.0.0-1ubuntu7.7 8.0.0-1ubuntu7.8 But none of the changelogs /seemed/ relevant. Anyway, the symptoms are funny, it always involves the clock stopping/not working after some period of time. When this happens, I cannot login with SSH. The ssh client connects, it even asks for the private key, but after confirmation it times out. The really funny things happen when I log in on the console - that I can do: When I try to ping anything from the host, it stops after the first successful packets (echo/reply) and then hangs (I can CTRL+C). Interestingly I can ping the VM from the hypervisor host indefinitely, but running tcpdump on the guest doesn't show anything immediately. In fact, looking at tcpdump while doing *anything* network related on the VM or to the VM doesn't result in any output right away. That being said, after a couple of minutes, output from tcpdump starts to flood the screen but I cannot say exactly why or when, it just suddenly happens. Running `sleep 1` just hangs. When I run `date` consecutively it shows: Fri Jan 26 04:20:42 CET 2024 Fri Jan 26 04:20:39 CET 2024 Fri Jan 26 04:20:40 CET 2024 Fri Jan 26 04:20:41 CET 2024 Fri Jan 26 04:20:42 CET 2024 Fri Jan 26 04:20:43 CET 2024 Fri Jan 26 04:20:41 CET 2024 Fri Jan 26 04:20:42 CET 2024 It always works again after a reboot - forced reset, because it cannot shut down gracefully. Originally I was using SP kernel but tried with MP recently too, just out of curiosity - no luck. I found two old posts seemingly related: https://marc.info/?t=152942296100002&r=1&w=2 ^^ I don't have that sysctl on the host and that kernel is very old there. https://www.reddit.com/r/openbsd/comments/13c9nh1/clock_issue_with_vmm_guest_on_73/ This is on an OpenBSD host, so I can't try that sysctl either. This has been set on the guest, though (defaults): kern.timecounter.tick=1 kern.timecounter.timestepwarnings=0 kern.timecounter.hardware=pvclock0 kern.timecounter.choice=i8254(0) pvclock0(1500) acpitimer0(1000) Any clues would be appreciated, Daniel dmesg: OpenBSD 7.4 (GENERIC.MP) #1397: Tue Oct 10 09:02:37 MDT 2023 dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 519929856 (495MB) avail mem = 484507648 (462MB) random: good seed from bootblocks mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5960 (10 entries) bios0: vendor SeaBIOS version "1.15.0-1" date 04/01/2014 bios0: QEMU Standard PC (i440FX + PIIX, 1996) acpi0 at bios0: ACPI 1.0 acpi0: sleep states S5 acpi0: tables DSDT FACP APIC WAET acpi0: wakeup devices acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: 12th Gen Intel(R) Core(TM) i7-12700K, 3609.77 MHz, 06-97-02 cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,PKU,WAITPKG,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,IBRS_ALL,SKIP_L1DFL,MDS_NO,IF_PSCHANGE,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line 16-way L2 cache, 16MB 64b/line 16-way L3 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 1000MHz cpu1 at mainbus0: apid 1 (application processor) cpu1: 12th Gen Intel(R) Core(TM) i7-12700K, 3609.78 MHz, 06-97-02 cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,PKU,WAITPKG,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,IBRS_ALL,SKIP_L1DFL,MDS_NO,IF_PSCHANGE,XSAVEOPT,XSAVEC,XGETBV1,XSAVES cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB 64b/line 16-way L2 cache, 16MB 64b/line 16-way L3 cache cpu1: smt 0, core 0, package 1 ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins acpiprt0 at acpi0: bus 0 (PCI0) "ACPI0006" at acpi0 not configured acpipci0 at acpi0 PCI0 com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo acpicmos0 at acpi0 "PNP0A06" at acpi0 not configured "PNP0A06" at acpi0 not configured "PNP0A06" at acpi0 not configured "QEMU0002" at acpi0 not configured "ACPI0010" at acpi0 not configured acpicpu0 at acpi0: C1(@1 halt!) acpicpu1 at acpi0: C1(@1 halt!) pvbus0 at mainbus0: KVM pvclock0 at pvbus0 pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02 pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00 pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility pciide0: channel 0 disabled (no drives) pciide0: channel 1 disabled (no drives) piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9 iic0 at piixpm0 vga1 at pci0 dev 2 function 0 "Qumranet Virtio 1.x GPU" rev 0x01 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00 vio0 at virtio0: address 52:54:XX:XX:XX:XX virtio0: msix per-VQ azalia0 at pci0 dev 4 function 0 "Intel 82801FB HD Audio" rev 0x01: apic 0 int 11 azalia0: No codecs found uhci0 at pci0 dev 5 function 0 "Intel 82801I USB" rev 0x03: apic 0 int 10 uhci1 at pci0 dev 5 function 1 "Intel 82801I USB" rev 0x03: apic 0 int 10 uhci2 at pci0 dev 5 function 2 "Intel 82801I USB" rev 0x03: apic 0 int 11 ehci0 at pci0 dev 5 function 7 "Intel 82801I USB" rev 0x03: apic 0 int 11 usb0 at ehci0: USB revision 2.0 uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 virtio1 at pci0 dev 6 function 0 "Qumranet Virtio Console" rev 0x00 virtio1: no matching child driver; not configured virtio2 at pci0 dev 7 function 0 "Qumranet Virtio Storage" rev 0x00 vioblk0 at virtio2 scsibus1 at vioblk0: 1 targets sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > sd0: 13312MB, 512 bytes/sector, 27262976 sectors virtio2: msix per-VQ virtio3 at pci0 dev 8 function 0 "Qumranet Virtio Memory Balloon" rev 0x00 viomb0 at virtio3 virtio3: apic 0 int 11 ahci0 at pci0 dev 9 function 0 "Intel 82801I AHCI" rev 0x02: apic 0 int 10, AHCI 1.0 ahci0: port 0: 1.5Gb/s scsibus2 at ahci0: 32 targets cd0 at scsibus2 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable virtio4 at pci0 dev 10 function 0 "Qumranet Virtio 1.x Input" rev 0x01 virtio4: no matching child driver; not configured isa0 at pcib0 isadma0 at isa0 fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 pckbc0 at isa0 port 0x60/5 irq 1 irq 12 pckbd0 at pckbc0 (kbd slot) wskbd0 at pckbd0: console keyboard, using wsdisplay0 pms0 at pckbc0 (aux slot) wsmouse0 at pms0 mux 0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 usb1 at uhci0: USB revision 1.0 uhub1 at usb1 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb2 at uhci1: USB revision 1.0 uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb3 at uhci2: USB revision 1.0 uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1 vmm0 at mainbus0: VMX/EPT uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB Tablet" rev 2.00/0.00 addr 2 uhidev0: iclass 3/0 ums0 at uhidev0: 3 buttons, Z dir wsmouse1 at ums0 mux 0 uhidev1 at uhub0 port 4 configuration 1 interface 0 "QEMU QEMU USB Keyboard" rev 2.00/0.00 addr 3 uhidev1: iclass 3/1 ukbd0 at uhidev1: 8 variable keys, 6 key codes wskbd1 at ukbd0 mux 1 wskbd1: connecting to wsdisplay0 vscsi0 at root scsibus3 at vscsi0: 256 targets softraid0 at root scsibus4 at softraid0: 256 targets root on sd0a (3e5e337a239a0fb2.a) swap on sd0b dump on sd0b WARNING: / was not properly unmounted fd0 at fdc0 drive 1: density unknown
