Lévai, Dániel <l...@ecentrum.hu> writes:
> Hi all! > > I have this OpenBSD 7.4 qemu/kvm VM managed by libvirt on an Ubuntu 22.04 > host. > > I started to notice this month that it started to act weird, it seems > like the clock stops every night. I couldn't pinpoint exactly what > caused the change in behavior, the host had two package updates that > raised suspicion: > 2024-01-11 06:51:04 upgrade linux-image-generic-hwe-22.04:amd64 > 6.2.0.39.40~22.04.16 6.5.0.14.14~22.04.7 > 2024-01-12 09:10:36 upgrade libvirt-daemon:amd64 8.0.0-1ubuntu7.7 > 8.0.0-1ubuntu7.8 > > But none of the changelogs /seemed/ relevant. > > Anyway, the symptoms are funny, it always involves the clock stopping/not > working after some period of time. > > When this happens, I cannot login with SSH. The ssh client connects, > it even asks for the private key, but after confirmation it times out. > > The really funny things happen when I log in on the console - that I can do: > > When I try to ping anything from the host, it stops after the first > successful packets (echo/reply) and then hangs (I can CTRL+C). > Interestingly I can ping the VM from the hypervisor host indefinitely, > but running tcpdump on the guest doesn't show anything immediately. In > fact, looking at tcpdump while doing *anything* network related on the > VM or to the VM doesn't result in any output right away. > That being said, after a couple of minutes, output from tcpdump starts > to flood the screen but I cannot say exactly why or when, it just > suddenly happens. > > Running `sleep 1` just hangs. > > When I run `date` consecutively it shows: > Fri Jan 26 04:20:42 CET 2024 > Fri Jan 26 04:20:39 CET 2024 > Fri Jan 26 04:20:40 CET 2024 > Fri Jan 26 04:20:41 CET 2024 > Fri Jan 26 04:20:42 CET 2024 > Fri Jan 26 04:20:43 CET 2024 > Fri Jan 26 04:20:41 CET 2024 > Fri Jan 26 04:20:42 CET 2024 Definitely should not be seeing time moving backwards. That's not good. > > It always works again after a reboot - forced reset, because it cannot shut > down gracefully. > > Originally I was using SP kernel but tried with MP recently too, just out of > curiosity - no luck. > > I found two old posts seemingly related: > https://marc.info/?t=152942296100002&r=1&w=2 > ^^ I don't have that sysctl on the host and that kernel is very old there. > What is your Linux kernel version? > https://www.reddit.com/r/openbsd/comments/13c9nh1/clock_issue_with_vmm_guest_on_73/ > This is on an OpenBSD host, so I can't try that sysctl either. > > This has been set on the guest, though (defaults): > kern.timecounter.tick=1 > kern.timecounter.timestepwarnings=0 > kern.timecounter.hardware=pvclock0 > kern.timecounter.choice=i8254(0) pvclock0(1500) acpitimer0(1000) > > So pvclock should be relying on KVM to properly deal with TSC paravirtualzation. Do you see this issue with Linux guests using kvmclock? (Or do your Linux guests decide on a different clocksource?) > Any clues would be appreciated, > Daniel > > > dmesg: > OpenBSD 7.4 (GENERIC.MP) #1397: Tue Oct 10 09:02:37 MDT 2023 > dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP > real mem = 519929856 (495MB) > avail mem = 484507648 (462MB) > random: good seed from bootblocks > mpath0 at root > scsibus0 at mpath0: 256 targets > mainbus0 at root > bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5960 (10 entries) > bios0: vendor SeaBIOS version "1.15.0-1" date 04/01/2014 > bios0: QEMU Standard PC (i440FX + PIIX, 1996) > acpi0 at bios0: ACPI 1.0 > acpi0: sleep states S5 > acpi0: tables DSDT FACP APIC WAET > acpi0: wakeup devices > acpitimer0 at acpi0: 3579545 Hz, 24 bits > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat > cpu0 at mainbus0: apid 0 (boot processor) > cpu0: 12th Gen Intel(R) Core(TM) i7-12700K, 3609.77 MHz, 06-97-02 > cpu0: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,PKU,WAITPKG,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,IBRS_ALL,SKIP_L1DFL,MDS_NO,IF_PSCHANGE,XSAVEOPT,XSAVEC,XGETBV1,XSAVES > cpu0: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB > 64b/line 16-way L2 cache, 16MB 64b/line 16-way L3 cache > cpu0: smt 0, core 0, package 0 > mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges > cpu0: apic clock running at 1000MHz > cpu1 at mainbus0: apid 1 (application processor) > cpu1: 12th Gen Intel(R) Core(TM) i7-12700K, 3609.78 MHz, 06-97-02 > cpu1: > FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,CLWB,SHA,UMIP,PKU,WAITPKG,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,IBRS_ALL,SKIP_L1DFL,MDS_NO,IF_PSCHANGE,XSAVEOPT,XSAVEC,XGETBV1,XSAVES > cpu1: 32KB 64b/line 8-way D-cache, 32KB 64b/line 8-way I-cache, 4MB > 64b/line 16-way L2 cache, 16MB 64b/line 16-way L3 cache > cpu1: smt 0, core 0, package 1 > ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins > acpiprt0 at acpi0: bus 0 (PCI0) > "ACPI0006" at acpi0 not configured > acpipci0 at acpi0 PCI0 > com0 at acpi0 COM1 addr 0x3f8/0x8 irq 4: ns16550a, 16 byte fifo > acpicmos0 at acpi0 > "PNP0A06" at acpi0 not configured > "PNP0A06" at acpi0 not configured > "PNP0A06" at acpi0 not configured > "QEMU0002" at acpi0 not configured > "ACPI0010" at acpi0 not configured > acpicpu0 at acpi0: C1(@1 halt!) > acpicpu1 at acpi0: C1(@1 halt!) > pvbus0 at mainbus0: KVM > pvclock0 at pvbus0 > pci0 at mainbus0 bus 0 > pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02 > pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00 > pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, > channel 0 wired to compatibility, channel 1 wired to compatibility > pciide0: channel 0 disabled (no drives) > pciide0: channel 1 disabled (no drives) > piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9 > iic0 at piixpm0 > vga1 at pci0 dev 2 function 0 "Qumranet Virtio 1.x GPU" rev 0x01 > wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) > wsdisplay0: screen 1-5 added (80x25, vt100 emulation) > virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00 > vio0 at virtio0: address 52:54:XX:XX:XX:XX > virtio0: msix per-VQ > azalia0 at pci0 dev 4 function 0 "Intel 82801FB HD Audio" rev 0x01: apic 0 > int 11 > azalia0: No codecs found > uhci0 at pci0 dev 5 function 0 "Intel 82801I USB" rev 0x03: apic 0 int 10 > uhci1 at pci0 dev 5 function 1 "Intel 82801I USB" rev 0x03: apic 0 int 10 > uhci2 at pci0 dev 5 function 2 "Intel 82801I USB" rev 0x03: apic 0 int 11 > ehci0 at pci0 dev 5 function 7 "Intel 82801I USB" rev 0x03: apic 0 int 11 > usb0 at ehci0: USB revision 2.0 > uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 > addr 1 > virtio1 at pci0 dev 6 function 0 "Qumranet Virtio Console" rev 0x00 > virtio1: no matching child driver; not configured > virtio2 at pci0 dev 7 function 0 "Qumranet Virtio Storage" rev 0x00 > vioblk0 at virtio2 > scsibus1 at vioblk0: 1 targets > sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, > > sd0: 13312MB, 512 bytes/sector, 27262976 sectors > virtio2: msix per-VQ > virtio3 at pci0 dev 8 function 0 "Qumranet Virtio Memory Balloon" rev 0x00 > viomb0 at virtio3 > virtio3: apic 0 int 11 > ahci0 at pci0 dev 9 function 0 "Intel 82801I AHCI" rev 0x02: apic 0 int 10, > AHCI 1.0 > ahci0: port 0: 1.5Gb/s > scsibus2 at ahci0: 32 targets > cd0 at scsibus2 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable > virtio4 at pci0 dev 10 function 0 "Qumranet Virtio 1.x Input" rev 0x01 > virtio4: no matching child driver; not configured > isa0 at pcib0 > isadma0 at isa0 > fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 > pckbc0 at isa0 port 0x60/5 irq 1 irq 12 > pckbd0 at pckbc0 (kbd slot) > wskbd0 at pckbd0: console keyboard, using wsdisplay0 > pms0 at pckbc0 (aux slot) > wsmouse0 at pms0 mux 0 > pcppi0 at isa0 port 0x61 > spkr0 at pcppi0 > usb1 at uhci0: USB revision 1.0 > uhub1 at usb1 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 > addr 1 > usb2 at uhci1: USB revision 1.0 > uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 > addr 1 > usb3 at uhci2: USB revision 1.0 > uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 > addr 1 > vmm0 at mainbus0: VMX/EPT > uhidev0 at uhub0 port 1 configuration 1 interface 0 "QEMU QEMU USB Tablet" > rev 2.00/0.00 addr 2 > uhidev0: iclass 3/0 > ums0 at uhidev0: 3 buttons, Z dir > wsmouse1 at ums0 mux 0 > uhidev1 at uhub0 port 4 configuration 1 interface 0 "QEMU QEMU USB Keyboard" > rev 2.00/0.00 addr 3 > uhidev1: iclass 3/1 > ukbd0 at uhidev1: 8 variable keys, 6 key codes > wskbd1 at ukbd0 mux 1 > wskbd1: connecting to wsdisplay0 > vscsi0 at root > scsibus3 at vscsi0: 256 targets > softraid0 at root > scsibus4 at softraid0: 256 targets > root on sd0a (3e5e337a239a0fb2.a) swap on sd0b dump on sd0b > WARNING: / was not properly unmounted > fd0 at fdc0 drive 1: density unknown
