On Sun, Apr 30, 2023, at 12:43 PM, Brian Conway wrote:
> On Tue, Apr 25, 2023, at 9:47 AM, Dave Voutila wrote:
>> tech@:
>>
>> The below diff splits out virtio device emulation for virtio block and
>> network devices into separate fork+exec'd & pledge(2)'d subprocesses.
>>
>> In order of priority, this diff:
>>
>> 1. Isolates common exploit targets (e.g. emulated network devices) from
>> the rest of the vm process, tightening pledge to "stdio" per device.
>>
>> 2. Increases responsiveness of guest i/o since we no longer have a
>> single thread servicing virtio pci and device emulation.
>>
>> I'd like to land this diff this week, so if you use atypical vmd
>> configurations like:
>>
>> 1. multiple vioblk disks per vm
>> 2. multiple nics per vm
>> 3. send/receive
>> 4. qcow2 base images
>>
>> This diff has lots of info logging enabled by default to help me
>> identify what's breaking, so please reply with log message output if
>> something goes sideways.
>>
>> -dv
>
> Apologies for being late to the party, I'm testing this now via
> snapshots. The host in question was last updated a few weeks ago, i.e.
> before the deluge of recent work. It's an amd64 VMM host running (among
> other things) an i386 7.3 release VM. After updating the host to the
> latest snapshot, things appear to run fine, but I did notice the
> following new messages consistently when powering off (halt -p) the
> i386 VM:
>
> Apr 29 18:41:09 coofun /bsd: uvn_flush: obj=0xfffffd815a44fee0,
> offset=0x43287000. error during pageout.
> Apr 29 18:41:09 coofun /bsd: uvn_flush: WARNING: changes to page may be
> lost!
>
> Apr 30 17:28:26 coofun /bsd: uvn_flush: obj=0xfffffd812ff90c08,
> offset=0x4378c000. error during pageout.
> Apr 30 17:28:26 coofun /bsd: uvn_flush: WARNING: changes to page may be
> lost!
I should have been more clear, these messages are on the amd64 VMM host.
Also, the guest is using a standard raw disk image, so I don't think it
actually meets any of your atypical scenarios 1-4.
Brian
> The guest *appears* to run fine and reboots with a clean filesystem,
> but I also haven't gone through the filesystem with a comb. I have not
> had a chance to spin up an amd64 guest this weekend yet.
>
> Thanks!
>
> Brian Conway
>
> amd64 host dmesg:
>
> OpenBSD 7.3-current (GENERIC.MP) #1168: Fri Apr 28 14:47:06 MDT 2023
> dera...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 4072996864 (3884MB)
> avail mem = 3929907200 (3747MB)
> random: good seed from bootblocks
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 3.2 @ 0x79851000 (17 entries)
> bios0: vendor American Megatrends Inc. version "GB1B 0.04" date
> 10/20/2021
> bios0: BESSTAR TECH LIMITED N40
> efi0 at bios0: UEFI 2.7
> efi0: American Megatrends rev 0x5000d
> acpi0 at bios0: ACPI 6.2
> acpi0: sleep states S0 S3 S4 S5
> acpi0: tables DSDT FACP FPDT FIDT MSDM MCFG SSDT DBG2 DBGP HPET LPIT
> APIC NPKT SSDT SSDT SSDT SSDT SSDT SSDT BGRT TPM2 DMAR WDAT NHLT WSMT
> acpi0: wakeup devices HDAS(S3) XHC_(S4) XDCI(S4) RP01(S4) PXSX(S4)
> RP02(S4) PXSX(S4) RP03(S4) PXSX(S4) RP04(S4) PXSX(S4) RP05(S4) PXSX(S4)
> RP06(S4) PXSX(S4)
> acpitimer0 at acpi0: 3579545 Hz, 32 bits
> acpimcfg0 at acpi0
> acpimcfg0: addr 0xe0000000, bus 0-255
> acpihpet0 at acpi0: 19200000 Hz
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Celeron(R) N4020 CPU @ 1.10GHz, 1095.50 MHz, 06-7a-08
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,SMEP,ERMS,MPX,RDSEED,SMAP,CLFLUSHOPT,PT,SHA,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
> cpu0: 24KB 64b/line 6-way D-cache, 32KB 64b/line 8-way I-cache, 4MB
> 64b/line 16-way L2 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> cpu0: apic clock running at 19MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.0.2.4.2.1.1, IBE
> cpu1 at mainbus0: apid 2 (application processor)
> cpu1: Intel(R) Celeron(R) N4020 CPU @ 1.10GHz, 1096.97 MHz, 06-7a-08
> cpu1:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,SMEP,ERMS,MPX,RDSEED,SMAP,CLFLUSHOPT,PT,SHA,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
> cpu1: 24KB 64b/line 6-way D-cache, 32KB 64b/line 8-way I-cache, 4MB
> 64b/line 16-way L2 cache
> cpu1: smt 0, core 1, package 0
> ioapic0 at mainbus0: apid 1 pa 0xfec00000, version 20, 120 pins
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus 4 (RP01)
> acpiprt2 at acpi0: bus 5 (RP02)
> acpiprt3 at acpi0: bus 1 (RP03)
> acpiprt4 at acpi0: bus 2 (RP04)
> acpiprt5 at acpi0: bus 3 (RP05)
> acpiprt6 at acpi0: bus -1 (RP06)
> acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
> "ESSX8336" at acpi0 not configured
> acpicmos0 at acpi0
> glkgpio0 at acpi0 GPO1 uid 1 addr 0xd0c40000/0xcef irq 14, 80 pins
> glkgpio1 at acpi0 GPO0 uid 2 addr 0xd0c50000/0xaff irq 14, 80 pins
> glkgpio2 at acpi0 GPO2 uid 3 addr 0xd0c90000/0x7bf irq 15, 20 pins
> glkgpio3 at acpi0 GPO3 uid 4 addr 0xd0c80000/0x82f irq 14, 35 pins
> acpibtn0 at acpi0: PWRB
> "INT33A1" at acpi0 not configured
> tpm0 at acpi0 TPM_ 2.0 (CRB) addr 0xfed40000/0x5000, device 0x00000000
> rev 0x0
> "PNP0C0B" at acpi0 not configured
> acpipwrres0 at acpi0: DRST
> acpipwrres1 at acpi0: DRST
> acpipwrres2 at acpi0: DRST
> acpipwrres3 at acpi0: DRST
> acpipwrres4 at acpi0: DRST
> acpipwrres5 at acpi0: DRST
> acpipwrres6 at acpi0: WRST
> acpicpu0 at acpi0: C3(10@150 mwait.1@0x60), C2(10@50 mwait.1@0x21),
> C1(1000@1 mwait.1@0x1), PSS
> acpicpu1 at acpi0: C3(10@150 mwait.1@0x60), C2(10@50 mwait.1@0x21),
> C1(1000@1 mwait.1@0x1), PSS
> acpipwrres7 at acpi0: FN00, resource for FAN0
> acpitz0 at acpi0: critical temperature is 95 degC
> acpivideo0 at acpi0: GFX0
> acpivout0 at acpivideo0: DD1F
> cpu0: Enhanced SpeedStep 1095 MHz: speeds: 1101, 1100, 1000, 900, 800
> MHz
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel Gemini Lake Host" rev 0x06
> "Intel Gemini Lake DPTF" rev 0x06 at pci0 dev 0 function 1 not
> configured
> inteldrm0 at pci0 dev 2 function 0 "Intel UHD Graphics 600" rev 0x06
> drm0 at inteldrm0
> inteldrm0: msi, GEMINILAKE, gen 9
> azalia0 at pci0 dev 14 function 0 "Intel Gemini Lake HD Audio" rev
> 0x06: msi
> azalia0: no supported codecs
> "Intel Gemini Lake MEI" rev 0x06 at pci0 dev 15 function 0 not
> configured
> ahci0 at pci0 dev 18 function 0 "Intel Gemini Lake AHCI" rev 0x06: msi,
> AHCI 1.3.1
> ahci0: PHY offline on port 0
> ahci0: port 1: 6.0Gb/s
> scsibus1 at ahci0: 32 targets
> sd0 at scsibus1 targ 1 lun 0: <ATA, NT-256 2242, SN11>
> naa.50000000000027b6
> sd0: 244198MB, 512 bytes/sector, 500118192 sectors, thin
> ppb0 at pci0 dev 19 function 0 "Intel Gemini Lake PCIE" rev 0xf6: msi
> pci1 at ppb0 bus 1
> iwm0 at pci1 dev 0 function 0 "Intel AC 3165" rev 0x79, msi
> ppb1 at pci0 dev 19 function 1 "Intel Gemini Lake PCIE" rev 0xf6: msi
> pci2 at ppb1 bus 2
> re0 at pci2 dev 0 function 0 "Realtek 8168" rev 0x15: RTL8168H/8111H
> (0x5400), msi, address 84:47:09:09:b7:8e
> rgephy0 at re0 phy 7: RTL8251 PHY, rev. 0
> ppb2 at pci0 dev 19 function 2 "Intel Gemini Lake PCIE" rev 0xf6: msi
> pci3 at ppb2 bus 3
> ppb3 at pci0 dev 20 function 0 "Intel Gemini Lake PCIE" rev 0xf6
> pci4 at ppb3 bus 4
> ppb4 at pci0 dev 20 function 1 "Intel Gemini Lake PCIE" rev 0xf6
> pci5 at ppb4 bus 5
> xhci0 at pci0 dev 21 function 0 "Intel Gemini Lake xHCI" rev 0x06: msi,
> xHCI 1.0
> usb0 at xhci0: USB revision 3.0
> uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev
> 3.00/1.00 addr 1
> dwiic0 at pci0 dev 22 function 0 "Intel Gemini Lake I2C" rev 0x06: apic
> 1 int 27
> iic0 at dwiic0
> "ESSX8336" at iic0 addr 0x11 not configured
> dwiic1 at pci0 dev 22 function 3 "Intel Gemini Lake I2C" rev 0x06: apic
> 1 int 30
> iic1 at dwiic1
> sdhc0 at pci0 dev 28 function 0 "Intel Gemini Lake eMMC" rev 0x06: apic
> 1 int 39
> sdhc0: SDHC 3.0, 200 MHz base clock
> sdmmc0 at sdhc0: 8-bit, sd high-speed, mmc high-speed, ddr52, dma
> pcib0 at pci0 dev 31 function 0 "Intel Gemini Lake LPC" rev 0x06
> ichiic0 at pci0 dev 31 function 1 "Intel Gemini Lake SMBus" rev 0x06:
> apic 1 int 20
> iic2 at ichiic0
> isa0 at pcib0
> isadma0 at isa0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> vmm0 at mainbus0: VMX/EPT
> efifb at mainbus0 not configured
> scsibus2 at sdmmc0: 2 targets, initiator 0
> sd1 at scsibus2 targ 1 lun 0: <SD/MMC, SCA64G, 0000> removable
> sd1: 59648MB, 512 bytes/sector, 122159104 sectors
> uhidev0 at uhub0 port 2 configuration 1 interface 0 "Logitech USB
> Keyboard" rev 1.10/64.00 addr 2
> uhidev0: iclass 3/1
> ukbd0 at uhidev0: 8 variable keys, 6 key codes
> wskbd0 at ukbd0: console keyboard
> uhidev1 at uhub0 port 2 configuration 1 interface 1 "Logitech USB
> Keyboard" rev 1.10/64.00 addr 2
> uhidev1: iclass 3/0, 3 report ids
> ucc0 at uhidev1 reportid 1: 2 usages, 3 keys, enum
> wskbd1 at ucc0 mux 1
> uhid0 at uhidev1 reportid 2: input=1, output=0, feature=0
> ucc1 at uhidev1 reportid 3: 21 usages, 14 keys, enum
> wskbd2 at ucc1 mux 1
> ugen0 at uhub0 port 6 "Generic USB2.0-CRW" rev 2.00/39.60 addr 3
> ugen1 at uhub0 port 7 "Intel Bluetooth" rev 2.00/0.01 addr 4
> vscsi0 at root
> scsibus3 at vscsi0: 256 targets
> softraid0 at root
> scsibus4 at softraid0: 256 targets
> root on sd0a (b6b9cb10be699de6.a) swap on sd0b dump on sd0b
> drm:pid0:intel_step_init *WARNING* [drm] Unknown revid 0x06
> drm:pid0:intel_dp_aux_wait_done *ERROR* [drm] *ERROR* AUX C/DDI C/PHY
> C: did not complete or timeout within 10ms (status 0xad4002ff)
> inteldrm0: 1920x1080, 32bpp
> wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation), using
> wskbd0
> wskbd1: connecting to wsdisplay0
> wskbd2: connecting to wsdisplay0
> wsdisplay0: screen 1-5 added (std, vt100 emulation)
> iwm0: hw rev 0x210, fw ver 29.4063824552.0, address a4:42:3b:34:d2:80
> uid 0 on /tmp: file system full
> uvn_flush: obj=0xfffffd812ff90c08, offset=0x4378c000. error during
> pageout.
> uvn_flush: WARNING: changes to page may be lost!
>
> i386 guest dmesg:
>
> OpenBSD 7.3 (GENERIC) #1: Thu Mar 30 11:37:43 UTC 2023
>
> bcon...@b73-i386.int.rcesoftware.com:/usr/src/sys/arch/i386/compile/GENERIC
> real mem = 1341657088 (1279MB)
> avail mem = 1300058112 (1239MB)
> random: good seed from bootblocks
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: date 06/23/99, BIOS32 rev. 0 @ 0xfefc2, SMBIOS rev.
> 2.4 @ 0xf3740 (10 entries)
> bios0: vendor SeaBIOS version "1.14.0p3-OpenBSD-vmm" date 01/01/2011
> bios0: OpenBSD VMM
> acpi at bios0 function 0x0 not configured
> pcibios at bios0 function 0x1a not configured
> bios0: ROM list: 0xef000/0x1000!
> cpu0 at mainbus0: (uniprocessor)
> cpu0: Intel(R) Celeron(R) N4020 CPU @ 1.10GHz ("GenuineIntel"
> 686-class) 24 MHz, 06-7a-08
> cpu0:
> FPU,V86,DE,PSE,TSC,MSR,PAE,CX8,SEP,PGE,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,RDRAND,HV,NXE,PAGE1GB,LONG,LAHF,3DNOWP,ITSC,FSGSBASE,SMEP,ERMS,RDSEED,SMAP,CLFLUSHOPT,SHA,UMIP,MD_CLEAR,MELTDOWN
> pvbus0 at mainbus0: OpenBSD
> pvclock0 at pvbus0
> pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
> pchb0 at pci0 dev 0 function 0 "OpenBSD VMM Host" rev 0x00
> virtio0 at pci0 dev 1 function 0 "Qumranet Virtio RNG" rev 0x00
> viornd0 at virtio0
> virtio0: irq 3
> virtio1 at pci0 dev 2 function 0 "Qumranet Virtio Network" rev 0x00
> vio0 at virtio1: address fe:e1:bb:d1:21:46
> virtio1: irq 5
> virtio2 at pci0 dev 3 function 0 "Qumranet Virtio Storage" rev 0x00
> vioblk0 at virtio2
> scsibus1 at vioblk0: 1 targets
> sd0 at scsibus1 targ 0 lun 0: <VirtIO, Block Device, >
> sd0: 20480MB, 512 bytes/sector, 41943040 sectors
> virtio2: irq 6
> virtio3 at pci0 dev 4 function 0 "OpenBSD VMM Control" rev 0x00
> vmmci0 at virtio3
> virtio3: irq 7
> isa0 at mainbus0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns8250, no fifo
> com0: console
> pcdisplay0 at isa0 port 0x3d0/16 iomem 0xb8000/32768
> wsdisplay0 at pcdisplay0 mux 1: console (80x25, vt100 emulation)
> npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
> vscsi0 at root
> scsibus2 at vscsi0: 256 targets
> softraid0 at root
> scsibus3 at softraid0: 256 targets
> root on sd0a (7b376d3efe69a188.a) swap on sd0b dump on sd0b
