On Fri, Nov 01, 2019 at 08:14:08AM +0100, Christian Ehrhardt wrote: > Hi everyone, > we've got a bug report recently - on handling qemu .so's through > upgrades - that got me wondering how to best handle it. > After checking with Paolo yesterday that there is no obvious solution > that I missed we agreed this should be brought up on the list for > wider discussion. > Maybe there already is a good best practise out there, or if it > doesn't exist we might want to agree upon one going forward. > Let me outline the case and the ideas brought up so far. > > Case > - You have qemu representing a Guest > - Due to other constraints e.g. PT you can't live migrate (which would > be preferred) > - You haven't used a specific shared object yet - lets say RBD storage > driver as example > - Qemu gets an update, packaging replaces the .so files on disk > - The Qemu process and the .so files on disk now have a mismatch in $buildid > - If you hotplug an RBD device it will fail to load the (now new) .so
What happens when it fails to load ? Does the user get a graceful error message or does QEMU abort ? I'd hope the former. > > On almost any other service than "qemu representing a VM" the answer > is "restart it", some even re-exec in place to keep things up and > running. > > Ideas so far: > a) Modules are checked by build-id, so keep them in a per build-id dir on disk > - qemu could be made looking preferred in -$buildid dir first > - do not remove the packages with .so's on upgrades > - needs a not-too-complex way to detect which buildids running qemu > processes > have for packaging to be able to "autoclean later" > - Needs some dependency juggling for Distro packaging but IMHO can be made > to work if above simple "probing buildid of running qemu" would exist So this needs a bunch of special QEMU hacks in package mgmt tools to prevent the package upgrade & cleanup later. This does not look like a viable strategy to me. > > b) Preload the modules before upgrade > - One could load the .so files before upgrade > - The open file reference will keep the content around even with the > on disk file gone > - lacking a 'load-module' command that would require fake hotplugs > which seems wrong > - Required additional upgrade pre-planning > - kills most benefits of modular code without an actual need for it > being loaded Well there's two benefits to modular approach - Allow a single build to be selectively installed on a host or container image, such that the install disk footprint is reduced - Allow a faster startup such that huge RBD libraries dont slow down startup of VMs not using RBD disks. Preloading the modules before upgrade doesn't have to the second benefit. We just have to make sure the pre loading doesn't impact the VM startup performance. IOW, register a SIGUSR2 handler which preloads all modules it finds on disk. Have a pre-uninstall option on the .so package that sends SIGUSR2 to all QEMU processes. The challenge of course is that signals are async. You might suggest a QMP command, but only 1 process can have the QMP monitor open at any time and that's libvirt. Adding a second QMP monitor instance is possible but kind of gross for this purpose. Another option would be to pre-load the modules during startup, but do it asynchronously, so that its not blocking overall VM startup. eg just before starting the mainloop, spawn a background thread to load all remaining modules. This will potentially degrade performance of the guest CPUs a bit, but avoids the latency spike from being synchronous in the startup path. > c) go back to non modular build > - No :-) > > d) anything else out there? e) Don't do upgrades on a host with running VMs :-) Upgrades can break the running VM even ignoring this particular QEMU module scenario. f) Simply document that if you upgrade with running VMs that some features like hotplug of RBD will become unavialable. Users can then avoid upgrades if that matters to them. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
