On Thu, Nov 07, 2024 at 12:33:17PM +0000, Daniel P. Berrangé wrote: > On Thu, Nov 07, 2024 at 05:35:06PM +0530, Prasad Pandit wrote: > > On Wed, 6 Nov 2024 at 18:41, Fabiano Rosas <faro...@suse.de> wrote: > > > What we're thinking is having an initial exchange of information between > > > src & dst as soon as migration starts and that would sync the > > > capabilities and parameters between both sides. Which would then be > > > followed by a channel establishment phase that would open each necessary > > > channel (according to caps) in order, removing the current ambiguity. > > > > > > > * Isn't that how it works? IIUC, libvirtd(8) sends migration command > > options to the destination and based on that the destination prepares > > for the multifd and/or postcopy migration. In case of 'Postcopy' the > > source sends 'postcopy advise' to the destination to indicate that > > postcopy might follow at the end of precopy. Also, in the discussion > > above Peter mentioned that libvirtd(8) may exchange list of features > > between source and destination to facilitate QMP clients. > > > > * What is the handshake doing differently? (just trying to understand) > > Libvirt does what it does because it has had no other choice, > not because it was good or desirable. > > This kind of handshake really does not belong in libvirt. A number > of exposed migration protocol feature knobs should be considered > private to QEMU only. > > It has the very negative consequence that every time QEMU wants to > provide a new feature in migration, it needs to be plumbed up through > libvirt, and often applications above, and those 3rd party projects > need to be told when & where to use the new features. The 3rd party > developers have their own project dev priorities so may not get > around to enable the new migration features for years, if ever, > undermining the work of QEMU's migration maintainers. > > As examples... > > If we had QEMU self-negotiation of features 10 years ago, everywhere > would already be using multifd out of the box. QEMU would have been > able to self-negotiate use of the new "multifd" protocol, and QEMU > would be well on its way to being able to delete the old single- > threaded migration code. > > Similarly post-copy would have been way easier for apps, QEMU would > auto-negotiate a channel for the post-copy async page fetching. All > migrations would be running with the post-copy feature available. > All that libvirt & apps would have needed was a API to initiate the > switch to post-copy mode. > > Or the hacks QEMU has put in place where we peek at incoming data > on some channels to identify the channel type would not exist. > > > TL;DR: once QEMU can self-negotiate features for migration itself, > the implementation burden for libvirt & applications is greatly > reduced. QEMU migration maintainers will control their own destiny, > able to deliver improvements to users much more quickly, be able > to delete obsolete features more quickly, and be able to make > migration *automatically* enable new features & pick the optimal > defaults on their own expert knowledge, not waitnig for 3rd parties > to pay attention years later. > > Some things will still need work & decisions in libvirt & apps, > but this burden should be reduced compared over the long term. > Ultimately everyone will win.
I'll comment on a few examples above, which I think some of them, even if handshake is ready, may still need mgmt layers to involve.. Multifd and postcopy are the two major features, and they, IMHO, all better need user involvements.. Multifd needs it because it relies on the channel being able to provide >1 channels. It means "| nc XXX > file" can stop working if we turn it on by default silently. We also see generic use case in containers now that when there're dedicated cores for vcpus and "no dedicate" / "one" core setup for qemu hypervisor threads, it means anything like multifd or even block layer iothreads can be pure overheads comparing to one thread / one channel does the job.. in those cases they're better disabled. Postcopy, when enabled manually like right now, has one benefit I can think of: when the user will 100% use postcopy, the failure can happen earlier if dest host doesn't even support it! So it'll generate an error upfront saying "postcopy not supported" before migration starts, comparing to fail later at migrate_start_postcopy, then fail that command, but maybe if so the user may not even start the migration at all, knowing workload too busy to converge. It's pretty common that postcopy is not supported on dest due to the fact that people are over cautious on what userfault could do (which I kind of agree.. it's a tricky but powerful feature), so unprivileged_userfaultfd=0 knob alone can disable it for many hosts, afaiu.. simply because kvm will be the default accel, and qemu needs kernel fault trapping capability to make postcopy work.. which requires unprivileged userfaults that qemu process may not have. Maybe the postcopy-preempt mode is the one closest to what we can enable by default, so if it's not a capability we could auto-enalbe it with/without handshake being there. However there's still the tricky part where it starts to require >1 channels, so if it's the default it could break anyone who has a proxy after the socket, for example, iff the proxy only supports one channel.. similar to multifd, but not completely the same. Mapped-ram, definitely needs user involvements, because it will change the image layout. So.. just to say, maybe we'll still need some mgmt help to explicitly enable many of the features after the handshake work, as the mgmt knows better on what is the context of the VM, rather than risking qemu default configs to fail on some existing but working setups. Though at least libvirt will only need to set caps/params only once for each migration. Thanks, -- Peter Xu
