On 12/08/2025 17:56, Cédric Le Goater wrote:
External email: Use caution opening links or attachments
On 8/12/25 16:08, Avihai Horon wrote:
On 11/08/2025 19:34, Cédric Le Goater wrote:
External email: Use caution opening links or attachments
Hello,
+ Avihai
On 8/11/25 18:02, Kunkun Jiang wrote:
Hi all,
While testing VFIO migration, I encountered an corner scenario case:
VFIO migration will not be aborted when the vfio device of dst-vm
fails to transition from RESUMING to RUNNING state in
vfio_vmstate_change.
I saw the comments in the vfio_vmstate_change but I don't
understand why no action is taken for this situation.
There is error handling in vfio_vmstate_change() :
/*
* Migration should be aborted in this case, but
vm_state_notify()
* currently does not support reporting failures.
*/
migration_file_set_error(ret, local_err);
Hmm, I think this only sets the error on src. On dst we don't have
MigrationState->to_dst_file, so we end up just reporting the error.
But even if we did set it, no one is checking if there is a migration
error after vm_start() is called in process_incoming_migration_bh().
Allowing the live migration process to continue could cause
unrecoverable damage to the VM.
What do you mean by unrecoverable damage to the VM?
If RESUMING->RUNNING transition fails, would a VFIO reset recover the
device and allow the VM to continue operation with damage limited
only to the VFIO device?
In this case, can we directly exit the dst-vm? Through the
return-path mechanism, the src-vm can continue to run.
Looking forward to your reply.
The straightforward solution, as you suggested, is to exit dst upon
error in RESUMING->RUNNING transition and notify about it to src
through the return-path.
However, I am not sure if failing the migration after vm_start() on
dst is a bit late (as we start vCPUs and do migration_block_activate,
etc.).
But I can think of another way to solve this, hopefully simpler.
According to VFIO migration uAPI [1]:
* RESUMING -> STOP
* Leaving RESUMING terminates a data transfer session and
indicates the
* device should complete processing of the data delivered by
write(). The
* kernel migration driver should complete the incorporation of
data written
* to the data transfer FD into the device internal state and perform
* final validity and consistency checking of the new device
state. If the
* user provided data is found to be incomplete, inconsistent, or
otherwise
* invalid, the migration driver must fail the SET_STATE ioctl and
* optionally go to the ERROR state as described below.
So, IIUC, we can add an explicit RESUMING->STOP transition [2] after
the device config is loaded (which is the last data the device is
expected to receive).
If this transition fails, it means something was wrong with
migration, and we can send src an error msg via return-path (and not
continue to vm_start()).
Maybe this approach is less complicated than the first one, and it
will also work if src VM was paused prior migration.
I already tested some POC and it seems to be working (at least with
an artificial error i injected in RESUMING->STOP transition).
Kunkun, can you apply the following diff [3] and check if this solves
the issue?
And in general, what do you think? Should we go with this approach or
do you have other ideas?
Thanks.
[1]
https://elixir.bootlin.com/linux/v6.16/source/include/uapi/linux/vfio.h#L1099
[2] Today RESUMING->STOP is done implicitly by the VFIO driver as
part of RESUMING->RUNNING transition.
[3]
Avihai,
Could you please send an RFC patch with Peter and Fabiano in cc: ?
This will help to discuss the proposal and keep track of the issue.
Yes, of course. But I am a bit busy with other stuff, so it may take me
a few days to do that.
Of course, If anyone would like, feel free to send your own proposal.
Thanks.
Kunkun Jiang,
Could you please share details on your environment ?
Thanks,
C.
