On Mon, May 11, 2020 at 06:22:47PM +0800, Kirti Wankhede wrote:
>
>
> On 5/9/2020 11:01 AM, Yan Zhao wrote:
> > On Wed, Mar 25, 2020 at 05:09:07AM +0800, Kirti Wankhede wrote:
> >> Added .save_live_pending, .save_live_iterate and
> >> .save_live_complete_precopy
> >> functions. These functions handles pre-copy and stop-and-copy phase.
> >>
> >> In _SAVING|_RUNNING device state or pre-copy phase:
> >> - read pending_bytes. If pending_bytes > 0, go through below steps.
> >> - read data_offset - indicates kernel driver to write data to staging
> >> buffer.
> >> - read data_size - amount of data in bytes written by vendor driver in
> >> migration region.
> > I think we should change the sequence of reading data_size and
> > data_offset. see the next comment below.
> >
> >> - read data_size bytes of data from data_offset in the migration region.
> >> - Write data packet to file stream as below:
> >> {VFIO_MIG_FLAG_DEV_DATA_STATE, data_size, actual data,
> >> VFIO_MIG_FLAG_END_OF_STATE }
> >>
> >> In _SAVING device state or stop-and-copy phase
> >> a. read config space of device and save to migration file stream. This
> >> doesn't need to be from vendor driver. Any other special config state
> >> from driver can be saved as data in following iteration.
> >> b. read pending_bytes. If pending_bytes > 0, go through below steps.
> >> c. read data_offset - indicates kernel driver to write data to staging
> >> buffer.
> >> d. read data_size - amount of data in bytes written by vendor driver in
> >> migration region.
> >> e. read data_size bytes of data from data_offset in the migration region.
> >> f. Write data packet as below:
> >> {VFIO_MIG_FLAG_DEV_DATA_STATE, data_size, actual data}
> >> g. iterate through steps b to f while (pending_bytes > 0)
> >> h. Write {VFIO_MIG_FLAG_END_OF_STATE}
> >>
> >> When data region is mapped, its user's responsibility to read data from
> >> data_offset of data_size before moving to next steps.
> >>
> >> Signed-off-by: Kirti Wankhede <kwankh...@nvidia.com>
> >> Reviewed-by: Neo Jia <c...@nvidia.com>
> >> ---
> >> hw/vfio/migration.c | 245
> >> +++++++++++++++++++++++++++++++++++++++++-
> >> hw/vfio/trace-events | 6 ++
> >> include/hw/vfio/vfio-common.h | 1 +
> >> 3 files changed, 251 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c
> >> index 033f76526e49..ecbeed5182c2 100644
> >> --- a/hw/vfio/migration.c
> >> +++ b/hw/vfio/migration.c
> >> @@ -138,6 +138,137 @@ static int vfio_migration_set_state(VFIODevice
> >> *vbasedev, uint32_t mask,
> >> return 0;
> >> }
> >>
> >> +static void *find_data_region(VFIORegion *region,
> >> + uint64_t data_offset,
> >> + uint64_t data_size)
> >> +{
> >> + void *ptr = NULL;
> >> + int i;
> >> +
> >> + for (i = 0; i < region->nr_mmaps; i++) {
> >> + if ((data_offset >= region->mmaps[i].offset) &&
> >> + (data_offset < region->mmaps[i].offset +
> >> region->mmaps[i].size) &&
> >> + (data_size <= region->mmaps[i].size)) {
> >> + ptr = region->mmaps[i].mmap + (data_offset -
> >> + region->mmaps[i].offset);
> >> + break;
> >> + }
> >> + }
> >> + return ptr;
> >> +}
> >> +
> >> +static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev)
> >> +{
> >> + VFIOMigration *migration = vbasedev->migration;
> >> + VFIORegion *region = &migration->region;
> >> + uint64_t data_offset = 0, data_size = 0;
> >> + int ret;
> >> +
> >> + ret = pread(vbasedev->fd, &data_offset, sizeof(data_offset),
> >> + region->fd_offset + offsetof(struct
> >> vfio_device_migration_info,
> >> + data_offset));
> >> + if (ret != sizeof(data_offset)) {
> >> + error_report("%s: Failed to get migration buffer data offset %d",
> >> + vbasedev->name, ret);
> >> + return -EINVAL;
> >> + }
> >> +
> >> + ret = pread(vbasedev->fd, &data_size, sizeof(data_size),
> >> + region->fd_offset + offsetof(struct
> >> vfio_device_migration_info,
> >> + data_size));
> >> + if (ret != sizeof(data_size)) {
> >> + error_report("%s: Failed to get migration buffer data size %d",
> >> + vbasedev->name, ret);
> >> + return -EINVAL;
> >> + }
> > data_size should be read first, and if it's 0, data_offset will not
> > be read further.
> >
> > the reasons are below:
> > 1. if there's no data region provided by vendor driver, there's no
> > reason to get a valid data_offset, so reading/writing of data_offset
> > should fail. And this should not be treated as a migration error.
> >
> > 2. even if pending_bytes is 0, vfio_save_iterate() is still possible to be
> > called and therefore vfio_save_buffer() is called.
> >
>
> As I mentioned in reply to Alex in:
> https://lists.nongnu.org/archive/html/qemu-devel/2020-05/msg02476.html
>
> With that, vfio_save_iterate() will read pending_bytes if its 0 and then
> if pending_bytes is not 0 then call vfio_save_buffer(). With that your
> above concerns should get resolved.
>
what if pending_bytes is not 0, but vendor driver just does not want to
send data in this iteration? isn't it right to get data_size first before
getting data_offset?
Thanks
Yan
