On Fri, Dec 7, 2018 at 7:37 AM Jason Wang <jasow...@redhat.com> wrote: > > > On 2018/12/6 下午8:44, Jason Wang wrote: > > > > On 2018/12/6 下午8:11, Jintack Lim wrote: > >> On Thu, Dec 6, 2018 at 2:33 AM Jason Wang <jasow...@redhat.com> wrote: > >>> > >>> On 2018/12/5 下午10:47, Jintack Lim wrote: > >>>> On Tue, Dec 4, 2018 at 8:30 PM Jason Wang <jasow...@redhat.com> wrote: > >>>>> On 2018/12/5 上午2:37, Jintack Lim wrote: > >>>>>> Hi, > >>>>>> > >>>>>> I'm wondering how the current implementation works when logging > >>>>>> dirty > >>>>>> pages during migration from vhost-net (in kernel) when used vIOMMU. > >>>>>> > >>>>>> I understand how vhost-net logs GPAs when not using vIOMMU. But when > >>>>>> we use vhost with vIOMMU, then shouldn't vhost-net need to log the > >>>>>> translated address (GPA) instead of the address written in the > >>>>>> descriptor (IOVA) ? The current implementation looks like vhost-net > >>>>>> just logs IOVA without translation in vhost_get_vq_desc() in > >>>>>> drivers/vhost/net.c. It seems like QEMU doesn't do any further > >>>>>> translation of the dirty log when syncing. > >>>>>> > >>>>>> I might be missing something. Could somebody shed some light on > >>>>>> this? > >>>>> Good catch. It looks like a bug to me. Want to post a patch for this? > >>>> Thanks for the confirmation. > >>>> > >>>> What would be a good setup to catch this kind of migration bug? I > >>>> tried to observe it in the VM expecting to see network applications > >>>> not getting data correctly on the destination, but it was not > >>>> successful (i.e. the VM on the destination just worked fine.) I didn't > >>>> even see anything going wrong when I disabled the vhost logging > >>>> completely without using vIOMMU. > >>>> > >>>> What I did is I ran multiple network benchmarks (e.g. netperf tcp > >>>> stream and my own one to check correctness of received data) in a VM > >>>> without vhost dirty page logging, and the benchmarks just ran fine in > >>>> the destination. I checked the used ring at the time the VM is stopped > >>>> in the source for migration, and it had multiple descriptors that is > >>>> (probably) not processed in the VM yet. Do you have any insight how it > >>>> could just work and what would be a good setup to catch this? > >>> > >>> According to past experience, it could be reproduced by doing scp from > >>> host to guest during migration. > >>> > >> Thanks. I actually tried that, but didn't see any problem either - I > >> copied a large file during migration from host to guest (the copy > >> continued on the destination), and checked md5 hashes using md5sum, > >> but the copied file had the same checksum as the one in the host. > >> > >> Do you recall what kind of symptom you observed when the dirty pages > >> were not migrated correctly with scp? > > > > > > Yes, the point is to make the migration converge before the end of > > scp (e.g set migration speed to a very big value). If scp end before > > migration, we won't catch the bug. And it's better to do several > > rounds of migration during scp. > > > > Anyway, let me try to reproduce it tomorrow. > > > > Looks like I can reproduce this, scp give the following error to me: > > scp /home/file root@192.168.100.4:/home > file 63% 1301MB 58.1MB/s > 00:12 ETAReceived disconnect from 192.168.100.4: 2: Packet corrupt > lost connection
Thanks for sharing this. I was able to reproduce the bug. I observed different md5sum in the host and the guest after several tries. I didn't observe the disconnect you saw, but the different md5sum is enough to show the bug, I guess. Thanks, Jintack > > FYI, I use the following cli: > > numactl --cpunodebind 0 --membind 0 $qemu_path $img_path \ > -netdev tap,id=hn0,vhost=on \ > -device ioh3420,id=root.1,chassis=1 \ > -device > virtio-net-pci,bus=root.1,netdev=hn0,ats=on,disable-legacy=on,disable-modern=off,iommu_platform=on > \ > -device intel-iommu,device-iotlb=on \ > -M q35 -m 4G -enable-kvm -cpu host -smp 2 $@ > > Thanks > > > > Thanks >
