Hi David, Thank your, now it's clear.
On Mon, Feb 13, 2017 at 06:16:02PM +0000, Dr. David Alan Gilbert wrote: > * Alexey Perevalov (a.pereva...@samsung.com) wrote: > > Hello David! > > Hi Alexey, > > > I have checked you series with 1G hugepage, but only in 1 Gbit/sec network > > environment. > > Can you show the qemu command line you're using? I'm just trying > to make sure I understand where your hugepages are; running 1G hostpages > across a 1Gbit/sec network for postcopy would be pretty poor - it would take > ~10 seconds to transfer the page. sure -hda ./Ubuntu.img -name PAU,debug-threads=on -boot d -net nic -net user -m 1024 -localtime -nographic -enable-kvm -incoming tcp:0:4444 -object memory-backend-file,id=mem,size=1G,mem-path=/dev/hugepages -mem-prealloc -numa node,memdev=mem -trace events=/tmp/events -chardev socket,id=charmonitor,path=/var/lib/migrate-vm-monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control > > > I started Ubuntu just with console interface and gave to it only 1G of > > RAM, inside Ubuntu I started stress command > > > (stress --cpu 4 --io 4 --vm 4 --vm-bytes 256000000 &) > > in such environment precopy live migration was impossible, it never > > being finished, in this case it infinitely sends pages (it looks like > > dpkg scenario). > > > > Also I modified stress utility > > http://people.seas.harvard.edu/~apw/stress/stress-1.0.4.tar.gz > > due to it wrote into memory every time the same value `Z`. My > > modified version writes every allocation new incremented value. > > I use google's stressapptest normally; although remember to turn > off the bit where it pauses. I decided to use it too stressapptest -s 300 -M 256 -m 8 -W > > > I'm using Arcangeli's kernel only at the destination. > > > > I got controversial results. Downtime for 1G hugepage is close to 2Mb > > hugepage and it took around 7 ms (in 2Mb hugepage scenario downtime was > > around 8 ms). > > I made that opinion by query-migrate. > > {"return": {"status": "completed", "setup-time": 6, "downtime": 6, > > "total-time": 9668, "ram": {"total": 1091379200, "postcopy-requests": 1, > > "dirty-sync-count": 2, "remaining": 0, "mbps": 879.786851, "transferred": > > 1063007296, "duplicate": 7449, "dirty-pages-rate": 0, "skipped": 0, > > "normal-bytes": 1060868096, "normal": 259001}}} > > > > Documentation says about downtime field - measurement unit is ms. > > The downtime measurement field is pretty meaningless for postcopy; it's only > the time from stopping the VM until the point where we tell the destination it > can start running. Meaningful measurements are only from inside the guest > really, or the place latencys. > Maybe improve it by receiving such information from destination? I wish to do that. > > So I traced it (I added additional trace into postcopy_place_page > > trace_postcopy_place_page_start(host, from, pagesize); ) > > > > postcopy_ram_fault_thread_request Request for HVA=7f6dc0000000 > > rb=/objects/mem offset=0 > > postcopy_place_page_start host=0x7f6dc0000000 from=0x7f6d70000000, > > pagesize=40000000 > > postcopy_place_page_start host=0x7f6e0e800000 from=0x55b665969619, > > pagesize=1000 > > postcopy_place_page_start host=0x7f6e0e801000 from=0x55b6659684e8, > > pagesize=1000 > > several pages with 4Kb step ... > > postcopy_place_page_start host=0x7f6e0e817000 from=0x55b6659694f0, > > pagesize=1000 > > > > 4K pages, started from 0x7f6e0e800000 address it's > > vga.ram, /rom@etc/acpi/tables etc. > > > > Frankly saying, right now, I don't have any ideas why hugepage wasn't > > resent. Maybe my expectation of it is wrong as well as understanding ) > > That's pretty much what I expect to see - before you get into postcopy > mode everything is sent as individual 4k pages (in order); once we're > in postcopy mode we send each page no more than once. So you're > huge page comes across once - and there it is. > > > stress utility also duplicated for me value into appropriate file: > > sec_since_epoch.microsec:value > > 1487003192.728493:22 > > 1487003197.335362:23 > > *1487003213.367260:24* > > *1487003238.480379:25* > > 1487003243.315299:26 > > 1487003250.775721:27 > > 1487003255.473792:28 > > > > It mean rewriting 256Mb of memory per byte took around 5 sec, but at > > the moment of migration it took 25 sec. > > right, now this is the thing that's more useful to measure. > That's not too surprising; when it migrates that data is changing rapidly > so it's going to have to pause and wait for that whole 1GB to be transferred. > Your 1Gbps network is going to take about 10 seconds to transfer that > 1GB page - and that's if you're lucky and it saturates the network. > SO it's going to take at least 10 seconds longer than it normally > would, plus any other overheads - so at least 15 seconds. > This is why I say it's a bad idea to use 1GB host pages with postcopy. > Of course it would be fun to find where the other 10 seconds went! > > You might like to add timing to the tracing so you can see the time between > the > fault thread requesting the page and it arriving. > yes, sorry I forgot about timing 20806@1487084818.270993:postcopy_ram_fault_thread_request Request for HVA=7f0280000000 rb=/objects/mem offset=0 20806@1487084818.271038:qemu_loadvm_state_section 8 20806@1487084818.271056:loadvm_process_command com=0x2 len=4 20806@1487084818.271089:qemu_loadvm_state_section 2 20806@1487084823.315919:postcopy_place_page_start host=0x7f0280000000 from=0x7f0240000000, pagesize=40000000 1487084823.315919 - 1487084818.270993 = 5.044926 sec. Machines connected w/o any routers, directly by cable. > > Another one request. > > QEMU could use mem_path in hugefs with share key simultaneously > > (-object > > memory-backend-file,id=mem,size=${mem_size},mem-path=${mem_path},share=on) > > and vm > > in this case will start and will properly work (it will allocate memory > > with mmap), but in case of destination for postcopy live migration > > UFFDIO_COPY ioctl will fail for > > such region, in Arcangeli's git tree there is such prevent check > > (if (!vma_is_shmem(dst_vma) && dst_vma->vm_flags & VM_SHARED). > > Is it possible to handle such situation at qemu? > > Imagine that you had shared memory; what semantics would you like > to see ? What happens to the other process? Honestly, initially, I thought to handle such error, but I quit forgot about vhost-user in ovs-dpdk. > Dave > > > On Mon, Feb 06, 2017 at 05:45:30PM +0000, Dr. David Alan Gilbert wrote: > > > * Dr. David Alan Gilbert (git) (dgilb...@redhat.com) wrote: > > > > From: "Dr. David Alan Gilbert" <dgilb...@redhat.com> > > > > > > > > Hi, > > > > The existing postcopy code, and the userfault kernel > > > > code that supports it, only works for normal anonymous memory. > > > > Kernel support for userfault on hugetlbfs is working > > > > it's way upstream; it's in the linux-mm tree, > > > > You can get a version at: > > > > git://git.kernel.org/pub/scm/linux/kernel/git/andrea/aa.git > > > > on the origin/userfault branch. > > > > > > > > Note that while this code supports arbitrary sized hugepages, > > > > it doesn't make sense with pages above the few-MB region, > > > > so while 2MB is fine, 1GB is probably a bad idea; > > > > this code waits for and transmits whole huge pages, and a > > > > 1GB page would take about 1 second to transfer over a 10Gbps > > > > link - which is way too long to pause the destination for. > > > > > > > > Dave > > > > > > Oops I missed the v2 changes from the message: > > > > > > v2 > > > Flip ram-size summary word/compare individual page size patches around > > > Individual page size comparison is done in ram_load if 'advise' has been > > > received rather than checking migrate_postcopy_ram() > > > Moved discard code into exec.c, reworked ram_discard_range > > > > > > Dave > > > > Thank your, right now it's not necessary to set > > postcopy-ram capability on destination machine. > > > > > > > > > > > Dr. David Alan Gilbert (16): > > > > postcopy: Transmit ram size summary word > > > > postcopy: Transmit and compare individual page sizes > > > > postcopy: Chunk discards for hugepages > > > > exec: ram_block_discard_range > > > > postcopy: enhance ram_block_discard_range for hugepages > > > > Fold postcopy_ram_discard_range into ram_discard_range > > > > postcopy: Record largest page size > > > > postcopy: Plumb pagesize down into place helpers > > > > postcopy: Use temporary for placing zero huge pages > > > > postcopy: Load huge pages in one go > > > > postcopy: Mask fault addresses to huge page boundary > > > > postcopy: Send whole huge pages > > > > postcopy: Allow hugepages > > > > postcopy: Update userfaultfd.h header > > > > postcopy: Check for userfault+hugepage feature > > > > postcopy: Add doc about hugepages and postcopy > > > > > > > > docs/migration.txt | 13 ++++ > > > > exec.c | 83 +++++++++++++++++++++++ > > > > include/exec/cpu-common.h | 2 + > > > > include/exec/memory.h | 1 - > > > > include/migration/migration.h | 3 + > > > > include/migration/postcopy-ram.h | 13 ++-- > > > > linux-headers/linux/userfaultfd.h | 81 +++++++++++++++++++--- > > > > migration/migration.c | 1 + > > > > migration/postcopy-ram.c | 138 > > > > +++++++++++++++++--------------------- > > > > migration/ram.c | 109 ++++++++++++++++++------------ > > > > migration/savevm.c | 32 ++++++--- > > > > migration/trace-events | 2 +- > > > > 12 files changed, 328 insertions(+), 150 deletions(-) > > > > > > > > -- > > > > 2.9.3 > > > > > > > > > > > -- > > > Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK > > > > -- > Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK > -- BR Alexey
