On 02/19/2013 11:39 AM, Orit Wasserman wrote:
Do you have some results as to the performance cost on demand pinning has?
Yes, there's a single graph on the QEMU wiki that shows this:
http://wiki.qemu.org/Features/RDMALiveMigration
Pay attention to the last group of bars, labelled "A", "B", and "C".
"A" is called "Chunked Page Registration" (i.e. on-demand pinning in
groups of about 1MB of the surrounding memory pages)
"B" pins everything in advance
On-demand has only a slightly larger cost, but not much.
For the guest memory pages, sending the pages directly without QemuFile (that
does buffering) is better, I would suggest implementing an QemuRDMAFile for
this. It will have a new API for the memory pages (zero copy) so instead of
using qemu_put_buffer we will call qemu_rdma_buffer or it can reimplement
qemu_put_buffer (you need to add offset). As for device state which is sent in
the last phase and is small you can modify the current implementation. (well
Paolo sent patches that are changing this but I think buffering is still an
option) The current migration implementation copies the device state into a
buffer and than send the data from the buffer (QemuBufferedFile). You only need
to pin this buffer, and RDMA it after all the device state was written into it.
Regards, Orit
I like it .......... can you help me understand - how much different is this design than
the "QEMUFileOps" design that Paulo suggested?
Or it basically the same? ....reimplementing qemu_put_buffer/get_buffer() for
RDMA purposes......
Yes it is basically the same :).
you will need the QEMUFileRDMA to store the rdma context (look at
QEMUFileSocket for example) and other specific rdma parameters.
you will the rdma_file_ops (QEMUFileOps) to implement sending guest pages
directly and pinning the buffer that contain the device state.
you won't need to change qemu_put_buffer for the device state but implement a
rdma_put_buffer ops that will pin the buffer and send it.
As for the guest memory pages you have two options:
update qemu_put_buffer/get_buffer
add new QEMUFileOps just for rdma and call it instead of qemu_put_buffer for
guest pages.
Understood.... I have some code reading to do with these pointers, then.
Will have to teach myself everything you just said =)
