[Qemu-devel] [RFC v5 20/26] qmp: support out-of-band (oob) execution

2017-12-04 Thread Peter Xu
Having "allow-oob" to true for a command does not mean that this command will always be run in out-of-band mode. The out-of-band quick path will only be executed if we specify the extra "run-oob" flag when sending the QMP request: { "execute": "command-that-allows-oob", "arguments": {

[Qemu-devel] [RFC v5 21/26] qmp: isolate responses into io thread

2017-12-04 Thread Peter Xu
For those monitors who has enabled IO thread, we'll offload the responding procedure into IO thread. The main reason is that chardev is not thread safe, and we need to do all the read/write IOs in the same thread. For use_io_thr=true monitors, that thread is the IO thread. We do this isolation i

[Qemu-devel] [RFC v5 19/26] qapi: introduce new cmd option "allow-oob"

2017-12-04 Thread Peter Xu
Here "oob" stands for "Out-Of-Band". When "allow-oob" is set, it means the command allows out-of-band execution. The "oob" idea is proposed by Markus Armbruster in following thread: https://lists.gnu.org/archive/html/qemu-devel/2017-09/msg02057.html This new "allow-oob" boolean will be expose

[Qemu-devel] [RFC v5 23/26] qmp: add command "x-oob-test"

2017-12-04 Thread Peter Xu
This command is only used to test OOB functionality. It should not be used for any other purposes. Signed-off-by: Peter Xu --- qapi-schema.json | 18 ++ qmp.c| 16 2 files changed, 34 insertions(+) diff --git a/qapi-schema.json b/qapi-schema.json in

[Qemu-devel] [RFC v5 24/26] docs: update QMP documents for OOB commands

2017-12-04 Thread Peter Xu
Update both the developer and spec for the new QMP OOB (Out-Of-Band) command. Signed-off-by: Peter Xu --- docs/devel/qapi-code-gen.txt | 51 +++- docs/interop/qmp-spec.txt| 49 -- 2 files changed, 89 insertions(+

[Qemu-devel] [RFC v5 26/26] tests: qmp-test: add oob test

2017-12-04 Thread Peter Xu
Test the new OOB capability. Here we used the new "x-oob-test" command. Firstly, we send a lock=true and oob=false command to hang the main thread. Then send another lock=false and oob=true command (which will be run inside parser this time) to free that hanged command. Signed-off-by: Peter Xu

[Qemu-devel] [RFC v5 25/26] tests: qmp-test: verify command batching

2017-12-04 Thread Peter Xu
OOB introduced DROP event for flow control. This should not affect old QMP clients. Add a command batching check to make sure of it. Signed-off-by: Peter Xu --- tests/qmp-test.c | 19 +++ 1 file changed, 19 insertions(+) diff --git a/tests/qmp-test.c b/tests/qmp-test.c index 2

Re: [Qemu-devel] [PATCH v3] rcu: reduce more than 7MB heap memory by malloc_trim()

2017-12-04 Thread Yang Zhong
On Mon, Dec 04, 2017 at 08:26:29PM +0800, Shannon Zhao wrote: > Hi Yang, > > On 2017/12/4 20:03, Yang Zhong wrote: > > On Fri, Dec 01, 2017 at 01:52:49PM +0100, Paolo Bonzini wrote: > >> > On 01/12/2017 11:56, Yang Zhong wrote: > >>> > > This issue should be caused by much times of system call b

[Qemu-devel] [PATCH v6 4/4] contrib/vhost-user-blk: introduce a vhost-user-blk sample application

2017-12-04 Thread Changpeng Liu
This commit introcudes a vhost-user-blk backend device, it uses UNIX domain socket to communicate with QEMU. The vhost-user-blk sample application should be used with QEMU vhost-user-blk-pci device. To use it, complie with: make vhost-user-blk and start like this: vhost-user-blk -b /dev/sdb -s /p

[Qemu-devel] [PATCH v6 0/4] Introduce a new vhost-user-blk host device to QEMU

2017-12-04 Thread Changpeng Liu
Although virtio scsi specification was designed as a replacement for virtio_blk, there are still many users using virtio_blk. Qemu 2.9 introduced a new device vhost user scsi which can process I/O in user space for virtio_scsi, this commit introduces a new vhost user block host device, which can su

[Qemu-devel] [PATCH v6 1/4] vhost-user: add new vhost user messages to support virtio config space

2017-12-04 Thread Changpeng Liu
Add VHOST_USER_GET_CONFIG/VHOST_USER_SET_CONFIG messages which can be used for live migration of vhost user devices, also vhost user devices can benefit from the messages to get/set virtio config space from/to the I/O target. For the purpose to support virtio config space change, VHOST_USER_SET_CON

[Qemu-devel] [PATCH v6 2/4] vhost-user-blk: introduce a new vhost-user-blk host device

2017-12-04 Thread Changpeng Liu
This commit introduces a new vhost-user device for block, it uses a chardev to connect with the backend, same with Qemu virito-blk device, Guest OS still uses the virtio-blk frontend driver. To use it, start QEMU with command line like this: qemu-system-x86_64 \ -chardev socket,id=char0,path=

[Qemu-devel] [PATCH v6 3/4] contrib/libvhost-user: enable virtio config space messages

2017-12-04 Thread Changpeng Liu
Enable VHOST_USER_GET_CONFIG/VHOST_USER_SET_CONFIG/VHOST_USER_SET_CONFIG_FD messages in libvhost-user library, users can implement their own I/O target based on the library. This enable the virtio config space delivered between Qemu host device and the I/O target, also event notifier is added in ca

[Qemu-devel] About the light VM solution!

2017-12-04 Thread Yang Zhong
Hello all, As you know, AWS has decided to switch to KVM in their clouds. This news make almost all china CSPs(clouds service provider) pay more attention on KVM/Qemu, especially light VM solution. Below are intel solution for light VM, qemu-lite. http://events.linuxfoundation.org/sites/events/

Re: [Qemu-devel] [for-2.12 0/7] PCI cleanups

2017-12-04 Thread Fam Zheng
On Tue, 12/05 16:11, David Gibson wrote: > On Tue, Dec 05, 2017 at 01:05:58PM +0800, Fam Zheng wrote: > > On Tue, 12/05 06:49, Michael S. Tsirkin wrote: > > > On Wed, Nov 29, 2017 at 05:18:47PM +0800, Fam Zheng wrote: > > > > On Wed, 11/29 01:02, no-re...@patchew.org wrote: > > > > > /tmp/cc3Czn0R.

[Qemu-devel] [PATCH v5 03/28] migration: provide postcopy_fault_thread_notify()

2017-12-04 Thread Peter Xu
A general helper to notify the fault thread. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/postcopy-ram.c | 35 --- migration/postcopy-ram.h | 2 ++ 2 files changed, 22 insertions(+), 15 deletions(-) diff --git a/migration/postcopy-r

[Qemu-devel] [PATCH v5 01/28] migration: better error handling with QEMUFile

2017-12-04 Thread Peter Xu
If the postcopy down due to some reason, we can always see this on dst: qemu-system-x86_64: RP: Received invalid message 0x length 0x However in most cases that's not the real issue. The problem is that qemu_get_be16() has no way to show whether the returned data is valid or not, and we

[Qemu-devel] [PATCH v5 04/28] migration: new postcopy-pause state

2017-12-04 Thread Peter Xu
Introducing a new state "postcopy-paused", which can be used when the postcopy migration is paused. It is targeted for postcopy network failure recovery. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/migration.c | 2 ++ qapi/migration.json | 5 - 2 files change

[Qemu-devel] [PATCH v5 00/28] Migration: postcopy failure recovery

2017-12-04 Thread Peter Xu
Tree is pushed here for better reference and testing (online tree includes monitor OOB series): https://github.com/xzpeter/qemu/tree/postcopy-recover-all This version removed quite a few patches related to migrate-incoming, instead I introduced a new command "migrate-recover" to trigger the rec

[Qemu-devel] [PATCH v5 13/28] migration: new state "postcopy-recover"

2017-12-04 Thread Peter Xu
Introducing new migration state "postcopy-recover". If a migration procedure is paused and the connection is rebuilt afterward successfully, we'll switch the source VM state from "postcopy-paused" to the new state "postcopy-recover", then we'll do the resume logic in the migration thread (along wit

[Qemu-devel] [PATCH v5 08/28] migration: allow send_rq to fail

2017-12-04 Thread Peter Xu
We will not allow failures to happen when sending data from destination to source via the return path. However it is possible that there can be errors along the way. This patch allows the migrate_send_rp_message() to return error when it happens, and further extended it to migrate_send_rp_req_page

[Qemu-devel] [PATCH v5 02/28] migration: reuse mis->userfault_quit_fd

2017-12-04 Thread Peter Xu
It was only used for quitting the page fault thread before. Let it be something more useful - now we can use it to notify a "wake" for the page fault thread (for any reason), and it only means "quit" if the fault_thread_quit is set. Since we changed what it does, renaming it to userfault_event_fd.

[Qemu-devel] [PATCH v5 06/28] migration: allow dst vm pause on postcopy

2017-12-04 Thread Peter Xu
When there is IO error on the incoming channel (e.g., network down), instead of bailing out immediately, we allow the dst vm to switch to the new POSTCOPY_PAUSE state. Currently it is still simple - it waits the new semaphore, until someone poke it for another attempt. One note is that here on ram

[Qemu-devel] [PATCH v5 05/28] migration: implement "postcopy-pause" src logic

2017-12-04 Thread Peter Xu
Now when network down for postcopy, the source side will not fail the migration. Instead we convert the status into this new paused state, and we will try to wait for a rescue in the future. If a recovery is detected, migration_thread() will reset its local variables to prepare for that. Reviewed

[Qemu-devel] [PATCH v5 18/28] migration: new message MIG_RP_MSG_RESUME_ACK

2017-12-04 Thread Peter Xu
Creating new message to reply for MIG_CMD_POSTCOPY_RESUME. One uint32_t is used as payload to let the source know whether destination is ready to continue the migration. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/migration.c | 37 +

[Qemu-devel] [PATCH v5 14/28] migration: wakeup dst ram-load-thread for recover

2017-12-04 Thread Peter Xu
On the destination side, we cannot wake up all the threads when we got reconnected. The first thing to do is to wake up the main load thread, so that we can continue to receive valid messages from source again and reply when needed. At this point, we switch the destination VM state from postcopy-p

[Qemu-devel] [PATCH v5 12/28] migration: rebuild channel on source

2017-12-04 Thread Peter Xu
This patch detects the "resume" flag of migration command, rebuild the channels only if the flag is set. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/migration.c | 92 ++- 1 file changed, 69 insertions(+), 23 deletions

[Qemu-devel] [PATCH v5 07/28] migration: allow src return path to pause

2017-12-04 Thread Peter Xu
Let the thread pause for network issues. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/migration.c | 35 +-- migration/migration.h | 1 + migration/trace-events | 2 ++ 3 files changed, 36 insertions(+), 2 deletions(-) diff --git

[Qemu-devel] [PATCH v5 21/28] migration: setup ramstate for resume

2017-12-04 Thread Peter Xu
After we updated the dirty bitmaps of ramblocks, we also need to update the critical fields in RAMState to make sure it is ready for a resume. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/ram.c| 37 - migration/trace-event

[Qemu-devel] [PATCH v5 19/28] migration: introduce SaveVMHandlers.resume_prepare

2017-12-04 Thread Peter Xu
This is hook function to be called when a postcopy migration wants to resume from a failure. For each module, it should provide its own recovery logic before we switch to the postcopy-active state. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- include/migration/register.h | 2

[Qemu-devel] [PATCH v5 15/28] migration: new cmd MIG_CMD_RECV_BITMAP

2017-12-04 Thread Peter Xu
Add a new vm command MIG_CMD_RECV_BITMAP to request received bitmap for one ramblock. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/savevm.c | 61 ++ migration/savevm.h | 1 + migration/trace-events | 2 ++ 3

[Qemu-devel] [PATCH v5 20/28] migration: synchronize dirty bitmap for resume

2017-12-04 Thread Peter Xu
This patch implements the first part of core RAM resume logic for postcopy. ram_resume_prepare() is provided for the work. When the migration is interrupted by network failure, the dirty bitmap on the source side will be meaningless, because even the dirty bit is cleared, it is still possible that

[Qemu-devel] [PATCH v5 26/28] migration: allow migrate_cancel to pause postcopy

2017-12-04 Thread Peter Xu
It was allowed in the past to even cancel a postcopy migration, but it does not really make sense, and no one should be using it, since cancelling a migration during postcopy means crashing the VM at no time. Let's just use re-use this command as a way to pause the postcopy migration when we detec

[Qemu-devel] [PATCH v5 09/28] migration: allow fault thread to pause

2017-12-04 Thread Peter Xu
Allows the fault thread to stop handling page faults temporarily. When network failure happened (and if we expect a recovery afterwards), we should not allow the fault thread to continue sending things to source, instead, it should halt for a while until the connection is rebuilt. When the dest ma

[Qemu-devel] [PATCH v5 16/28] migration: new message MIG_RP_MSG_RECV_BITMAP

2017-12-04 Thread Peter Xu
Introducing new return path message MIG_RP_MSG_RECV_BITMAP to send received bitmap of ramblock back to source. This is the reply message of MIG_CMD_RECV_BITMAP, it contains not only the header (including the ramblock name), and it was appended with the whole ramblock received bitmap on the destina

[Qemu-devel] [PATCH v5 24/28] migration: init dst in migration_object_init too

2017-12-04 Thread Peter Xu
Though we may not need it, now we init both the src/dst migration objects in migration_object_init() so that even incoming migration object would be thread safe (it was not). Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/migration.c | 28 +++-

Re: [Qemu-devel] [virtio-dev] [PATCH v3 0/7] Vhost-pci for inter-VM communication

2017-12-04 Thread Jason Wang
On 2017年12月05日 11:33, Wei Wang wrote: Vhost-pci is a point-to-point based inter-VM communication solution. This patch series implements the vhost-pci-net device setup and emulation. The device is implemented as a virtio device, and it is set up via the vhost-user protocol to get the neessary in

[Qemu-devel] [PATCH v5 10/28] qmp: hmp: add migrate "resume" option

2017-12-04 Thread Peter Xu
It will be used when we want to resume one paused migration. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- hmp-commands.hx | 7 --- hmp.c | 4 +++- migration/migration.c | 2 +- qapi/migration.json | 5 - 4 files changed, 12 insertions(+), 6 del

[Qemu-devel] [PATCH v5 23/28] migration: free SocketAddress where allocated

2017-12-04 Thread Peter Xu
Freeing the SocketAddress struct in socket_start_incoming_migration is slightly confusing. Let's free the address in the same context where we allocated it. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/socket.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(

[Qemu-devel] [PATCH v5 27/28] qmp/migration: new command migrate-recover

2017-12-04 Thread Peter Xu
The first allow-oob=true command. It's used on destination side when the postcopy migration is paused and ready for a recovery. After execution, a new migration channel will be established for postcopy to continue. Signed-off-by: Peter Xu --- migration/migration.c | 18 ++ qapi

[Qemu-devel] [PATCH v5 11/28] migration: pass MigrationState to migrate_init()

2017-12-04 Thread Peter Xu
Let the callers take the object, then pass it to migrate_init(). Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/migration.c | 7 ++- migration/migration.h | 2 +- migration/savevm.c| 5 - 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/migra

[Qemu-devel] [PATCH v5 17/28] migration: new cmd MIG_CMD_POSTCOPY_RESUME

2017-12-04 Thread Peter Xu
Introducing this new command to be sent when the source VM is ready to resume the paused migration. What the destination does here is basically release the fault thread to continue service page faults. Reviewed-by: Dr. David Alan Gilbert Signed-off-by: Peter Xu --- migration/savevm.c | 35

Re: [Qemu-devel] [PATCH v5 00/28] Migration: postcopy failure recovery

2017-12-04 Thread Peter Xu
On Tue, Dec 05, 2017 at 02:52:39PM +0800, Peter Xu wrote: > Tree is pushed here for better reference and testing (online tree > includes monitor OOB series): > > https://github.com/xzpeter/qemu/tree/postcopy-recover-all Sorry, now this series is depending on the OOB series. Hello, Patchew, hop

[Qemu-devel] [PATCH v2] hw/ide: Remove duplicated definitions from ahci_internal.h

2017-12-04 Thread Thomas Huth
The same definitions can also be found in include/hw/ide/ahci.h so let's remove these #defines from ahci_internal.h. Signed-off-by: Thomas Huth --- v2: Also remove TYPE_ICH9_AHCI as suggested by John hw/ide/ahci_internal.h | 12 1 file changed, 12 deletions(-) diff --git a/hw/ide

Re: [Qemu-devel] [virtio-dev] [PATCH v3 0/7] Vhost-pci for inter-VM communication

2017-12-04 Thread Wei Wang
On 12/05/2017 03:01 PM, Jason Wang wrote: On 2017年12月05日 11:33, Wei Wang wrote: Vhost-pci is a point-to-point based inter-VM communication solution. This patch series implements the vhost-pci-net device setup and emulation. The device is implemented as a virtio device, and it is set up via t

[Qemu-devel] [PATCH v5 22/28] migration: final handshake for the resume

2017-12-04 Thread Peter Xu
Finish the last step to do the final handshake for the recovery. First source sends one MIG_CMD_RESUME to dst, telling that source is ready to resume. Then, dest replies with MIG_RP_MSG_RESUME_ACK to source, telling that dest is ready to resume (after switch to postcopy-active state). When sourc

[Qemu-devel] [PATCH v5 25/28] io: let watcher of the channel run in same ctx

2017-12-04 Thread Peter Xu
Per-thread gcontext is only used in IOThread (please refer to callers of g_main_context_push_thread_default), so this patch only affects anything that will be run in an IOThread. It lets the watcher object be run in the same context as the caller that added the watcher. This patch is critical to

[Qemu-devel] [PATCH v5 28/28] hmp/migration: add migrate_recover command

2017-12-04 Thread Peter Xu
Sister command to migrate-recover in QMP. Signed-off-by: Peter Xu --- hmp-commands.hx | 13 + hmp.c | 10 ++ hmp.h | 1 + 3 files changed, 24 insertions(+) diff --git a/hmp-commands.hx b/hmp-commands.hx index 32fdd52212..79e28c5228 100644 --- a/hmp-comma

[Qemu-devel] [PATCH for-2.12] scsi-block: Add share-rw option

2017-12-04 Thread Fam Zheng
Scsi-block doesn't use the DEFINE_BLOCK_PROPERTIES() macro so it didn't gain the share-rw back when it was added to all other storage devices. This option is meaningful here, and need to be used when attaching a shared storage to guest. Signed-off-by: Fam Zheng --- hw/scsi/scsi-disk.c | 1 + 1 f

Re: [Qemu-devel] [virtio-dev] [PATCH v3 0/7] Vhost-pci for inter-VM communication

2017-12-04 Thread Jason Wang
On 2017年12月05日 15:15, Wei Wang wrote: On 12/05/2017 03:01 PM, Jason Wang wrote: On 2017年12月05日 11:33, Wei Wang wrote: Vhost-pci is a point-to-point based inter-VM communication solution. This patch series implements the vhost-pci-net device setup and emulation. The device is implemented a

[Qemu-devel] [PATCH] MAITAINERS: List Fam Zheng as reviewer for SCSI patches

2017-12-04 Thread Fam Zheng
Just so that I notice those patches more easily. Signed-off-by: Fam Zheng --- MAINTAINERS | 2 ++ 1 file changed, 2 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 0255113470..64a138c380 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1000,6 +1000,7 @@ T: git git://github.com/jasowang

Re: [Qemu-devel] [PATCH v3 2/6] tests: migration test deprecated commands

2017-12-04 Thread Peter Xu
On Fri, Dec 01, 2017 at 01:58:09PM +0100, Juan Quintela wrote: > We now test the deprecated commands everytime that we test the new > commands. This makes unnecesary to add tests for deprecated commands. > > Signed-off-by: Juan Quintela > --- > tests/migration-test.c | 32 ++

Re: [Qemu-devel] [PATCH 3/3] s390x: deprecate s390-squash-mcss machine prop

2017-12-04 Thread Dong Jia Shi
* Cornelia Huck [2017-12-04 17:11:24 +0100]: [...] This one looks good to me too, so: Reviewed-by: Dong Jia Shi > > (...) > > > > Looks sane. We should put a note into the 2.12 changelog as well. > > > > > > > I agree. Who would be responsible for updating the changelog. I'm not > > fami

Re: [Qemu-devel] [PATCH 3/3] s390x: deprecate s390-squash-mcss machine prop

2017-12-04 Thread Dong Jia Shi
* Dong Jia Shi [2017-12-05 15:43:00 +0800]: > * Cornelia Huck [2017-12-04 17:11:24 +0100]: > > [...] > > This one looks good to me too, so: > Reviewed-by: Dong Jia Shi > BTW, since we are deprecating s390-squash-mcss, I think any more improment on it (e.g. more accurate and helpful hint mess

<    1   2   3