Re: [Qemu-devel] [PATCH 0/2] coroutine: dynamically scale pool size

2014-07-04 Thread Ming Lei
On Fri, Jul 4, 2014 at 2:31 PM, Paolo Bonzini wrote: > Il 04/07/2014 07:10, Ming Lei ha scritto: > >> With both the two coroutine patches and the block plug&unplug patches, >> performance of qemu.git/master virtio-blk dataplane can recover to level >> of >> QEMU 2.0. > > > So the excessive writes

Re: [Qemu-devel] [RFC PATCH 00/11] Adding FreeBSD's Capsicum security framework (part 1)

2014-07-04 Thread Paolo Bonzini
Il 03/07/2014 20:39, David Drysdale ha scritto: > On Thu, Jul 03, 2014 at 11:12:33AM +0200, Paolo Bonzini wrote: >> Given Linux's previous experience with BPF filters, what do you >> think about attaching specific BPF programs to file descriptors? >> Then whenever a syscall is run that affects a f

Re: [Qemu-devel] [PATCH 0/2] coroutine: dynamically scale pool size

2014-07-04 Thread Paolo Bonzini
Il 04/07/2014 09:02, Ming Lei ha scritto: > So the excessive writes (to eventfd and irqfd) are not a problem? That'd be > a relief. :) I mean it is in a level, but your aio_notify() patch still can improve virtioblk dataplane performance some, in my test, with 5~10K IOPS improvement, which shoul

Re: [Qemu-devel] [PATCH for-2.1?!?] AioContext: speed up aio_notify

2014-07-04 Thread Ming Lei
On Fri, Jul 4, 2014 at 2:51 PM, Paolo Bonzini wrote: > Il 03/07/2014 20:17, Stefan Hajnoczi ha scritto: > >> On Thu, Jul 3, 2014 at 6:59 PM, Paolo Bonzini wrote: >> >> Looks like a very interesting optimization for the block layer. If we >> can get some performance results - especially how it im

Re: [Qemu-devel] [PATCH for-2.1?!?] AioContext: speed up aio_notify

2014-07-04 Thread Paolo Bonzini
Il 04/07/2014 09:23, Ming Lei ha scritto: I think it is good and better to go to 2.1, and it should save lots of write syscall. Also should regression be caused, the per thread trick may be resorted to, which should be simple. If we have the "right" solution (which we do, unlike the plug/unplu

Re: [Qemu-devel] [RFC] alpha qemu arithmetic exceptions

2014-07-04 Thread Al Viro
On Thu, Jul 03, 2014 at 09:30:05PM -0700, Richard Henderson wrote: > > Another one is probably not worth bothering - PERR, CTPOP, CTLZ, UNPKBx and > > PKxB > > don't accept literal argument. For one thing, as(1) won't let you generate > > those, so it would have to be explicit > > .long 0x70

Re: [Qemu-devel] [RFC] icount: warp in the main_loop.

2014-07-04 Thread Frederic Konrad
On 01/07/2014 18:13, fred.kon...@greensocs.com wrote: From: KONRAD Frederic This fixes a bug where QEMU stall in icount mode. It happens when a simple timer callback is created on VIRTUAL CLOCK modding itself regularly. The actual warping mechanism is called once and then the time didn't grow

Re: [Qemu-devel] [PATCH for 2.1 2/2] memory-backend-file: improve error handling

2014-07-04 Thread Hu Tao
On Thu, Jul 03, 2014 at 10:05:56PM -0600, Eric Blake wrote: > On 07/03/2014 08:56 PM, Hu Tao wrote: > > On Thu, Jul 03, 2014 at 06:33:28AM -0600, Eric Blake wrote: > >> On 07/03/2014 12:10 AM, Hu Tao wrote: > >>> This patch fixes two problems of memory-backend-file: > >>> > >>> 1. If user adds a me

Re: [Qemu-devel] [PATCH for 2.1 2/2] memory-backend-file: improve error handling

2014-07-04 Thread Paolo Bonzini
Il 04/07/2014 09:43, Hu Tao ha scritto: The memory size is rounded up to hpagesize when allocating memory, we can waste at most hpagesize-1 bytes memory. I don't think it's a problem. In the other side, do you think we should give an error in the case like -object memory-backend-ram,size=M(g

Re: [Qemu-devel] [PATCH 0/2] coroutine: dynamically scale pool size

2014-07-04 Thread Stefan Hajnoczi
On Fri, Jul 04, 2014 at 01:10:39PM +0800, Ming Lei wrote: > On Thu, Jul 3, 2014 at 9:51 PM, Stefan Hajnoczi wrote: > > The coroutine pool reuses exited coroutines to make qemu_coroutine_create() > > cheap. The size of the pool is capped to prevent it from hogging memory > > after > > a period of

Re: [Qemu-devel] [RFC] icount: warp in the main_loop.

2014-07-04 Thread Paolo Bonzini
Il 04/07/2014 09:30, Frederic Konrad ha scritto: +/* + * In icount mode, sometimes the VCPU is blocked and an event is needed to + * continue. + * Just warp to make the time grows and have a chance to run the CPU. + */ +qemu_clock_warp(QEMU_CLOCK_VIRTUAL); return r

Re: [Qemu-devel] from which version qemu support clone on rbd

2014-07-04 Thread Stefan Hajnoczi
On Wed, Jul 02, 2014 at 11:17:40PM +0800, yue wrote: > i now look at qemu 2.0, i do not find rbd-api related to clone. > if qemu support this function? and from which version? > clone api of rbd is very simple(one api), why qemu does not implement?what is > the reason? rbd_snap_create()? QEMU su

Re: [Qemu-devel] [RFC] COLO HA Project proposal

2014-07-04 Thread Dr. David Alan Gilbert
* Dong, Eddie (eddie.d...@intel.com) wrote: > > > > > > I didn't quite understand a couple of things though, perhaps you can > > > explain: > > >1) If we ignore the TCP sequence number problem, in an SMP machine > > > don't we get other randomnesses - e.g. which core completes something > > > f

Re: [Qemu-devel] [PATCH v5 2/3] linux-aio: implement io plug, unplug and flush io queue

2014-07-04 Thread Stefan Hajnoczi
On Fri, Jul 04, 2014 at 11:06:41AM +0800, Ming Lei wrote: > +static int ioq_submit(struct qemu_laio_state *s) > +{ > +int ret, i = 0; > +int len = s->io_q.idx; > + > +do { > +ret = io_submit(s->ctx, len, s->io_q.iocbs); > +} while (i++ < 3 && ret == -EAGAIN); > + > +/* e

[Qemu-devel] [PATCH v2 2/2] block: bump coroutine pool size for drives

2014-07-04 Thread Stefan Hajnoczi
When a BlockDriverState is associated with a storage controller DeviceState we expect guest I/O. Use this opportunity to bump the coroutine pool size by 64. This patch ensures that the coroutine pool size scales with the number of drives attached to the guest. It should increase coroutine pool u

[Qemu-devel] [PATCH v2 1/2] coroutine: make pool size dynamic

2014-07-04 Thread Stefan Hajnoczi
Allow coroutine users to adjust the pool size. For example, if the guest has multiple emulated disk drives we should keep around more coroutines. Signed-off-by: Stefan Hajnoczi --- include/block/coroutine.h | 11 +++ qemu-coroutine.c | 26 +++--- 2 files cha

[Qemu-devel] [PATCH v2 0/2] coroutine: dynamically scale pool size

2014-07-04 Thread Stefan Hajnoczi
v2: * Assert that callers never reduce pool below default size [eblake] The coroutine pool reuses exited coroutines to make qemu_coroutine_create() cheap. The size of the pool is capped to prevent it from hogging memory after a period of high coroutine activity. Previously the max size was hard

Re: [Qemu-devel] [PATCH v4 2/3] linux-aio: implement io plug, unplug and flush io queue

2014-07-04 Thread Ming Lei
Hi Stefan, Sorry for missing your comments. On Thu, Jul 3, 2014 at 8:24 PM, Stefan Hajnoczi wrote: > On Wed, Jul 02, 2014 at 08:18:47PM +0800, Ming Lei wrote: >> @@ -36,9 +38,19 @@ struct qemu_laiocb { >> QLIST_ENTRY(qemu_laiocb) node; >> }; >> >> +struct laio_queue { > > This file doesn't

Re: [Qemu-devel] [PATCH v4 2/3] linux-aio: implement io plug, unplug and flush io queue

2014-07-04 Thread Ming Lei
On Fri, Jul 4, 2014 at 5:18 PM, Ming Lei wrote: >> >> This leaks requests when ret < len. I think the loop below should be >> used in that case to fail unsubmitted requests with -EIO. > > I thought about the problem before, but looks it may not return 'ret' > which is less than > len, follows the

Re: [Qemu-devel] Help debugging audio problem

2014-07-04 Thread Anshul Makkar
Use of glue is heavily uses in audio code. I completely redesigned it for Virtualbox and removed all the hard to understand glue code :) . Not sure if this glue magic is such heavily used anywhere else also. Moreover audio code uses one big monolythic big audio file audio.c . So bringing modulari

[Qemu-devel] [PATCH] virtio-pci: fix MSI memory region use after tree

2014-07-04 Thread Paolo Bonzini
After memory region QOMification QEMU is stricter in detecting wrong usage of the memory region API. Here it detected a memory_region_destroy done before the corresponding memory_region_del_subregion; the memory_region_destroy is done by msix_uninit_exclusive_bar, the memory_region_del_subregion i

Re: [Qemu-devel] [PATCH v2 2/2] block: bump coroutine pool size for drives

2014-07-04 Thread Markus Armbruster
Stefan Hajnoczi writes: > When a BlockDriverState is associated with a storage controller > DeviceState we expect guest I/O. Use this opportunity to bump the > coroutine pool size by 64. > > This patch ensures that the coroutine pool size scales with the number > of drives attached to the guest.

[Qemu-devel] [PATCH v6 0/3] linux-aio: introduce submit I/O as a batch

2014-07-04 Thread Ming Lei
Hi, The commit 580b6b2aa2(dataplane: use the QEMU block layer for I/O) introduces ~40% throughput regression on virtio-blk dataplane, and one of causes is that submitting I/O as a batch is removed. This patchset trys to introduce this mechanism on block, at least, linux-aio can benefit from that.

[Qemu-devel] [PATCH v6 1/3] block: block: introduce APIs for submitting IO as a batch

2014-07-04 Thread Ming Lei
This patch introduces three APIs so that following patches can support queuing I/O requests and submitting them as a batch for improving I/O performance. Reviewed-by: Paolo Bonzini Signed-off-by: Ming Lei --- block.c | 31 +++ include/block/block.

[Qemu-devel] [PATCH v6 3/3] dataplane: submit I/O as a batch

2014-07-04 Thread Ming Lei
Before commit 580b6b2aa2(dataplane: use the QEMU block layer for I/O), dataplane for virtio-blk submits block I/O as a batch. This commit 580b6b2aa2 replaces the custom linux AIO implementation(including submit I/O as a batch) with QEMU block layer, but this commit causes ~40% throughput regressio

[Qemu-devel] [PATCH v6 2/3] linux-aio: implement io plug, unplug and flush io queue

2014-07-04 Thread Ming Lei
This patch implements .bdrv_io_plug, .bdrv_io_unplug and .bdrv_flush_io_queue callbacks for linux-aio Block Drivers, so that submitting I/O as a batch can be supported on linux-aio. Signed-off-by: Ming Lei --- block/linux-aio.c | 96 +++-- block/

Re: [Qemu-devel] [PATCH for-2.1?!?] AioContext: speed up aio_notify

2014-07-04 Thread Kevin Wolf
Am 04.07.2014 um 09:27 hat Paolo Bonzini geschrieben: > Il 04/07/2014 09:23, Ming Lei ha scritto: > >I think it is good and better to go to 2.1, and it should save lots of > >write syscall. > > > >Also should regression be caused, the per thread trick may be > >resorted to, which should be simple.

Re: [Qemu-devel] [RFC] icount: warp in the main_loop.

2014-07-04 Thread Frederic Konrad
On 04/07/2014 09:57, Paolo Bonzini wrote: Il 04/07/2014 09:30, Frederic Konrad ha scritto: +/* + * In icount mode, sometimes the VCPU is blocked and an event is needed to + * continue. + * Just warp to make the time grows and have a chance to run the CPU. + */ +qemu_clo

Re: [Qemu-devel] [PATCH v2 2/2] block: bump coroutine pool size for drives

2014-07-04 Thread Lluís Vilanova
Stefan Hajnoczi writes: > When a BlockDriverState is associated with a storage controller > DeviceState we expect guest I/O. Use this opportunity to bump the > coroutine pool size by 64. > This patch ensures that the coroutine pool size scales with the number > of drives attached to the guest.

Re: [Qemu-devel] [RFC] icount: warp in the main_loop.

2014-07-04 Thread Paolo Bonzini
> Right, but when I put a timer eg on QEMU_VIRTUAL_CLOCK the guest is stuck. > icount_warp_rt is not called neither qemu_clock_warp(..).. It should be. timer_mod_ns -> timerlist_rearm -> qemu_clock_warp. > So yes as you said seems a qemu_clock_warp is missing somewhere. > > Shouldn't icount_wa

Re: [Qemu-devel] [PATCH for-2.1?!?] AioContext: speed up aio_notify

2014-07-04 Thread Paolo Bonzini
> We have a rather long freeze phase this time (which I think is a good > thing). This patch fixes a regression, even if it may not be the most > important one because it is in experimental code. But I still think that > this time in the hard freeze is the right time to commit patches like > this.

Re: [Qemu-devel] [RFC] COLO HA Project proposal

2014-07-04 Thread Andreas Färber
Am 01.07.2014 14:12, schrieb Dr. David Alan Gilbert: > Are there any standard APIs for the heartbeet thing we can already > tie into? Maybe the http://www.linux-ha.org/wiki/Heartbeat daemon? Andreas -- SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Jeff Hawn, Jennifer Guil

Re: [Qemu-devel] [PATCH 06/10] mm: sys_remap_anon_pages

2014-07-04 Thread Michael Kerrisk
[CC+=linux-api@] Hi Andrea, On Wed, Jul 2, 2014 at 6:50 PM, Andrea Arcangeli wrote: > This new syscall will move anon pages across vmas, atomically and > without touching the vmas. Please CC linux-api on patches that change the API/ABI. (See https://www.kernel.org/doc/man-pages/linux-api-ml.htm

[Qemu-devel] [PATCH V2 2/6] runner: Tool for fuzz tests execution

2014-07-04 Thread Maria Kustova
The purpose of the test runner is to prepare test environment (e.g. create a work directory, a test image, etc), execute the program under test with parameters, indicate a test failure if the program was killed during test execution and collect core dumps, logs and other test artifacts. The test r

[Qemu-devel] [PATCH V2 1/6] docs: Specification for the image fuzzer

2014-07-04 Thread Maria Kustova
'Overall fuzzer requirements' chapter contains the current product vision and features done and to be done. This chapter is still in progress. v1 -> v2: * added description for fuzzer configuration files * added the example fuzzer.conf file * various fixes based on the Eric's review Signed-off

[Qemu-devel] [PATCH V2 3/6] fuzz: Fuzzing functions for qcow2 images

2014-07-04 Thread Maria Kustova
Fuzz submodule of qcow2 image generator contains fuzzing functions for image fields. Each fuzzing function contains list of constraints and call of a helper function that randomly selects a fuzzed value satisfied to one of constraints. For now constraints are only known as invalid or potentially da

[Qemu-devel] [PATCH V2 5/6] package: Public API for image-fuzzer/runner/runner.py

2014-07-04 Thread Maria Kustova
__init__.py provides the public API required by the test runner Signed-off-by: Maria Kustova --- tests/image-fuzzer/qcow2/__init__.py | 1 + 1 file changed, 1 insertion(+) create mode 100644 tests/image-fuzzer/qcow2/__init__.py diff --git a/tests/image-fuzzer/qcow2/__init__.py b/tests/image-f

[Qemu-devel] [PATCH V2 0/6] tests: Add the image fuzzer with qcow2 support

2014-07-04 Thread Maria Kustova
This patch series introduces the image fuzzer, a tool for stability and reliability testing. Its approach is to run large amount of tests in background. During every test a program (e.g. qemu-img) is called to read or modify an invalid test image. A test image has valid inner structure defined by i

[Qemu-devel] [PATCH V2 6/6] image-fuzzer: GPLv2 license file

2014-07-04 Thread Maria Kustova
Signed-off-by: Maria Kustova --- tests/image-fuzzer/LICENSE | 339 + 1 file changed, 339 insertions(+) create mode 100644 tests/image-fuzzer/LICENSE diff --git a/tests/image-fuzzer/LICENSE b/tests/image-fuzzer/LICENSE new file mode 100644 index 00

[Qemu-devel] [PATCH V2 4/6] layout: Generator of fuzzed qcow2 images

2014-07-04 Thread Maria Kustova
Layout submodule of qcow2 package creates a random valid image, randomly selects some amount of its fields, fuzzes them and write the fuzzed image to the file. Now only header and header extensions are generated, a remaining file is filled by zeroes. v1 -> v2: * Added support of fuzzer configurat

Re: [Qemu-devel] Super nested KVM

2014-07-04 Thread Vasiliy Tolstov
2014-07-04 1:00 GMT+04:00 Richard W.M. Jones : > Well I got bored with the TCG test after about 4 hours. It appears to > hang launching the L3 guest, reproducible on two different hosts both > running qemu 2.0.0. In my tests qemu tcg sometimes fail when run first vm. (kernel 3.12) -- Vasiliy T

Re: [Qemu-devel] [RFC] COLO HA Project proposal

2014-07-04 Thread Dr. David Alan Gilbert
* Dong, Eddie (eddie.d...@intel.com) wrote: > > > > > > Let me clarify on this issue. COLO didn't ignore the TCP sequence > > > number, but uses a new implementation to make the sequence number to > > > be best effort identical between the primary VM (PVM) and secondary VM > > > (SVM). Likely, VMM

[Qemu-devel] [PATCH 2/2] virtio-blk: dataplane: notify guest as a batch

2014-07-04 Thread Ming Lei
Now requests are submitted as a batch, so it is natural to notify guest as a batch too. This may supress interrupt notification to VM: - in my test, decreased by ~13K/sec Signed-off-by: Ming Lei --- hw/block/dataplane/virtio-blk.c | 13 - hw/block/virtio-blk.c

[Qemu-devel] [PATCH 0/2] virtio-blk: dataplane: one fix plus one optimization

2014-07-04 Thread Ming Lei
Hi, The first one fixes one problem introduced recently. The second one supresses notifications to guest. Thanks, -- Ming Lei

[Qemu-devel] [PATCH 1/2] virtio-blk: data-plane: fix save/set .complete_request in start

2014-07-04 Thread Ming Lei
The callback has to be saved and reset in virtio_blk_data_plane_start(), otherwise dataplane's requests will be completed in qemu aio context. Signed-off-by: Ming Lei --- hw/block/dataplane/virtio-blk.c |7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/hw/block/datapl

Re: [Qemu-devel] Super nested KVM

2014-07-04 Thread Richard W.M. Jones
On Fri, Jul 04, 2014 at 04:17:11PM +0400, Vasiliy Tolstov wrote: > 2014-07-04 1:00 GMT+04:00 Richard W.M. Jones : > > Well I got bored with the TCG test after about 4 hours. It appears to > > hang launching the L3 guest, reproducible on two different hosts both > > running qemu 2.0.0. > > > In m

[Qemu-devel] [PATCH] qemu-char: add chr_add_watch support in mux chardev

2014-07-04 Thread Kirill Batuzov
Forward chr_add_watch call from mux chardev to underlying implementation. This should fix bug #1335444 Signed-off-by: Kirill Batuzov --- qemu-char.c |9 + 1 file changed, 9 insertions(+) diff --git a/qemu-char.c b/qemu-char.c index 51917de..e1bd6f5 100644 --- a/qemu-char.c +++ b/qe

Re: [Qemu-devel] [PATCH 2/2] virtio-blk: dataplane: notify guest as a batch

2014-07-04 Thread Paolo Bonzini
Il 04/07/2014 14:27, Ming Lei ha scritto: Now requests are submitted as a batch, so it is natural to notify guest as a batch too. This may supress interrupt notification to VM: - in my test, decreased by ~13K/sec I don't think this can work. Requests are not completed FIFO. What yo

Re: [Qemu-devel] [PATCH v5 00/10] Clean up around bdrv_getlength()

2014-07-04 Thread Markus Armbruster
Series contains a few bug fixes; you might want to consider it for 2.1. Markus Armbruster writes: > Issues addressed in this series: > > * BlockDriver method bdrv_getlength() generally returns -errno, but > some implementations return -1 instead. Fix them [PATCH 1]. > > * Frequent conversions

Re: [Qemu-devel] [PATCH] qemu-char: add chr_add_watch support in mux chardev

2014-07-04 Thread Paolo Bonzini
Il 04/07/2014 14:43, Kirill Batuzov ha scritto: Forward chr_add_watch call from mux chardev to underlying implementation. This should fix bug #1335444 Signed-off-by: Kirill Batuzov --- qemu-char.c |9 + 1 file changed, 9 insertions(+) diff --git a/qemu-char.c b/qemu-char.c index

Re: [Qemu-devel] [RFC] COLO HA Project proposal

2014-07-04 Thread Dong, Eddie
> > > > I didn't quite understand a couple of things though, perhaps you can > > explain: > >1) If we ignore the TCP sequence number problem, in an SMP machine > > don't we get other randomnesses - e.g. which core completes something > > first, or who wins a lock contention, so the output strea

Re: [Qemu-devel] [RFC] COLO HA Project proposal

2014-07-04 Thread Dong, Eddie
> > > > Let me clarify on this issue. COLO didn't ignore the TCP sequence > > number, but uses a new implementation to make the sequence number to > > be best effort identical between the primary VM (PVM) and secondary VM > > (SVM). Likely, VMM has to synchronize the emulation of randomization > >

[Qemu-devel] [PATCH v2 2.1 1/4] virtio-blk: Factor common checks out of virtio_blk_handle_read/write()

2014-07-04 Thread Markus Armbruster
Signed-off-by: Markus Armbruster Reviewed-by: Fam Zheng --- hw/block/virtio-blk.c | 24 ++-- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/hw/block/virtio-blk.c b/hw/block/virtio-blk.c index aec3146..d946fa9 100644 --- a/hw/block/virtio-blk.c +++ b/hw/block/

[Qemu-devel] [PATCH v2 2.1 2/4] virtio-blk: Bypass error action and I/O accounting on invalid r/w

2014-07-04 Thread Markus Armbruster
When a device model's I/O operation fails, we execute the error action. This lets layers above QEMU implement thin provisioning, or attempt to correct errors before they reach the guest. But when the I/O operation fails because it's invalid, reporting the error to the guest is the only sensible a

[Qemu-devel] [PATCH v2 2.1 0/4] Suppress error action on r/w beyond end

2014-07-04 Thread Markus Armbruster
When a device model's I/O operation fails, we execute the error action. This lets layers above QEMU implement thin provisioning, or attempt to correct errors before they reach the guest. But when the I/O operation fails because its invalid, reporting the error to the guest is the only sensible ac

[Qemu-devel] [PATCH v2 2.1 4/4] ide: Treat read/write beyond end as invalid

2014-07-04 Thread Markus Armbruster
The block layer fails such reads and writes just fine. However, they then get treated like valid operations that fail: the error action gets executed. Unwanted; reporting the error to the guest is the only sensible action. Reject them before passing them to the block layer. This bypasses the er

[Qemu-devel] [PATCH v2 2.1 3/4] virtio-blk: Treat read/write beyond end as invalid

2014-07-04 Thread Markus Armbruster
The block layer fails such reads and writes just fine. However, they then get treated like valid operations that fail: the error action gets executed. Unwanted; reporting the error to the guest is the only sensible action. Reject them before passing them to the block layer. This bypasses the er

[Qemu-devel] [PATCH] watchdog: fix deadlock with -watchdog-action pause

2014-07-04 Thread Paolo Bonzini
qemu_clock_enable says: /* Disabling the clock will wait for related timerlists to stop * executing qemu_run_timers. Thus, this functions should not * be used from the callback of a timer that is based on @clock. * Doing so would cause a deadlock. */ and it indeed does: vm_stop uses qemu_clo

Re: [Qemu-devel] [PATCH] watchdog: fix deadlock with -watchdog-action pause

2014-07-04 Thread Luiz Capitulino
On Fri, 4 Jul 2014 15:48:50 +0200 Paolo Bonzini wrote: > qemu_clock_enable says: > > /* Disabling the clock will wait for related timerlists to stop > * executing qemu_run_timers. Thus, this functions should not > * be used from the callback of a timer that is based on @clock. > * Doing so

Re: [Qemu-devel] [PATCH] Makefile: Don't build generated headers before Makefile is reread

2014-07-04 Thread Peter Maydell
On 30 June 2014 13:09, Paolo Bonzini wrote: > Il 28/06/2014 18:59, Peter Maydell ha scritto: > >> Having a direct dependency >>Makefile: $(GENERATED_HEADERS) >> can result in not-from-clean builds failing sometimes, because it means >> that when Make does its "is any makefile or include out of

Re: [Qemu-devel] [PATCH 2/2] virtio-blk: dataplane: notify guest as a batch

2014-07-04 Thread Ming Lei
On Fri, Jul 4, 2014 at 8:55 PM, Paolo Bonzini wrote: > Il 04/07/2014 14:27, Ming Lei ha scritto: > >> Now requests are submitted as a batch, so it is natural >> to notify guest as a batch too. >> >> This may supress interrupt notification to VM: >> >> - in my test, decreased by ~13K/sec >

Re: [Qemu-devel] [PATCH RFC V2 6/6] xen: introduce xenpv-softmmu.mak

2014-07-04 Thread Stefano Stabellini
I realize now that patch #5 and #6 of this series feel through the cracks. Could you please rebase and resend? On Mon, 27 Jan 2014, Wei Liu wrote: > Signed-off-by: Wei Liu > --- > configure |7 +-- > default-configs/xenpv-softmmu.mak |2 ++ > 2 files changed,

[Qemu-devel] Atomic instruction.

2014-07-04 Thread Frederic Konrad
Hi everybody, We are experimenting with multi-core QEMU. We have Multiple QEMU cores running on multiple host threads and we are now looking at the issue of ‘atomic’ instructions. Our initial thought was to push some sort of set of flags out to the memory chain (eg: ATOMIC or NORMAL much as th

Re: [Qemu-devel] [RFC PATCH V3 0/6] icount: Implement delay algorithm between guest and host clocks

2014-07-04 Thread Sebastian Tanase
- Mail original - > De: "Paolo Bonzini" > À: "Sebastian Tanase" , qemu-devel@nongnu.org > Cc: aligu...@amazon.com, afaer...@suse.de, r...@twiddle.net, "peter maydell" > , > mich...@walle.cc, a...@alex.org.uk, stefa...@redhat.com, > lcapitul...@redhat.com, crobi...@redhat.com, > arm...@

Re: [Qemu-devel] [PATCH] Makefile: Don't build generated headers before Makefile is reread

2014-07-04 Thread Paolo Bonzini
Il 04/07/2014 16:33, Peter Maydell ha scritto: On 30 June 2014 13:09, Paolo Bonzini wrote: Il 28/06/2014 18:59, Peter Maydell ha scritto: Having a direct dependency Makefile: $(GENERATED_HEADERS) can result in not-from-clean builds failing sometimes, because it means that when Make does it

Re: [Qemu-devel] [PATCH 2/2] virtio-blk: dataplane: notify guest as a batch

2014-07-04 Thread Paolo Bonzini
Il 04/07/2014 16:52, Ming Lei ha scritto: > What you can do is change notify_guest to something like > > qemu_bh_schedule(req->dev->dataplane->notify_guest_bh); > > and do the actual notification in the bottom half. This should ensure that > multiple notifications are coalesced, but it may a

Re: [Qemu-devel] [RFC PATCH V3 0/6] icount: Implement delay algorithm between guest and host clocks

2014-07-04 Thread Paolo Bonzini
Il 04/07/2014 17:36, Sebastian Tanase ha scritto: I've also made some changes to patch 4 (added Sleep() function call when compiling for Windows and other small modifications); is it ok if I merge them with the existing patch 4 on your "icount" branch and then rebase the other patches? Sure.

Re: [Qemu-devel] [RFC] COLO HA Project proposal

2014-07-04 Thread Dong, Eddie
> > Thanks Dave: > > Whether the randomness value/branch/code path the PVM and SVM > may > > have, It is only a performance issue. COLO never assumes the PVM and > > SVM has same internal Machine state. From correctness p.o.v, as if > > the PVM and SVM generate Identical response, we can view

[Qemu-devel] [PATCH 0/4] block: Fix qiov sizes

2014-07-04 Thread Kevin Wolf
Some callers use larger qiovs than nb_sectors for their read requests. This is bad because raw-posix uses nb_sectors to allocate a bounce buffer and then copies the whole iov over. This series fixes some more cases (mirroring was already fixed earlier this week) and adds assertions to catch any ot

[Qemu-devel] [PATCH 4/4] block: Assert qiov length matches request length

2014-07-04 Thread Kevin Wolf
At least raw-posix relies on this because it can allocate bounce buffers based on the request length, but access it using all of the qiov entries later. Signed-off-by: Kevin Wolf --- block.c | 2 ++ block/raw-posix.c | 15 +++ 2 files changed, 13 insertions(+), 4 deletions

[Qemu-devel] [PATCH 2/4] qcow2: Make qiov match request size until backing file EOF

2014-07-04 Thread Kevin Wolf
If a qcow2 image has a shorter backing file and a read request to unallocated clusters goes across EOF of the backing file, the backing file sees a shortened request and the rest is filled with zeros. However, the original too long qiov was used with the shortened request. This patch makes the qio

[Qemu-devel] [PATCH 3/4] qed: Make qiov match request size until backing file EOF

2014-07-04 Thread Kevin Wolf
If a QED image has a shorter backing file and a read request to unallocated clusters goes across EOF of the backing file, the backing file sees a shortened request and the rest is filled with zeros. However, the original too long qiov was used with the shortened request. This patch makes the qiov

[Qemu-devel] [PATCH 1/4] block: Make qiov match the request size until EOF

2014-07-04 Thread Kevin Wolf
If a read request goes across EOF, the block driver sees a shortened request that stops at EOF (the rest is memsetted in block.c), however the original qiov was used for this request. This patch makes the qiov size match the request size, avoiding a potential buffer overflow in raw-posix. Signed-

Re: [Qemu-devel] [PATCH 2/2] virtio-blk: dataplane: notify guest as a batch

2014-07-04 Thread Ming Lei
On Fri, Jul 4, 2014 at 11:48 PM, Paolo Bonzini wrote: > Il 04/07/2014 16:52, Ming Lei ha scritto: > >>> > What you can do is change notify_guest to something like >>> > >>> > qemu_bh_schedule(req->dev->dataplane->notify_guest_bh); >>> > >>> > and do the actual notification in the bottom half.

Re: [Qemu-devel] Atomic instruction.

2014-07-04 Thread Peter Maydell
On 4 July 2014 16:35, Frederic Konrad wrote: > Hi everybody, > > We are experimenting with multi-core QEMU. We have Multiple QEMU cores > running > on multiple host threads and we are now looking at the issue of ‘atomic’ > instructions. > > Our initial thought was to push some sort of set of flags

Re: [Qemu-devel] [RFC PATCH V3 0/6] icount: Implement delay algorithm between guest and host clocks

2014-07-04 Thread Michael Tokarev
Can we pretty please take me off the list of recepients. I have nothing to do with this, I receive all list emails already, and where I am now it is quite dificult to sort mail, and it costs me quite some time to do so too. When I'm back I'll concider dropping mails sent to list and Cc'd to me.

Re: [Qemu-devel] [PATCH 2/2] virtio-blk: dataplane: notify guest as a batch

2014-07-04 Thread Paolo Bonzini
Il 04/07/2014 17:57, Ming Lei ha scritto: But we have two cases to consider: - one submitted IO includes requests from multi vq(virtio-blk or virtio-scsi maybe), and each vq has to notify guest - one submitted IO includes requests from multi bs for scsi device The 2nd case should be easy to ha

[Qemu-devel] [PATCH 00/46] Postcopy implementation

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Hi, This is the 1st cut of my version of postcopy; it is designed for use with the Linux kernel additions recently posted by Andrea Arcangeli here: http://lists.gnu.org/archive/html/qemu-devel/2014-07/msg00525.html The current status is: 1) It works - I've

[Qemu-devel] [PATCH 26/46] Postcopy page-map-incoming (PMI) structure

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" The PMI holds the state of each page on the incoming side, so that we can tell if the page is missing, already received or there is a request outstanding for it. Signed-off-by: Dr. David Alan Gilbert --- include/migration/migration.h| 18 ++ include/migr

[Qemu-devel] [PATCH 03/46] QEMUSizedBuffer/QEMUFile

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Stefan Berger's patch to create a QEMUFile that goes to a memory buffer; from: http://lists.gnu.org/archive/html/qemu-devel/2013-03/msg05036.html Using the QEMUFile interface, this patch adds support functions for operating on in-memory sized buffers that can be w

[Qemu-devel] [PATCH 04/46] improve DPRINTF macros, add to savevm

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Improve the existing DPRINTF macros in migration.c and arch_init by: 1) Making them go to stderr rather than stdout (so you can run with -nographic and redirect your debug to a file) 2) Making them print the ms time with each debug - useful for debugging latency

[Qemu-devel] [PATCH 07/46] Return path: Open a return path on QEMUFile for sockets

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Postcopy needs a method to send messages from the destination back to the source, this is the 'return path'. Wire it up for 'socket' QEMUFile's using a dup'd fd. Signed-off-by: Dr. David Alan Gilbert --- include/migration/qemu-file.h | 8 + qemu-file.c

[Qemu-devel] [PATCH 37/46] Add assertion to check migration_dirty_pages doesn't go -ve; have seen it happen once but not sure why

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Signed-off-by: Dr. David Alan Gilbert --- arch_init.c | 1 + 1 file changed, 1 insertion(+) diff --git a/arch_init.c b/arch_init.c index c006d21..58eccc1 100644 --- a/arch_init.c +++ b/arch_init.c @@ -439,6 +439,7 @@ ram_addr_t migration_bitmap_find_and_reset_di

[Qemu-devel] [PATCH 08/46] Return path: socket_writev_buffer: Block even on non-blocking fd's

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" The return path uses a non-blocking fd so as not to block waiting for the (possibly broken) destination to finish returning a message, however we still want outbound data to behave in the same way and block. Signed-off-by: Dr. David Alan Gilbert --- qemu-file.c |

[Qemu-devel] [PATCH 09/46] Migration commands

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Create QEMU_VM_COMMAND section type for sending commands from source to destination. These commands are not intended to convey guest state but to control the migration process. For use in postcopy. Signed-off-by: Dr. David Alan Gilbert --- include/migration/mig

[Qemu-devel] [PATCH 10/46] Return path: Control commands

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Add two src->dest commands: * OPENRP - To request that the destination open the return path * REQACK - Request an acknowledge from the destination Signed-off-by: Dr. David Alan Gilbert --- include/migration/migration.h | 2 ++ include/sysemu/sysemu.h

[Qemu-devel] [PATCH 46/46] Start documenting how postcopy works.

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Signed-off-by: Dr. David Alan Gilbert --- docs/migration.txt | 148 + 1 file changed, 148 insertions(+) diff --git a/docs/migration.txt b/docs/migration.txt index 0492a45..dbd5e5f 100644 --- a/docs/migration.txt

[Qemu-devel] [PATCH 44/46] postcopy: Use userfaultfd

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" userfaultfd is a Linux syscall that gives an fd that receives a stream of notifications of accesses to pages marked as MADV_USERFAULT, and allows the program to acknowledge those stalls and tell the accessing thread to carry on. Signed-off-by: Dr. David Alan Gilber

[Qemu-devel] [PATCH 23/46] MIG_STATE_POSTCOPY_ACTIVE: Add new migration state

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" 'MIG_STATE_POSTCOPY_ACTIVE' is entered after the precopy timelimit has expired and migration switches to postcopy. 'migration_postcopy_phase' is provided for other sections to know if they're in postcopy. Signed-off-by: Dr. David Alan Gilbert --- include/migrati

[Qemu-devel] [PATCH 45/46] End of migration for postcopy

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Tweak the end of migration cleanup; we don't want to close stuff down at the end of the main stream, since the postcopy is still sending pages on the other thread. Signed-off-by: Dr. David Alan Gilbert --- migration.c | 20 1 file changed, 20

[Qemu-devel] [PATCH 25/46] Postcopy: Maintain sentmap during postcopy pre phase

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Where postcopy is preceeded by a period of precopy, the destination will have received pages that may have been dirtied on the source after the page was sent. The destination must throw these pages away before starting it's CPUs. Maintain a 'sentmap' of pages that

[Qemu-devel] [PATCH 24/46] qemu_savevm_state_complete: Postcopy changes

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" When postcopy calls qemu_savevm_state_complete it's not really the end of migration, so skip: a) Finishing postcopiable iterative devices - they'll carry on b) The termination byte on the end of the stream. We then also add: qemu_savevm_state_postcopy_compl

[Qemu-devel] [PATCH 27/46] postcopy: Add incoming_init/cleanup functions

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Provide functions to be called before the start of a postcopy enabled migration (even if it's not eventually used) and at the end. During the init we must disable huge pages in the RAM that we will receive postcopy data into, since if they start off as hugepage and

[Qemu-devel] [PATCH 28/46] postcopy: Incoming initialisation

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Signed-off-by: Dr. David Alan Gilbert --- arch_init.c | 11 +++ include/migration/migration.h | 1 + migration.c | 2 ++ 3 files changed, 14 insertions(+) diff --git a/arch_init.c b/arch_init.c index 134ea7e..fd7399c

[Qemu-devel] [PATCH 29/46] postcopy: ram_enable_notify to switch on userfault

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Signed-off-by: Dr. David Alan Gilbert --- include/migration/postcopy-ram.h | 5 + postcopy-ram.c | 36 +++- 2 files changed, 40 insertions(+), 1 deletion(-) diff --git a/include/migration/postcopy-ram.h b/inc

[Qemu-devel] [PATCH 36/46] Page request: Consume pages off the post-copy queue

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" When transmitting RAM pages, consume pages that have been queued by MIG_RPCOMM_REQPAGE commands and send them ahead of normal page scanning. Note: a) After a queued page the linear walk carries on from after the unqueued page; there is a reasonable chance that th

[Qemu-devel] [PATCH 30/46] Postcopy: postcopy_start

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" postcopy_start: Perform all the initialisation associated with starting up postcopy mode from the source. Signed-off-by: Dr. David Alan Gilbert --- migration.c | 85 + 1 file changed, 85 insertions(+)

[Qemu-devel] [PATCH 31/46] Postcopy: Rework migration thread for postcopy mode

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Switch to postcopy if: 1) There's still a significant amount to transfer 2) Postcopy is enabled 3) It's taken longer than the time set by the parameter. and change the cleanup at the end of migration to match. Signed-off-by: Dr. David Alan Gilbert --- m

[Qemu-devel] [PATCH 32/46] mig fd_connect: open return path

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Signed-off-by: Dr. David Alan Gilbert --- migration.c | 12 1 file changed, 12 insertions(+) diff --git a/migration.c b/migration.c index c73fcfa..c7ba6a1 100644 --- a/migration.c +++ b/migration.c @@ -1164,6 +1164,18 @@ void migrate_fd_connect(Migra

[Qemu-devel] [PATCH 33/46] Postcopy: Create a fault handler thread before marking the ram as userfault

2014-07-04 Thread Dr. David Alan Gilbert (git)
From: "Dr. David Alan Gilbert" Signed-off-by: Dr. David Alan Gilbert --- include/migration/migration.h | 3 +++ postcopy-ram.c| 23 +++ 2 files changed, 26 insertions(+) diff --git a/include/migration/migration.h b/include/migration/migration.h index 397f41

  1   2   >