[Qemu-devel] [PATCH v2 00/11] block: incremental backup transactions

2015-03-27 Thread John Snow
of primary interest to libvirt, in particular, but only as a convenience feature. Thank you, --John Snow v2: Key: [] : patches are identical [] : number of functional differences between upstream/downstream patch [down] : patch is downstream-only The flags [FC] indicate (F)unctio

[Qemu-devel] [PULL for-2.3 1/2] AHCI: Do not (re)map FB/CLB buffers while not running

2015-03-27 Thread John Snow
, so we should not attempt to re-map these values until the engines are reactivated. Reported-by: Jordan Hargrave Signed-off-by: John Snow Reviewed-by: Stefan Hajnoczi Message-id: 1426283454-15590-2-git-send-email-js...@redhat.com --- hw/ide/ahci.c | 50

[Qemu-devel] [PULL for-2.3 2/2] AHCI: Protect cmd register

2015-03-27 Thread John Snow
. Signed-off-by: John Snow Reviewed-by: Stefan Hajnoczi Message-id: 1426283454-15590-3-git-send-email-js...@redhat.com --- hw/ide/ahci.c | 26 +- hw/ide/ahci.h | 2 ++ 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/hw/ide/ahci.c b/hw/ide/ahci.c index

[Qemu-devel] [PULL for-2.3 0/2] Ide patches

2015-03-27 Thread John Snow
quest for you to fetch changes up to fc3d8e1138cd0c843d6fd75272633a31be6554ef: AHCI: Protect cmd register (2015-03-27 15:48:11 -0400) ---- John Snow (2): AHCI: Do not

[Qemu-devel] [PULL for-2.3 0/2] Ide patches

2015-03-27 Thread John Snow
quest for you to fetch changes up to fc3d8e1138cd0c843d6fd75272633a31be6554ef: AHCI: Protect cmd register (2015-03-27 15:48:11 -0400) ---- John Snow (2): AHCI: Do not

Re: [Qemu-devel] [PATCH RFC for-2.3? 5/8] fdb: Move FDCtrlISABus to header

2015-03-30 Thread John Snow
You probably meant 'fdc' ! On 03/29/2015 01:53 PM, Andreas Färber wrote: To be used for embedding the device. Add gtk-doc private/public markers for parent field. Signed-off-by: Andreas Färber --- hw/block/fdc.c | 87 - include/hw/blo

Re: [Qemu-devel] Bouncing maintainers

2015-03-31 Thread John Snow
On 03/31/2015 06:01 AM, Markus Armbruster wrote: John Snow writes: On 03/26/2015 09:19 AM, Kevin Wolf wrote: Am 25.03.2015 um 16:45 hat John Snow geschrieben: On 03/25/2015 10:48 AM, Markus Armbruster wrote: I just had another cc: to an address gotten from MAINTAINERS bounce with "

Re: [Qemu-devel] [PATCH 4/4] ahci-test: improve rw buffer patterns

2015-03-31 Thread John Snow
On 03/31/2015 04:55 PM, Ed Maste wrote: On 23 March 2015 at 12:56, John Snow wrote: My pattern was cyclical every 256 bytes, so it missed a fairly obvious failure case. Add some rand() pepper into the test pattern, and for large patterns that exceed 256 sectors, start writing an ID per

Re: [Qemu-devel] [PATCH 4/4] ahci-test: improve rw buffer patterns

2015-03-31 Thread John Snow
On 03/31/2015 05:29 PM, Ed Maste wrote: On 31 March 2015 at 16:58, John Snow wrote: which test case does it fail under? io_rw_simple is shared by a number of different tests. I'm not sure off hand how to run an individual test to confirm, but the testrun output up to the failu

Re: [Qemu-devel] [PATCH 4/4] ahci-test: improve rw buffer patterns

2015-03-31 Thread John Snow
On 03/31/2015 05:55 PM, Ed Maste wrote: On 31 March 2015 at 17:44, John Snow wrote: My apologies; Not at all, thanks for your help! Head into your build directory and try this: make tests/ahci-test export QTEST_QEMU_BINARY=i386-softmmu/qemu-system-i386 export QTEST_QEMU_IMG=./qemu-img

Re: [Qemu-devel] [PATCH 4/4] ahci-test: improve rw buffer patterns

2015-03-31 Thread John Snow
On 03/31/2015 08:29 PM, Ed Maste wrote: On 31 March 2015 at 17:55, Ed Maste wrote: feynman% ./tests/ahci-test /i386/ahci/sanity: OK /i386/ahci/pci_spec: OK /i386/ahci/pci_enable: OK /i386/ahci/hba_spec: OK /i386/ahci/hba_enable: OK /i386/ahci/identify: OK /i386/ahci/io/pio/lba28/simple: ** E

[Qemu-devel] qemu-img behavior for locating backing files

2015-04-01 Thread John Snow
Kevin, what's the correct behavior for qemu-img and relative paths when creating a new qcow2 file? Example: (in e.g. /home/qemu/build/ or anywhere not /home: ) qemu-img create -f qcow2 base.qcow2 32G qemu-img create -f qcow2 -F qcow2 -b base.qcow2 /home/overlay.qcow2 In 1.7.0., this produces a

Re: [Qemu-devel] [PATCH] qtest: Add assertion that required environment variable is set

2015-04-01 Thread John Snow
mu == NULL) { fprintf(stderr, "..."); g_assert_not_reached(); } Though that does read a little strangely. ("Here's a nice error message for something we are asserting will never happen.") Well, either way, it's better than segfaulting, so: Reviewed-by: John Snow

Re: [Qemu-devel] [Qemu-block] [PATCH v4 15/20] block: Resize bitmaps on bdrv_truncate

2015-04-02 Thread John Snow
On 04/02/2015 09:37 AM, Stefan Hajnoczi wrote: On Fri, Mar 20, 2015 at 03:16:58PM -0400, John Snow wrote: +void hbitmap_truncate(HBitmap *hb, uint64_t size) +{ +bool shrink; +unsigned i; +uint64_t num_elements = size; +uint64_t old; + +/* Size comes in as logical elements

Re: [Qemu-devel] thread-pool.c race condition?

2015-04-02 Thread John Snow
On 04/02/2015 12:43 PM, Paolo Bonzini wrote: On 02/04/2015 18:26, Stefan Hajnoczi wrote: John Snow has reported that qemu-io can hang when the host is under heavy load. He made the following observations in gdb: 1. The program is sitting in aio_poll() (called by bdrv_prwv_co()) waiting

Re: [Qemu-devel] [Qemu-block] [PATCH v4 10/20] qmp: Add support of "dirty-bitmap" sync mode for drive-backup

2015-04-02 Thread John Snow
On 04/02/2015 08:44 AM, Stefan Hajnoczi wrote: On Fri, Mar 20, 2015 at 03:16:53PM -0400, John Snow wrote: +} else if (job->sync_mode == MIRROR_SYNC_MODE_DIRTY_BITMAP) { +/* Dirty Bitmap sync has a slightly different iteration method */ +HBitmapIter hbi; +int6

Re: [Qemu-devel] [Qemu-block] [PATCH v4 18/20] iotests: add QMP event waiting queue

2015-04-02 Thread John Snow
On 04/02/2015 09:57 AM, Stefan Hajnoczi wrote: On Fri, Mar 20, 2015 at 03:17:01PM -0400, John Snow wrote: +# Test if 'match' is a recursive subset of 'event' +def event_match(event, match = None): Not worth respinning but PEP8 says there should be no spaces around

Re: [Qemu-devel] [PATCH] qtest: Add assertion that required environment variable is set

2015-04-06 Thread John Snow
On 04/03/2015 07:18 AM, Peter Maydell wrote: On 2 April 2015 at 20:31, Ed Maste wrote: On 1 April 2015 at 18:45, Peter Maydell wrote: Also irritating is the way it silently requires the binary to have a name in the shape it was expecting, which can catch you out if you were trying to set it

Re: [Qemu-devel] [Qemu-block] Migration sometimes fails with IDE and Qemu 2.2.1

2015-04-06 Thread John Snow
On 04/06/2015 02:47 PM, Peter Lieven wrote: Hi all, is there a known issue in Qemu 2.2.1 where IDE stalls sometimes after a migration with Qemu 2.2.1? The migration succeeds, but it seems that the complete I/O is hanging. This happens only sometimes and only with extreme old Linux Guests (SL

Re: [Qemu-devel] [PATCH v4 19/20] iotests: add simple incremental backup case

2015-04-06 Thread John Snow
On 04/02/2015 10:27 AM, Stefan Hajnoczi wrote: On Fri, Mar 20, 2015 at 03:17:02PM -0400, John Snow wrote: Signed-off-by: John Snow --- tests/qemu-iotests/124 | 153 + tests/qemu-iotests/124.out | 4 +- 2 files changed, 155 insertions

Re: [Qemu-devel] qemu-img behavior for locating backing files

2015-04-06 Thread John Snow
On 04/02/2015 05:38 AM, Kevin Wolf wrote: Am 01.04.2015 um 18:16 hat John Snow geschrieben: Kevin, what's the correct behavior for qemu-img and relative paths when creating a new qcow2 file? Example: (in e.g. /home/qemu/build/ or anywhere not /home: ) qemu-img create -f qcow2 base.qcow

Re: [Qemu-devel] qemu-img behavior for locating backing files

2015-04-07 Thread John Snow
On 04/07/2015 04:44 AM, Kevin Wolf wrote: Am 07.04.2015 um 02:31 hat John Snow geschrieben: On 04/02/2015 05:38 AM, Kevin Wolf wrote: Am 01.04.2015 um 18:16 hat John Snow geschrieben: Kevin, what's the correct behavior for qemu-img and relative paths when creating a new qcow2

Re: [Qemu-devel] [Qemu-block] [PATCH v4 15/20] block: Resize bitmaps on bdrv_truncate

2015-04-07 Thread John Snow
On 04/07/2015 08:57 AM, Stefan Hajnoczi wrote: On Thu, Apr 02, 2015 at 11:57:59AM -0400, John Snow wrote: On 04/02/2015 09:37 AM, Stefan Hajnoczi wrote: On Fri, Mar 20, 2015 at 03:16:58PM -0400, John Snow wrote: +void hbitmap_truncate(HBitmap *hb, uint64_t size) +{ +bool shrink

Re: [Qemu-devel] Getting VM state from outside QEMU?

2015-04-07 Thread John Snow
On 04/07/2015 09:31 AM, Erik Rull wrote: Hi all, I need a pretty simple way to get the current state of the VM running in QEMU - I only need the VM state (e.g. running, paused,...). Since my environment does not have any perl, python or other high level scripting capabilities, a simple way e.g

Re: [Qemu-devel] [Qemu-block] Migration sometimes fails with IDE and Qemu 2.2.1

2015-04-07 Thread John Snow
On 04/07/2015 02:44 PM, Peter Lieven wrote: Am 07.04.2015 um 17:29 schrieb Dr. David Alan Gilbert: * Peter Lieven (p...@kamp.de) wrote: Hi David, Am 07.04.2015 um 10:43 schrieb Dr. David Alan Gilbert: Any particular workload or reproducer? Workload is almost zero. I try to figure out if th

Re: [Qemu-devel] [Qemu-block] Migration sometimes fails with IDE and Qemu 2.2.1

2015-04-07 Thread John Snow
On 04/07/2015 03:02 PM, Peter Lieven wrote: Am 07.04.2015 um 20:56 schrieb John Snow: On 04/07/2015 02:44 PM, Peter Lieven wrote: Am 07.04.2015 um 17:29 schrieb Dr. David Alan Gilbert: * Peter Lieven (p...@kamp.de) wrote: Hi David, Am 07.04.2015 um 10:43 schrieb Dr. David Alan Gilbert

Re: [Qemu-devel] [Qemu-block] [PATCH v4 10/20] qmp: Add support of "dirty-bitmap" sync mode for drive-backup

2015-04-07 Thread John Snow
On 04/02/2015 08:44 AM, Stefan Hajnoczi wrote: On Fri, Mar 20, 2015 at 03:16:53PM -0400, John Snow wrote: +} else if (job->sync_mode == MIRROR_SYNC_MODE_DIRTY_BITMAP) { +/* Dirty Bitmap sync has a slightly different iteration method */ +HBitmapIter hbi; +int6

[Qemu-devel] [PATCH v10 02/13] qapi: Add optional field "name" to block dirty bitmap

2014-12-22 Thread John Snow
bdrv_find_dirty_bitmap to find a dirty bitmap by name, will be used later when other QMP commands want to reference dirty bitmap by name. Add bdrv_dirty_bitmap_make_anon. This unsets the name of dirty bitmap. Signed-off-by: Fam Zheng Signed-off-by: John Snow Reviewed-by: Max Reitz --- block.c

[Qemu-devel] [PATCH v10 00/13] block: Incremental backup series (RFC)

2014-12-22 Thread John Snow
changelog. Enough changes have been made that most Reviewed-By lines from previous iterations have been removed. (Sorry!) This series was originally authored by Fam Zheng; his cover letter is included below. ~John Snow = This is the

[Qemu-devel] [PATCH v10 05/13] hbitmap: Add hbitmap_copy

2014-12-22 Thread John Snow
From: Fam Zheng This makes a deep copy of an HBitmap. Signed-off-by: Fam Zheng Signed-off-by: John Snow --- include/qemu/hbitmap.h | 8 util/hbitmap.c | 20 2 files changed, 28 insertions(+) diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h

[Qemu-devel] [PATCH v10 04/13] block: Introduce bdrv_dirty_bitmap_granularity()

2014-12-22 Thread John Snow
From: Fam Zheng This returns the granularity (in bytes) of dirty bitmap, which matches the QMP interface and the existing query interface. Signed-off-by: Fam Zheng Signed-off-by: John Snow --- block.c | 9 +++-- include/block/block.h | 2 ++ 2 files changed, 9 insertions

[Qemu-devel] [PATCH v10 01/13] block: fix spoiling all dirty bitmaps by mirror and migration

2014-12-22 Thread John Snow
in future, old functions bdrv_(set,reset)_dirty are made static, for internal block usage. Signed-off-by: Vladimir Sementsov-Ogievskiy CC: John Snow CC: Fam Zheng CC: Denis V. Lunev CC: Stefan Hajnoczi CC: Kevin Wolf Signed-off-by: John Snow --- block.c | 23

[Qemu-devel] [PATCH v10 03/13] qmp: Add block-dirty-bitmap-add and block-dirty-bitmap-remove

2014-12-22 Thread John Snow
The types added to block-core.json will be re-used in future patches in this series, see: 'qapi: Add transaction support to block-dirty-bitmap-{add, enable, disable}' Signed-off-by: Fam Zheng Signed-off-by: John Snow --- block.c | 20 ++ block/mirr

[Qemu-devel] [PATCH v10 12/13] qmp: Add dirty bitmap 'enabled' field in query-block

2014-12-22 Thread John Snow
From: Fam Zheng Signed-off-by: Fam Zheng Signed-off-by: John Snow --- block.c | 1 + qapi/block-core.json | 5 - 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/block.c b/block.c index 3d1337b..2466ba8 100644 --- a/block.c +++ b/block.c @@ -5444,6 +5444,7

[Qemu-devel] [PATCH v10 07/13] hbitmap: add hbitmap_merge

2014-12-22 Thread John Snow
running speed for particularly sparse bitmaps by using iterators, but the running time for dense maps will be worse. We present the simpler solution first, and we can refine it later if needed. Signed-off-by: John Snow --- include/qemu/hbitmap.h | 11 +++ util/hbitmap.c | 28

[Qemu-devel] [PATCH v10 06/13] block: Add bdrv_copy_dirty_bitmap and bdrv_clear_dirty_bitmap

2014-12-22 Thread John Snow
Signed-off-by: Fam Zheng Signed-off-by: John Snow --- block.c | 39 +++ include/block/block.h | 4 2 files changed, 39 insertions(+), 4 deletions(-) diff --git a/block.c b/block.c index a1d9e88..f9e0767 100644 --- a/block.c +++ b/block.c

[Qemu-devel] [PATCH v10 10/13] qmp: Add support of "dirty-bitmap" sync mode for drive-backup

2014-12-22 Thread John Snow
For "dirty-bitmap" sync mode, the block job will iterate through the given dirty bitmap to decide if a sector needs backup (backup all the dirty clusters and skip clean ones), just as allocation conditions of "top" sync mode. Signed-off-by: Fam Zheng Signed-off-by: Joh

[Qemu-devel] [PATCH v10 08/13] block: add bdrv_reclaim_dirty_bitmap

2014-12-22 Thread John Snow
BdrvDirtyBitmap associated with a failed operation and merges the bitmap back with the BdrvDirtyBitmap that spawned it, then frees this failed bitmap. Signed-off-by: John Snow --- block.c | 13 + include/block/block.h | 2 ++ 2 files changed, 15 insertions(+) diff --git a

[Qemu-devel] [PATCH v10 13/13] qemu-iotests: Add tests for drive-backup sync=dirty-bitmap

2014-12-22 Thread John Snow
From: Fam Zheng Signed-off-by: Fam Zheng Signed-off-by: John Snow --- tests/qemu-iotests/056| 33 ++--- tests/qemu-iotests/056.out| 4 ++-- tests/qemu-iotests/iotests.py | 8 3 files changed, 40 insertions(+), 5 deletions(-) diff --git a

[Qemu-devel] [PATCH v10 09/13] qmp: Add block-dirty-bitmap-enable and block-dirty-bitmap-disable

2014-12-22 Thread John Snow
From: Fam Zheng This allows to put the dirty bitmap into a disabled state where no more writes will be tracked. It will be used before backup or writing to persistent file. Signed-off-by: Fam Zheng Signed-off-by: John Snow --- block.c | 19 ++- blockdev.c

[Qemu-devel] [PATCH v10 11/13] qapi: Add transaction support to block-dirty-bitmap-{add, enable, disable}

2014-12-22 Thread John Snow
From: Fam Zheng This adds three qmp commands to transactions. Users can stop a dirty bitmap, start backup of it, and start another dirty bitmap atomically, so that the dirty bitmap is tracked incrementally and we don't miss any write. Signed-off-by: Fam Zheng Signed-off-by: John

Re: [Qemu-devel] [PATCH v10 00/13] block: Incremental backup series (RFC)

2014-12-23 Thread John Snow
On 12/23/2014 06:48 AM, Vladimir Sementsov-Ogievskiy wrote: On 23.12.2014 04:12, John Snow wrote: For convenience, this patchset is available on github: https://github.com/jnsnow/qemu/commits/dbm-backup - old version here. My apologies. I've corrected it now. Than

Re: [Qemu-devel] [PATCH 00/10] block/dmg: (compatibility) fixes and bzip2 support

2015-01-02 Thread John Snow
QEMU contributor. I have CCed John Snow. John: If you are busy, please CC someone else or let us know so this series can get reviewed. Stefan Just recomposing myself post-vacation, I will start looking this over today. --John

Re: [Qemu-devel] [PATCH 00/10] block/dmg: (compatibility) fixes and bzip2 support

2015-01-02 Thread John Snow
don my brevity, top-posting and formatting, sent from my phone) On January 2, 2015 5:31:33 PM CET, John Snow wrote: On 01/02/2015 09:14 AM, Stefan Hajnoczi wrote: On Sat, Dec 27, 2014 at 04:01:34PM +0100, Peter Wu wrote: These series improve QEMU support for DMG image files: Hi, Thanks

Re: [Qemu-devel] [PATCH 01/10] block/dmg: properly detect the UDIF trailer

2015-01-02 Thread John Snow
begin); +ret = read_uint64(bs, offset + 0x28, &info_begin); if (ret < 0) { goto fail; } else if (info_begin == 0) { If there really is no convenient way to retrieve the real length ... (Stefan: Would that be difficult to add?) Reviewed-by: John Snow

Re: [Qemu-devel] [PATCH 02/10] block/dmg: extract mish block decoding functionality

2015-01-02 Thread John Snow
hat I can see. } /* initialize zlib engine */ s->compressed_chunk = qemu_try_blockalign(bs->file, - max_compressed_size + 1); + ds.max_compressed_size + 1); s->uncompressed_chunk = qemu_try_blockalign(bs->file, -512 * max_sectors_per_chunk); +512 * ds.max_sectors_per_chunk); if (s->compressed_chunk == NULL || s->uncompressed_chunk == NULL) { ret = -ENOMEM; goto fail; Thank you for splitting this monolithic function up into nicer pieces. With or without addressing my comments above: Reviewed-by: John Snow

Re: [Qemu-devel] [PATCH 03/10] block/dmg: extract processing of resource forks

2015-01-02 Thread John Snow
On 12/27/2014 10:01 AM, Peter Wu wrote: Besides the offset, also read the resource length. This length is now used in the extracted function to verify the end of the resource fork against "count" from the resource fork. Signed-off-by: Peter Wu --- block/dmg.c | 90 ++

Re: [Qemu-devel] [PATCH 04/10] block/dmg: process a buffer instead of reading ints

2015-01-02 Thread John Snow
t XML implementation has to decode base64 data, this function needs to be able to operate on a buffer instead of taking an offset into the file, because we may need to do some processing on the mish data first. Reviewed-by: John Snow

Re: [Qemu-devel] [PATCH 05/10] block/dmg: validate chunk size to avoid overflow

2015-01-02 Thread John Snow
On 12/27/2014 10:01 AM, Peter Wu wrote: Previously the chunk size was not checked, allowing for a large memory allocation. This patch checks whether the chunks size is within the resource fork length. Signed-off-by: Peter Wu --- block/dmg.c | 2 +- 1 file changed, 1 insertion(+), 1 deletio

Re: [Qemu-devel] [PATCH 06/10] block/dmg: process XML plists

2015-01-02 Thread John Snow
On 12/27/2014 10:01 AM, Peter Wu wrote: The format is simple enough to avoid using a full-blown XML parser. The offsets are based on the description at http://newosxbook.com/DMG.html Signed-off-by: Peter Wu --- block/dmg.c | 69 +

Re: [Qemu-devel] [PATCH 07/10] block/dmg: set virtual size to a non-zero value

2015-01-02 Thread John Snow
On 12/27/2014 10:01 AM, Peter Wu wrote: Right now the virtual size is always reported as zero which makes it impossible to convert between formats. After this patch, the number of sectors will be read from the BLXX ("mish") header. Do you mean to say from the 'koly' header? http://en.wikip

Re: [Qemu-devel] [PATCH 08/10] block/dmg: fix sector data offset calculation

2015-01-02 Thread John Snow
On 12/27/2014 10:01 AM, Peter Wu wrote: This patch addresses two issues: - The data fork offset was not taken into account, resulting in failure to read an InstallESD.dmg file (5164763151 bytes) which had a non-zero DataForkOffset field. - The offset of the previous block ("partiti

Re: [Qemu-devel] [PATCH 06/10] block/dmg: process XML plists

2015-01-05 Thread John Snow
On 01/03/2015 06:54 AM, Peter Wu wrote: On Friday 02 January 2015 19:04:32 John Snow wrote: On 12/27/2014 10:01 AM, Peter Wu wrote: The format is simple enough to avoid using a full-blown XML parser. The offsets are based on the description at http://newosxbook.com/DMG.html Signed-off-by

Re: [Qemu-devel] [PATCH 3/3] misc: Fix new typos in comments

2015-01-05 Thread John Snow
On 01/03/2015 08:41 AM, Stefan Weil wrote: recieve -> receive suprise -> surprise Cc: Igor Mammedov Cc: John Snow Signed-off-by: Stefan Weil --- include/hw/hotplug.h |2 +- tests/ahci-test.c|2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/incl

Re: [Qemu-devel] [PATCH 06/10] block/dmg: process XML plists

2015-01-05 Thread John Snow
{ ret = -EINVAL; goto fail; After discussion, and because the dmg2img program uses a similar strategy, this is probably safe enough -- failures do not seem likely and if they occur, we will simply ignore the erroneous data. We can complicate this in the future if we need to, as stated. Reviewed-by: John Snow

Re: [Qemu-devel] [PATCH 09/10] block/dmg: support bzip2 block entry types

2015-01-05 Thread John Snow
On 12/27/2014 10:01 AM, Peter Wu wrote: This patch adds support for bzip2-compressed block entries as introduced with OS X 10.4 (source: https://en.wikipedia.org/wiki/Apple_Disk_Image). It was tested against a 5.2G "OS X Yosemite" installation image which stores the BLXX block in the XML prope

Re: [Qemu-devel] [PATCH 10/10] block/dmg: improve zeroes handling

2015-01-05 Thread John Snow
On 12/27/2014 10:01 AM, Peter Wu wrote: Disk images may contain large all-zeroes gaps (1.66k sectors or 812 MiB is seen in the real world). These blocks (type 2) do not need to be extracted into a temporary buffer, there is no need to allocate memory for these blocks nor to check its length. (

Re: [Qemu-devel] [PATCH v2 03/12] block/dmg: extract processing of resource forks

2015-01-07 Thread John Snow
o be 0x100 (but check that it is within bounds though). - Further improve boundary checking (resource data must be within the resource fork). - Use correct value for resource data length (spotted by John Snow) - Consider the resource data offset when determining info_end. This fixes

Re: [Qemu-devel] [PATCH v2 05/12] block/dmg: validate chunk size to avoid overflow

2015-01-07 Thread John Snow
il; +} if (rsrc_fork_length != 0) { ret = dmg_read_resource_fork(bs, &ds, rsrc_fork_offset, rsrc_fork_length); Reviewed-by: John Snow

Re: [Qemu-devel] [PATCH v2 06/12] block/dmg: process XML plists

2015-01-07 Thread John Snow
sed on the description at http://newosxbook.com/DMG.html Signed-off-by: Peter Wu --- v2: added offset check, allow resource fork to be located at beginning of file (removed `rsrc_fork_offset != 0` condition) It got a `Reviewed-by: John Snow ` for v1. I have not copied the R-b as I wanted to be

Re: [Qemu-devel] [PATCH v2 07/12] block/dmg: set virtual size to a non-zero value

2015-01-07 Thread John Snow
ngth != 0) { ret = dmg_read_resource_fork(bs, &ds, rsrc_fork_offset, rsrc_fork_length); You may want to adjust the commit message here to reflect the information in your new cover letter regarding the bug in dmg2img, but that's not worth re-spinning for. Reviewed-by: John Snow

Re: [Qemu-devel] [PATCH v2 08/12] block/dmg: fix sector data offset calculation

2015-01-07 Thread John Snow
read_uint64(bs, offset + 0x18, &ds.data_fork_offset); +if (ret < 0) { +goto fail; +} else if (ds.data_fork_offset > offset) { +ret = -EINVAL; +goto fail; +} + /* offset of resource fork (RsrcForkOffset) */ ret = read_uint64(bs, offset + 0x28, &rsrc_fork_offset); if (ret < 0) { Reviewed-by: John Snow

Re: [Qemu-devel] [PATCH v2 10/12] block/dmg: factor out block type check

2015-01-07 Thread John Snow
ize, &ds->max_sectors_per_chunk); +offset += 40; } s->n_chunks += chunk_count; return 0; Reviewed-by: John Snow

Re: [Qemu-devel] [PATCH v2 11/12] block/dmg: support bzip2 block entry types

2015-01-07 Thread John Snow
k/dmg.c | 43 ++- configure | 31 +++ 3 files changed, 74 insertions(+), 1 deletion(-) [snip] Reviewed-by: John Snow

Re: [Qemu-devel] [PATCH v2 12/12] block/dmg: improve zeroes handling

2015-01-07 Thread John Snow
offset_in_chunk = sector_num + i - s->sectors[s->current_chunk]; memcpy(buf + i * 512, s->uncompressed_chunk + sector_offset_in_chunk * 512, 512); Reviewed-by: John Snow

Re: [Qemu-devel] [PATCH v2 09/12] block/dmg: use SectorNumber from BLKX header

2015-01-07 Thread John Snow
patch after suggestions from John Snow to read and use these fields. --- block/dmg.c | 12 +--- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/block/dmg.c b/block/dmg.c index 130efac..57922c5 100644 --- a/block/dmg.c +++ b/block/dmg.c @@ -189,7 +189,6 @@ typedef struct

Re: [Qemu-devel] [PATCH 00/15] AHCI test helper refactors

2015-01-08 Thread John Snow
On 09/18/2014 07:43 PM, John Snow wrote: The original version of the AHCI test base which is now staged for being merged, processes the ahci_identify test in a monolithic fashion. In authoring new tests, it became necessary and obvious as to how the operation of this device should be factored

Re: [Qemu-devel] [PATCH 1/9] block: rename bdrv_reset_dirty_bitmap

2015-01-08 Thread John Snow
On 12/11/2014 09:17 AM, Vladimir Sementsov-Ogievskiy wrote: We will need functions for set/unset a subregion of BdrvDirtyBitmap, to fix migration and mirror (accordingly to the following patch "block: fix spoiling all dirty bitmaps by mirror and migration"). Having the old function 'bdrv_reset_

Re: [Qemu-devel] [PATCH 3/9] block: fix spoiling all dirty bitmaps by mirror and migration

2015-01-08 Thread John Snow
, to prevent such mistakes in future, old functions bdrv_(set,reset)_dirty are made static, for internal block usage. Signed-off-by: Vladimir Sementsov-Ogievskiy CC: John Snow CC: Fam Zheng CC: Denis V. Lunev CC: Stefan Hajnoczi CC: Kevin Wolf --- block-migration.c | 5 +++-- block.c

Re: [Qemu-devel] [PATCH 2/9] block-migration: fix pending() return value

2015-01-08 Thread John Snow
On 12/11/2014 09:17 AM, Vladimir Sementsov-Ogievskiy wrote: Because of wrong return value of .save_live_pending() in block-migration, migration finishes before the whole disk is transferred. Such situation occures when the migration (occurs) process is fast enouth, for example when source and

Re: [Qemu-devel] [PATCH 4/9] hbitmap: store / restore

2015-01-08 Thread John Snow
On 12/11/2014 09:17 AM, Vladimir Sementsov-Ogievskiy wrote: Functions to store / restore HBitmap. HBitmap should be saved to linear bitmap format independently of endianess. Because of restoring in several steps, every step writes only the last level of the bitmap. All other levels are restore

Re: [Qemu-devel] [PATCH 5/9] block: BdrvDirtyBitmap store/restore interface

2015-01-08 Thread John Snow
On 12/11/2014 09:17 AM, Vladimir Sementsov-Ogievskiy wrote: Several functions to provide necessary access to BdrvDirtyBitmap for block-migration.c Signed-off-by: Vladimir Sementsov-Ogievskiy --- block.c | 61 +-- include/block/b

Re: [Qemu-devel] [PATCH 6/9] block-migration: tiny refactoring

2015-01-08 Thread John Snow
-g_free(blk); +blk_free(blk); block_mig_state.read_done--; block_mig_state.transferred++; @@ -612,8 +617,7 @@ static void blk_mig_cleanup(void) while ((blk = QSIMPLEQ_FIRST(&block_mig_state.blk_list)) != NULL) { QSIMPLEQ_REMOVE_HEAD(&block_mig_state.blk_list, entry); -g_free(blk->buf); -g_free(blk); +blk_free(blk); } blk_mig_unlock(); } Reviewed-by: John Snow

Re: [Qemu-devel] [PATCH 7/9] block-migration: remove not needed iothread lock

2015-01-08 Thread John Snow
On 12/11/2014 09:17 AM, Vladimir Sementsov-Ogievskiy wrote: Instead of locking iothread, we can just swap these calls. So, if some write to our range occures before resetting the bitmap, then it will get into subsequent aio read, becouse it occures, in any case, after resetting the bitmap. s

Re: [Qemu-devel] [PATCH 8/9] migration: add dirty parameter

2015-01-08 Thread John Snow
CC'ing Eric Blake for monitor interface review. On 12/11/2014 09:17 AM, Vladimir Sementsov-Ogievskiy wrote: Add dirty parameter to qmp-migrate command. If this parameter is true, block-migration.c will migrate dirty bitmaps. This parameter can be used without "blk" parameter to migrate only dirt

Re: [Qemu-devel] [PATCH 9/9] block-migration: add named dirty bitmaps migration

2015-01-08 Thread John Snow
tion that qemu_loadvm_state launches a post_load method for that can invoke bdrv_dbm_restore_finish for us? Alternatively, can this not be safely done inside of block_load? somewhere around here: printf("Completed %d %%%c", (int)addr, (addr == 100) ? '\n' : &#

Re: [Qemu-devel] [PATCH 8/9] migration: add dirty parameter

2015-01-08 Thread John Snow
On 01/08/2015 05:29 PM, Eric Blake wrote: On 01/08/2015 02:51 PM, John Snow wrote: CC'ing Eric Blake for monitor interface review. Indeed, I already saw and reviewed the monitor interface in a mail dated Dec 11. Sorry, I missed that one. Thank you, though :) On 12/11/2014 09:

Re: [Qemu-devel] [PATCH v10 06/13] block: Add bdrv_copy_dirty_bitmap and bdrv_clear_dirty_bitmap

2015-01-09 Thread John Snow
think this should remain in sectors, for now. I really want to keep the granularity in bytes in this case, because I want to match the existing interface, and size makes sense to me in sectors. What I will do instead is just to document this quirk. Look out for V11 :) --John On 23.12.201

[Qemu-devel] [PATCH v11 06/13] hbitmap: add hbitmap_merge

2015-01-12 Thread John Snow
running speed for particularly sparse bitmaps by using iterators, but the running time for dense maps will be worse. We present the simpler solution first, and we can refine it later if needed. Signed-off-by: John Snow --- include/qemu/hbitmap.h | 11 +++ util/hbitmap.c | 28

[Qemu-devel] [PATCH v11 00/13] block: Incremental backup series

2015-01-12 Thread John Snow
retrieval of dirty bitmaps to and from permanent storage. Enough changes have been made that most Reviewed-By lines from previous iterations have been removed. (Sorry!) This series was originally authored by Fam Zheng; his cover letter is included below. ~John Snow

[Qemu-devel] [PATCH v11 04/13] block: Introduce bdrv_dirty_bitmap_granularity()

2015-01-12 Thread John Snow
From: Fam Zheng This returns the granularity (in bytes) of dirty bitmap, which matches the QMP interface and the existing query interface. Signed-off-by: Fam Zheng Signed-off-by: John Snow --- block.c | 9 +++-- include/block/block.h | 2 ++ 2 files changed, 9 insertions

[Qemu-devel] [PATCH v11 05/13] block: Add bdrv_clear_dirty_bitmap

2015-01-12 Thread John Snow
Signed-off-by: Fam Zheng Signed-off-by: John Snow --- block.c | 24 include/block/block.h | 1 + 2 files changed, 21 insertions(+), 4 deletions(-) diff --git a/block.c b/block.c index a1d9e88..7bf7079 100644 --- a/block.c +++ b/block.c @@ -53,6 +53,8

[Qemu-devel] [PATCH v11 08/13] block: Add bitmap successors

2015-01-12 Thread John Snow
y re-enable it. Signed-off-by: John Snow --- block.c | 123 -- blockdev.c| 14 ++ include/block/block.h | 10 3 files changed, 144 insertions(+), 3 deletions(-) diff --git a/block.c b/block.c index bd4b449..3f33

[Qemu-devel] [PATCH v11 03/13] qmp: Add block-dirty-bitmap-add and block-dirty-bitmap-remove

2015-01-12 Thread John Snow
The types added to block-core.json will be re-used in future patches in this series, see: 'qapi: Add transaction support to block-dirty-bitmap-{add, enable, disable}' Signed-off-by: Fam Zheng Signed-off-by: John Snow --- block.c | 20 ++ block/mirr

[Qemu-devel] [PATCH v11 11/13] qmp: Add dirty bitmap status fields in query-block

2015-01-12 Thread John Snow
From: Fam Zheng Adds the "enabled" and "frozen" status booleans. Signed-off-by: Fam Zheng Signed-off-by: John Snow --- block.c | 2 ++ qapi/block-core.json | 7 ++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/block.c b/block.c index 5eb

[Qemu-devel] [PATCH v11 13/13] block: BdrvDirtyBitmap miscellaneous fixup

2015-01-12 Thread John Snow
(1) granularity type consistency: Update the granularity to be uint64_t in all places. This value is always in bytes. (2) Some documentation for the fields within BdrvDirtyBitmap. Signed-off-by: John Snow --- block.c | 16 include/block/block.h | 2 +- 2

[Qemu-devel] [PATCH v11 12/13] qemu-iotests: Add tests for drive-backup sync=dirty-bitmap

2015-01-12 Thread John Snow
From: Fam Zheng Signed-off-by: Fam Zheng Signed-off-by: John Snow --- tests/qemu-iotests/056| 33 ++--- tests/qemu-iotests/056.out| 4 ++-- tests/qemu-iotests/iotests.py | 8 3 files changed, 40 insertions(+), 5 deletions(-) diff --git a

[Qemu-devel] [PATCH v11 01/13] block: fix spoiling all dirty bitmaps by mirror and migration

2015-01-12 Thread John Snow
in future, old functions bdrv_(set,reset)_dirty are made static, for internal block usage. Signed-off-by: Vladimir Sementsov-Ogievskiy CC: John Snow CC: Fam Zheng CC: Denis V. Lunev CC: Stefan Hajnoczi CC: Kevin Wolf Signed-off-by: John Snow --- block.c | 23

[Qemu-devel] [PATCH v11 09/13] qmp: Add support of "dirty-bitmap" sync mode for drive-backup

2015-01-12 Thread John Snow
For "dirty-bitmap" sync mode, the block job will iterate through the given dirty bitmap to decide if a sector needs backup (backup all the dirty clusters and skip clean ones), just as allocation conditions of "top" sync mode. Signed-off-by: Fam Zheng Signed-off-by: Joh

[Qemu-devel] [PATCH v11 07/13] qmp: Add block-dirty-bitmap-enable and block-dirty-bitmap-disable

2015-01-12 Thread John Snow
From: Fam Zheng This allows to put the dirty bitmap into a disabled state where no more writes will be tracked. It will be used before backup or writing to persistent file. Signed-off-by: Fam Zheng Signed-off-by: John Snow --- block.c | 24 +++- blockdev.c

[Qemu-devel] [PATCH v11 02/13] qapi: Add optional field "name" to block dirty bitmap

2015-01-12 Thread John Snow
bdrv_find_dirty_bitmap to find a dirty bitmap by name, will be used later when other QMP commands want to reference dirty bitmap by name. Add bdrv_dirty_bitmap_make_anon. This unsets the name of dirty bitmap. Signed-off-by: Fam Zheng Signed-off-by: John Snow Reviewed-by: Max Reitz --- block.c

[Qemu-devel] [PATCH v11 10/13] qapi: Add transaction support to block-dirty-bitmap-{add, enable, disable}

2015-01-12 Thread John Snow
From: Fam Zheng This adds three qmp commands to transactions. Users can stop a dirty bitmap, start backup of it, and start another dirty bitmap atomically, so that the dirty bitmap is tracked incrementally and we don't miss any write. Signed-off-by: Fam Zheng Signed-off-by: John

Re: [Qemu-devel] [PATCH 9/9] block-migration: add named dirty bitmaps migration

2015-01-12 Thread John Snow
On 01/12/2015 11:48 AM, Paolo Bonzini wrote: On 11/12/2014 15:17, Vladimir Sementsov-Ogievskiy wrote: Just migrate parts of dirty bitmaps, corresponding to the block being migrated. Also, skip block migration if it is disabled (blk parameter of migrate command is false). So I have misread

[Qemu-devel] [PATCH 05/14] libqos: add alloc_init_flags

2015-01-12 Thread John Snow
Allow a generic interface to alloc_init_flags, not just through pc_alloc_init_flags. Signed-off-by: John Snow --- tests/libqos/malloc-pc.c | 4 +--- tests/libqos/malloc.c| 8 tests/libqos/malloc.h| 2 ++ 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/tests

[Qemu-devel] [PATCH 00/14] ahci-test preliminary refactoring

2015-01-12 Thread John Snow
facilitate migration testing inside of qtests, for which I intend to export to all of qtests to be re-used for other tests. >From this point forward, the churn to my AHCI refactor series should be significantly reduced and more pleasant to read. Thanks, John. John Snow (14): libqos: Split ap

[Qemu-devel] [PATCH 10/14] qtest/ahci: remove guest_malloc global

2015-01-12 Thread John Snow
Make helper routines rely on the earmarked guest allocator object with AHCIQState/QOSSTate instead. Signed-off-by: John Snow --- tests/ahci-test.c | 18 +- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/tests/ahci-test.c b/tests/ahci-test.c index 1981b6a..b84f942

[Qemu-devel] [PATCH 01/14] libqos: Split apart pc_alloc_init

2015-01-12 Thread John Snow
Move the list-specific initialization over into malloc.c, to keep all of the list implementation details within the same file. The allocation and freeing of these structures are now both back within the same layer. Signed-off-by: John Snow --- tests/libqos/malloc-pc.c | 20

[Qemu-devel] [PATCH 04/14] libqos: add qtest_vboot

2015-01-12 Thread John Snow
Add a va_list variant of the qtest_boot function. Signed-off-by: John Snow --- tests/libqos/libqos.c | 25 +++-- tests/libqos/libqos.h | 1 + 2 files changed, 20 insertions(+), 6 deletions(-) diff --git a/tests/libqos/libqos.c b/tests/libqos/libqos.c index c478bc9..c8b3ef0

[Qemu-devel] [PATCH 06/14] libqos: add pc specific interface

2015-01-12 Thread John Snow
-by: John Snow --- tests/Makefile | 6 +++--- tests/ahci-test.c| 4 ++-- tests/libqos/libqos-pc.c | 24 tests/libqos/libqos-pc.h | 9 + tests/libqos/libqos.c| 16 +--- tests/libqos/libqos.h| 10 -- 6 files changed, 55

[Qemu-devel] [PATCH 07/14] qtest/ahci: Store hba_base in AHCIQState

2015-01-12 Thread John Snow
*hba_base/(AHCIQState *ahci/' # Replace (ahci->dev, hba_base) calls with unified parameter 's/(ahci->dev, &\?hba_base)/(ahci)/' # Replace calls to PCI config space using "ahci" with "ahci->dev" 's/qpci_config_\(read\|write\)\(.\)(ahc

<    2   3   4   5   6   7   8   9   10   11   >