As of version 3, the qcow2 file format supports different widths for
refcount entries, ranging from 1 to 64 bit (only powers of two).
Currently, qemu only supports 16 bit, which is the only width supported
by version 2 (compat=0.10) images.
This series adds support to qemu for all other valid refcount orders.
This is mainly done by adding two function pointers into the
BDRVQcowState structure for reading and writing refcount values
independently of the current refcount entry width; all in-memory
refcount arrays (mostly cached refcount blocks) now are void pointers
and are accessed through these functions alone.
Thanks to previous work of making the qemu code agnostic of e.g. the
number of refcount entries per refcount block, the rest is fairly
trivial. The most complex patch in this series is patch 18 which
implements changing the refcount order through qemu-img amend.
To test different refcount widths, simply invoke the qemu-iotests check
program with -o refcount_width=${your_desired_width}. The final test in
this series adds some tests for operations which do not work with
certain refcount orders and for refcount order amendment.
As of this version, this series depends on version 3 of my
"chardev: Add -qmp-pretty" series (due to different test output of test
067, which makes changing it here much nicer).
v2:
- Patch 2:
- Added justification for always emitting refcount-width to the commit
message [Eric]
- Due to making this series dependent on the -qmp-pretty series, the
reference output of iotest 067 is changed differently
- Patch 3: Added justification for using int64_t instead of uint64_t to
the commit message [Eric]
- Patch 6: This replaces patch 7 from v1.
- Added an assertion and an explanation why refcount_array_byte_size()
cannot overflow [Eric]
- Added a helper function for reallocating a refcount array (this is
what we really want); make use of that function
- Patch 7: Was patch 6 in v1, swapped the order to reduce the diffstat
and not having to explain why a truncating division does not truncate
here [Eric]
- Added overflow check in update_refcount()
- Consequent use of int64_t for refcounts [Eric]
- Use g_try_malloc0_n() instead of g_try_malloc0() once to prevent a
multiplication overflow [Eric]
- Dropped the temporary on_disk_refblock buffer in
rebuild_refcount_structure(); just directly write the in-memory
refcount array to disk
- Patch 8: Added an assertion in set_refcount_ro6() that the MSb of the
given value is not set (which makes the value fit into an int64_t)
[Eric]
- Patch 9: Dropped the refcount width specified by the image header from
the output to prevent overflows [Eric]
- Patch 10: Kept the same, because this calculation was never meant to
be exact
- Patch 11:
- Added comments why certain refcount_widths to do not work with some
tests [Eric]
- Changed the _unsupported_imgopts argument for allowing only
refcount_width=16 to disallow any other refcount width instead of
listing them all (your proposal did not work, though, Eric)
- Fixed comment in test 007 [Eric]
- 079 does support any refcount width, we only need to use
_make_test_img instead of calling qemu-img create manually, so do
that
- Disallowing 090 for refcount_width=1 was an artifact from before v1
(where qcow2_alloc_bytes() simply returned an error when a refcount
overflow was about to occur, instead of skipping to the next
cluster), so remove that limitation
- Patch 12: Dropped 079 reference output change because patch 11 took
care of that (by using _make_test_img)
- Patch 14: Removed trailing full stops in error_report() calls [Eric]
- Patch 17:
- Fixed indentation for the function header of qcow2_amend_helper_cb()
[Eric]
- Renamed arguments of qcow2_amend_helper_cb() to make it more obvious
what they mean in the context of this function [Eric, in a way]
- Added to assertions regarding total_operations [Eric]
- Patch 18:
- Add a typedef for the function pointer which is given to
walk_over_reftable(); the benefit is that this makes documentation
of the parameters easier [Eric]
- Add an "allocated" parameter to that RefblockFinishOp and
consequently to walk_over_reftable() so that changes in the original
refcount structures can be tracked; if anything has been changed,
walk_over_reftable() is rerun with alloc_refblock() until no new
allocations were necessary anymore
- Some changes to alloc_refblock() and flush_refblock() to accommodate
to this change
- Call the status_cb additionally once at the end of
walk_over_reftable() [Eric]
- Fix leak of the new reftable clusters on error [Eric]
- Patch 21:
- Added comment that snapshotting an image with a small refcount width
fails for now, but may work in the future [Eric]
- %s/shoud/should/ [Eric]
- 64s/qocw2/qcow2/
- Added comment that failing to amend an image because of refcount
overflows may work in the future [Eric]
- 232s/entriy/entry/
- Added a test for multiple walks during refcount width amendment (for
the changes to patch 18)
git-backport-diff against v1:
Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
001/21:[----] [--] 'qcow2: Add two new fields to BDRVQcowState'
002/21:[0015] [FC] 'qcow2: Add refcount_width to format-specific info'
003/21:[----] [--] 'qcow2: Use 64 bits for refcount values'
004/21:[----] [--] 'qcow2: Respect error in qcow2_alloc_bytes()'
005/21:[----] [--] 'qcow2: Refcount overflow and qcow2_alloc_bytes()'
006/21:[down] 'qcow2: Helper for refcount array reallocation'
007/21:[0062] [FC] 'qcow2: Helper function for refcount modification'
008/21:[0003] [FC] 'qcow2: More helpers for refcount modification'
009/21:[0004] [FC] 'qcow2: Open images with refcount order != 4'
010/21:[----] [--] 'qcow2: refcount_order parameter for qcow2_create2'
011/21:[0096] [FC] 'iotests: Prepare for refcount_width option'
012/21:[0018] [FC] 'qcow2: Allow creation with refcount order != 4'
013/21:[----] [--] 'block: Add opaque value to the amend CB'
014/21:[0018] [FC] 'qcow2: Use error_report() in qcow2_amend_options()'
015/21:[----] [--] 'qcow2: Use abort() instead of assert(false)'
016/21:[----] [--] 'qcow2: Split upgrade/downgrade paths for amend'
017/21:[0014] [FC] 'qcow2: Use intermediate helper CB for amend'
018/21:[0159] [FC] 'qcow2: Add function for refcount order amendment'
019/21:[----] [--] 'qcow2: Invoke refcount order amendment function'
020/21:[----] [--] 'qcow2: Point to amend function in check'
021/21:[0051] [FC] 'iotests: Add test for different refcount widths'
Max Reitz (21):
qcow2: Add two new fields to BDRVQcowState
qcow2: Add refcount_width to format-specific info
qcow2: Use 64 bits for refcount values
qcow2: Respect error in qcow2_alloc_bytes()
qcow2: Refcount overflow and qcow2_alloc_bytes()
qcow2: Helper for refcount array reallocation
qcow2: Helper function for refcount modification
qcow2: More helpers for refcount modification
qcow2: Open images with refcount order != 4
qcow2: refcount_order parameter for qcow2_create2
iotests: Prepare for refcount_width option
qcow2: Allow creation with refcount order != 4
block: Add opaque value to the amend CB
qcow2: Use error_report() in qcow2_amend_options()
qcow2: Use abort() instead of assert(false)
qcow2: Split upgrade/downgrade paths for amend
qcow2: Use intermediate helper CB for amend
qcow2: Add function for refcount order amendment
qcow2: Invoke refcount order amendment function
qcow2: Point to amend function in check
iotests: Add test for different refcount widths
block.c | 4 +-
block/qcow2-cluster.c | 23 +-
block/qcow2-refcount.c | 946 +++++++++++++++++++++++++++++++++------
block/qcow2.c | 256 ++++++++---
block/qcow2.h | 24 +-
include/block/block.h | 4 +-
include/block/block_int.h | 4 +-
qapi/block-core.json | 5 +-
qemu-img.c | 5 +-
tests/qemu-iotests/007 | 3 +
tests/qemu-iotests/015 | 2 +
tests/qemu-iotests/026 | 7 +
tests/qemu-iotests/029 | 1 +
tests/qemu-iotests/049.out | 112 ++---
tests/qemu-iotests/051 | 3 +
tests/qemu-iotests/058 | 2 +
tests/qemu-iotests/060.out | 1 +
tests/qemu-iotests/061.out | 14 +-
tests/qemu-iotests/065 | 23 +-
tests/qemu-iotests/067 | 2 +
tests/qemu-iotests/067.out | 5 +
tests/qemu-iotests/079 | 10 +-
tests/qemu-iotests/079.out | 38 +-
tests/qemu-iotests/080 | 2 +
tests/qemu-iotests/082.out | 48 +-
tests/qemu-iotests/085.out | 38 +-
tests/qemu-iotests/089 | 2 +
tests/qemu-iotests/089.out | 2 +
tests/qemu-iotests/108 | 2 +
tests/qemu-iotests/112 | 252 +++++++++++
tests/qemu-iotests/112.out | 131 ++++++
tests/qemu-iotests/common.filter | 3 +-
tests/qemu-iotests/group | 1 +
33 files changed, 1644 insertions(+), 331 deletions(-)
create mode 100755 tests/qemu-iotests/112
create mode 100644 tests/qemu-iotests/112.out
--
1.9.3
