On Fri, Apr 17, 2015 at 07:49:48PM -0400, John Snow wrote:
> It's spring! The winter snow has thawed and so here is a new
> patch series to enter your life and warm your heart.
>
> This patchset enables the in-memory part of the incremental backup
> feature, without transactional support.
>
> Support for transactions was separated into a separate series which
> is also now available on-list. Getting this portion of the series
> committed will help stabilize work on bitmap persistence and bitmap
> migration.
>
> Thanks to Fam Zheng for the original versions of this patchset,
> And thanks to Max and Eric for reviewing 2,396 versions of it since.
>
> ===
> v6:
> ===
>
> 01: s/underlaying/underlying/
> Removed a reference to 'disabled' bitmaps.
> Touching up inconsistent list indentation.
> Added FreeBSD Documentation License, primarily to be difficult
> 07: More in-line documentation for hbitmap_merge, for return value.
> Fix size cache index to be uint64_t.
> 09: Grammar fixes from Eric Blake, kept R-Bs.
> 10: Moved yield into the do{}while(). Now we check to see if we should
> yield/cancel after each unsuccessful sector we transfer.
> Some documentation additions for Eric Blake.
> 15: corrected 'num_elements' to 'start'
> 18: Refactored qmp.py event functions,
> Added in more explicit exception classes.
> No changes to iotests.py, just qmp.py.
>
> Key:
> [----] : patches are identical
> [####] : number of functional differences between upstream/downstream patch
> [down] : patch is downstream-only
> The flags [FC] indicate (F)unctional and (C)ontextual differences,
> respectively
>
> 001/21:[0057] [FC] 'docs: incremental backup documentation'
> 002/21:[----] [--] 'qapi: Add optional field "name" to block dirty bitmap'
> 003/21:[----] [--] 'qmp: Ensure consistent granularity type'
> 004/21:[----] [--] 'qmp: Add block-dirty-bitmap-add and
> block-dirty-bitmap-remove'
> 005/21:[----] [--] 'block: Introduce bdrv_dirty_bitmap_granularity()'
> 006/21:[----] [--] 'hbitmap: cache array lengths'
> 007/21:[0008] [FC] 'hbitmap: add hbitmap_merge'
> 008/21:[----] [--] 'block: Add bitmap disabled status'
> 009/21:[0008] [FC] 'block: Add bitmap successors'
> 010/21:[0013] [FC] 'qmp: Add support of "dirty-bitmap" sync mode for
> drive-backup'
> 011/21:[----] [--] 'qmp: add block-dirty-bitmap-clear'
> 012/21:[----] [--] 'qmp: Add dirty bitmap status field in query-block'
> 013/21:[----] [--] 'block: add BdrvDirtyBitmap documentation'
> 014/21:[----] [--] 'block: Ensure consistent bitmap function prototypes'
> 015/21:[0002] [FC] 'block: Resize bitmaps on bdrv_truncate'
> 016/21:[----] [--] 'hbitmap: truncate tests'
> 017/21:[----] [-C] 'iotests: add invalid input incremental backup tests'
> 018/21:[0095] [FC] 'iotests: add QMP event waiting queue'
> 019/21:[----] [--] 'iotests: add simple incremental backup case'
> 020/21:[----] [--] 'iotests: add incremental backup failure recovery test'
> 021/21:[----] [--] 'iotests: add incremental backup granularity tests'
>
> ===
> v5:
> ===
>
> 10: Code has been moved into backup_run_incremental()
> 'polyrhythm' check is removed,
> clusters_per_iter variable is introduced instead.
> If the bitmap granularity is larger than the backup granularity,
> loop over the backup_do_cow call multiple times.
> If the bitmap granularity is smaller, skip the iterator ahead as
> we had been doing previously.
> 14: Only whitespace changes caused by patch 10.
> 15: Changed my approach for clearing out data for the hbitmap
> truncate shrink case, as suggested by Stefan
> 18: Added a proper timeout mechanism to qmp.pull_event():
> wait=False or wait=0.0 implies non-blocking.
> wait=True implies blocking.
> wait=60.0 implies a 60 second timeout.
> VM.event_wait() now uses a 60 second timeout by default.
> 19: Many things:
> The big picture is to add a set of full backups alongside the
> incremental backups created during the test to be able to test
> the validity of each incremental at the conclusion of the test
> when we can shut the VM down.
>
> To do this, there are two basic changes:
> (1) Keep a list of pairs of backup filenames (incremental, reference);
> create a full reference backup for every incremental created.
> (2) Refactor the backup helper functions a bit.
> 20: Naming fallout from 19
> Added calls to vm.shutdown() and check_backups().
> 21: NEW, adds granularity tests that cover the changes in patch 10.
>
> ===
> v4:
> ===
>
> 04: Some in-line documentation for block_dirty_bitmap_lookup
> Changed behavior with respect to aio_context
> (always acquire, release if pbs == null, give to user otherwise)
> 10: Removed vestigial (currently nop) bdrv_enable_dirty_bitmap call
> Kept R-B.
> 16: Added some comments to test_check_boundary_bits.
> Kept R-B.
> 17: Folded in refactor from "incremental transactions v1" (Poor Kitty)
> 18: Pulled forward from "incremental transactions v1"
> Kept R-B from that series.
> 19: Folded in refactor from "incremental transactions v1"
> Added offset assertions into wait_incremental
> 20: Removed error tolerance from wait_until_completed, as
> these patches no longer make use of that feature.
>
> ===
> v3:
> ===
>
> 01: Removed enabled/disabled modes information.
> Elaborated on events that can occur during error cases.
> 04: Added an AioContext out parameter to block_dirty_bitmap_lookup.
> 06: NEW:
> Cache the array lengths for hbitmap.
> 07: hbitmap_merge now uses the cached array lengths.
> 11: block-dirty-bitmap-clear is edited for the new block_dirty_bitmap_lookup.
> 12: Removed the "disabled" status, leaving just "Frozen."
> 15: Moved bdrv_truncate_dirty_bitmap to be static local
> Inlined dirty_bitmap_truncate function.
> Removed size[] caching into new patch (06, above)
> hbitmap_truncate now keeps correct bit population count
> hbitmap_truncate now uses hbitmap_reset BEFORE the truncate,
> to avoid tricky out-of-bounds usages.
> Remove g_realloc_n call that is not available in glib 2.12 (or 2.22)
> Renamed "truncate" to "shrink" to make that more clear
> to people who aren't me (at last count: 7+ billion)
> 16 NEW:
> hbitmap_truncate tests.
>
> ===
> v2:
> ===
>
> 01: Added a new opening blurb.
> Adjusted codeblock indentations to be 4 spaces instead of 3,
> so it works as MD or GFMD.
> Adjusted errors explanation.
> Make visual separations between json data and shell commands
> Eliminate any ligering single quotes
>
> 07: Remember that memset takes bytes, not n_items ...
>
> ===
> v1:
> ===
>
> Deletions:
> - Removed Transactions, to be added later.
> - Removed Transaction tests, as above.
>
> Changes:
> 01: Indentation fixes.
> Removed enable/disable documentation.
> Added a note that transactions aren't implemented yet.
> Removed my needless commas
> Added error case documentation.
>
> 07: QMP enable/disable commands are deleted.
>
> 14: Some comments concerning assertions.
> Scrub re-alloc memory if we expand the array.
> Do not attempt to scrub memory if fix_count is 0
>
> Changes made with Reviews kept:
>
> 02: Since 2.4
> 04: Since 2.4
> Demingled the QMP command documentation.
> 08: Additions to what was qmp_block_dirty_enable/disable
> are no longer present as those function no longer exist.
> 09: Since 2.4
> 10: Since 2.4
> Demingled QMP command documentation.
> 11: Since 2.4
> 15: Test 112 --> 124
> 17: Number of tests altered. (Only 4, now.)
>
> Fam Zheng (1):
> qapi: Add optional field "name" to block dirty bitmap
>
> John Snow (20):
> docs: incremental backup documentation
> qmp: Ensure consistent granularity type
> qmp: Add block-dirty-bitmap-add and block-dirty-bitmap-remove
> block: Introduce bdrv_dirty_bitmap_granularity()
> hbitmap: cache array lengths
> hbitmap: add hbitmap_merge
> block: Add bitmap disabled status
> block: Add bitmap successors
> qmp: Add support of "dirty-bitmap" sync mode for drive-backup
> qmp: add block-dirty-bitmap-clear
> qmp: Add dirty bitmap status field in query-block
> block: add BdrvDirtyBitmap documentation
> block: Ensure consistent bitmap function prototypes
> block: Resize bitmaps on bdrv_truncate
> hbitmap: truncate tests
> iotests: add invalid input incremental backup tests
> iotests: add QMP event waiting queue
> iotests: add simple incremental backup case
> iotests: add incremental backup failure recovery test
> iotests: add incremental backup granularity tests
>
> block.c | 243 ++++++++++++++++++++++++++--
> block/backup.c | 155 +++++++++++++++---
> block/mirror.c | 46 +++---
> blockdev.c | 176 +++++++++++++++++++-
> docs/bitmaps.md | 352 ++++++++++++++++++++++++++++++++++++++++
> hmp.c | 3 +-
> include/block/block.h | 33 +++-
> include/block/block_int.h | 4 +-
> include/qemu/hbitmap.h | 23 +++
> migration/block.c | 9 +-
> qapi/block-core.json | 91 ++++++++++-
> qmp-commands.hx | 93 ++++++++++-
> scripts/qmp/qmp.py | 95 +++++++----
> tests/qemu-iotests/124 | 363
> ++++++++++++++++++++++++++++++++++++++++++
> tests/qemu-iotests/124.out | 5 +
> tests/qemu-iotests/group | 1 +
> tests/qemu-iotests/iotests.py | 38 +++++
> tests/test-hbitmap.c | 255 +++++++++++++++++++++++++++++
> util/hbitmap.c | 85 ++++++++++
> 19 files changed, 1953 insertions(+), 117 deletions(-)
> create mode 100644 docs/bitmaps.md
> create mode 100644 tests/qemu-iotests/124
> create mode 100644 tests/qemu-iotests/124.out
>
> --
> 2.1.0
> Thanks, applied to my block-next tree: https://github.com/stefanha/qemu/commits/block-next Stefan
pgp15lVd_ZrgL.pgp
Description: PGP signature
