> -----Original Message-----
> From: Stefan Hajnoczi [mailto:stefa...@gmail.com]
> Sent: Tuesday, August 02, 2011 9:06 PM
> To: Shribman, Aidan
> Cc: qemu-devel@nongnu.org; Anthony Liguori
> Subject: Re: [PATCH v3] XBZRLE delta for live migration of
> large memory apps
>
> On Tue, Aug 02, 2011 at 03:45:56PM +0200, Shribman, Aidan wrote:
> > Subject: [PATCH v3] XBZRLE delta for live migration of
> large memory apps
> > From: Aidan Shribman <aidan.shrib...@sap.com>
> >
> > By using XBZRLE (Xor Binary Zero Run-Length-Encoding) we
> can reduce VM downtime
> > and total live-migration time for VMs running memory write
> intensive workloads
> > typical of large enterprise applications such as SAP ERP
> Systems, and generally
> > speaking for representative of any application with a
> sparse memory update pattern.
> >
> > On the sender side XBZRLE is used as a compact delta
> encoding of page updates,
> > retrieving the old page content from an LRU cache (default
> size of 64 MB). The
> > receiving side uses the existing page content and XBZRLE to
> decode the new page
> > content.
> >
> > Work was originally based on research results published VEE
> 2011: Evaluation of
> > Delta Compression Techniques for Efficient Live Migration
> of Large Virtual
> > Machines by Benoit, Svard, Tordsson and Elmroth.
> Additionally the delta encoder
> > XBRLE was improved further using XBZRLE instead.
> >
> > XBZRLE has a sustained bandwidth of 1.5-2.2 GB/s for
> typical workloads making it
> > ideal for in-line, real-time encoding such as is needed for
> live-migration.
>
> What is the CPU cost of xbzrle live migration on the source host? I'm
> thinking about a graph showing CPU utilization (e.g. from mpstat(1))
> that has two datasets: migration without xbzrle and migration with
> xbzrle.
>
zbzrle.out indicates that xbzrle is using 50% of the compute capacity during
the xbzrle live-migration (which completed is few seconds), In vanilla.out
between 30%-60% of compute is directed toward the live-migration itself - in
this case live-migration is not able to complete.
-----
root@ilrsh01:~#
root@ilrsh01:~# cat xbzrle.out
Linux 2.6.35-22-server (ilrsh01) 08/07/2011 _x86_64_ (2 CPU)
10:55:37 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
10:55:38 AM all 40.50 0.00 1.00 1.50 0.00 9.00 0.00
0.00 48.00
10:55:38 AM 0 0.00 0.00 1.00 3.00 0.00 0.00 0.00
0.00 96.00
10:55:38 AM 1 81.00 0.00 1.00 0.00 0.00 18.00 0.00
0.00 0.00
10:55:38 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
10:55:39 AM all 47.00 0.00 1.50 0.00 0.00 2.00 0.00
0.00 49.50
10:55:39 AM 0 0.00 0.00 1.00 0.00 0.00 0.00 0.00
0.00 99.00
10:55:39 AM 1 94.00 0.00 2.00 0.00 0.00 4.00 0.00
0.00 0.00
10:55:39 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
10:55:40 AM all 50.00 0.00 0.50 0.00 0.00 0.00 0.00
0.00 49.50
10:55:40 AM 0 0.00 0.00 1.00 0.00 0.00 0.00 0.00
0.00 99.00
10:55:40 AM 1 100.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
10:55:40 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
10:55:41 AM all 49.75 0.00 1.99 0.00 0.00 0.00 0.00
0.00 48.26
10:55:41 AM 0 10.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 90.00
10:55:41 AM 1 89.11 0.00 3.96 0.00 0.00 0.00 0.00
0.00 6.93
10:55:41 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
10:55:42 AM all 47.26 0.00 8.96 0.00 0.00 1.99 0.00
0.00 41.79
10:55:42 AM 0 51.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 49.00
10:55:42 AM 1 43.56 0.00 17.82 0.00 0.00 3.96 0.00
0.00 34.65
10:55:42 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
10:55:43 AM all 50.00 0.00 11.50 2.00 0.00 1.00 0.00
0.00 35.50
10:55:43 AM 0 100.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
10:55:43 AM 1 0.00 0.00 23.00 4.00 0.00 2.00 0.00
0.00 71.00
10:55:43 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
10:55:44 AM all 50.00 0.00 22.00 0.00 0.00 0.00 0.00
0.00 28.00
10:55:44 AM 0 100.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
10:55:44 AM 1 0.00 0.00 44.00 0.00 0.00 0.00 0.00
0.00 56.00
10:55:44 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
10:55:45 AM all 49.50 0.00 23.50 0.00 0.00 2.00 0.00
0.00 25.00
10:55:45 AM 0 99.00 0.00 1.00 0.00 0.00 0.00 0.00
0.00 0.00
10:55:45 AM 1 0.00 0.00 46.00 0.00 0.00 4.00 0.00
0.00 50.00
10:55:45 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
10:55:46 AM all 13.02 0.00 5.58 0.00 0.00 0.00 0.00
0.00 81.40
10:55:46 AM 0 25.71 0.00 0.00 0.00 0.00 0.00 0.00
0.00 74.29
10:55:46 AM 1 0.91 0.00 10.91 0.00 0.00 0.00 0.00
0.00 88.18
10:55:46 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
10:55:47 AM all 0.00 0.00 0.44 0.00 0.00 0.00 0.00
0.00 99.56
10:55:47 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00
10:55:47 AM 1 0.00 0.00 0.83 0.00 0.00 0.00 0.00
0.00 99.17
10:55:47 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
10:55:48 AM all 0.00 0.00 1.28 2.13 0.00 0.00 0.00
0.00 96.60
10:55:48 AM 0 0.00 0.00 0.94 0.94 0.00 0.00 0.00
0.00 98.11
10:55:48 AM 1 0.00 0.00 1.55 3.10 0.00 0.00 0.00
0.00 95.35
10:55:48 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
10:55:49 AM all 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00
10:55:49 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00
10:55:49 AM 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00
-----
root@ilrsh01:~# cat vanilla.out
Linux 2.6.35-22-server (ilrsh01) 08/07/2011 _x86_64_ (2 CPU)
11:13:53 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:13:54 AM all 49.50 0.00 0.50 0.50 0.00 0.00 0.00
0.00 49.50
11:13:54 AM 0 0.00 0.00 0.00 1.00 0.00 0.00 0.00
0.00 99.00
11:13:54 AM 1 99.00 0.00 1.00 0.00 0.00 0.00 0.00
0.00 0.00
11:13:54 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:13:55 AM all 49.50 0.00 0.50 0.00 0.00 0.00 0.00
0.00 50.00
11:13:55 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00
11:13:55 AM 1 99.00 0.00 1.00 0.00 0.00 0.00 0.00
0.00 0.00
11:13:55 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:13:56 AM all 44.78 0.00 15.42 0.00 0.00 1.00 0.00
0.00 38.81
11:13:56 AM 0 37.00 0.00 7.00 0.00 0.00 0.00 0.00
0.00 56.00
11:13:56 AM 1 52.48 0.00 23.76 0.00 0.00 1.98 0.00
0.00 21.78
11:13:56 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:13:57 AM all 48.50 0.00 25.00 0.00 0.00 2.00 0.00
0.00 24.50
11:13:57 AM 0 97.00 0.00 3.00 0.00 0.00 0.00 0.00
0.00 0.00
11:13:57 AM 1 0.00 0.00 47.00 0.00 0.00 4.00 0.00
0.00 49.00
11:13:57 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:13:58 AM all 48.50 0.00 17.00 0.00 0.00 3.00 0.00
0.00 31.50
11:13:58 AM 0 97.00 0.00 2.00 0.00 0.00 1.00 0.00
0.00 0.00
11:13:58 AM 1 0.00 0.00 32.00 0.00 0.00 5.00 0.00
0.00 63.00
11:13:58 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:13:59 AM all 47.50 0.00 1.00 0.50 0.00 2.50 0.00
0.00 48.50
11:13:59 AM 0 8.00 0.00 0.00 1.00 0.00 0.00 0.00
0.00 91.00
11:13:59 AM 1 87.00 0.00 2.00 0.00 0.00 5.00 0.00
0.00 6.00
11:13:59 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:00 AM all 48.02 0.00 4.95 0.00 0.00 1.98 0.00
0.00 45.05
11:14:00 AM 0 56.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 44.00
11:14:00 AM 1 40.20 0.00 9.80 0.00 0.00 3.92 0.00
0.00 46.08
11:14:00 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:01 AM all 48.76 0.00 14.43 0.00 0.00 1.99 0.00
0.00 34.83
11:14:01 AM 0 59.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 41.00
11:14:01 AM 1 38.61 0.00 28.71 0.00 0.00 3.96 0.00
0.00 28.71
11:14:01 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:02 AM all 46.77 0.00 9.45 0.00 0.00 1.99 0.00
0.00 41.79
11:14:02 AM 0 30.00 0.00 4.00 0.00 0.00 0.00 0.00
0.00 66.00
11:14:02 AM 1 63.37 0.00 14.85 0.00 0.00 3.96 0.00
0.00 17.82
11:14:02 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:03 AM all 48.76 0.00 7.46 0.00 0.00 2.49 0.00
0.00 41.29
11:14:03 AM 0 54.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 46.00
11:14:03 AM 1 43.56 0.00 14.85 0.00 0.00 4.95 0.00
0.00 36.63
11:14:03 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:04 AM all 47.50 0.00 5.50 0.00 0.00 2.50 0.00
0.00 44.50
11:14:04 AM 0 28.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 72.00
11:14:04 AM 1 67.00 0.00 11.00 0.00 0.00 5.00 0.00
0.00 17.00
11:14:04 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:05 AM all 46.77 0.00 3.48 0.00 0.00 2.99 0.00
0.00 46.77
11:14:05 AM 0 16.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 84.00
11:14:05 AM 1 77.23 0.00 6.93 0.00 0.00 5.94 0.00
0.00 9.90
11:14:05 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:06 AM all 49.25 0.00 13.43 0.00 0.00 1.49 0.00
0.00 35.82
11:14:06 AM 0 62.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 38.00
11:14:06 AM 1 36.63 0.00 26.73 0.00 0.00 2.97 0.00
0.00 33.66
11:14:06 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:07 AM all 48.50 0.00 15.00 0.00 0.00 3.00 0.00
0.00 33.50
11:14:07 AM 0 67.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 33.00
11:14:07 AM 1 30.00 0.00 30.00 0.00 0.00 6.00 0.00
0.00 34.00
11:14:07 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:08 AM all 46.77 0.00 14.93 0.00 0.00 1.99 0.00
0.00 36.32
11:14:08 AM 0 64.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 36.00
11:14:08 AM 1 29.70 0.00 29.70 0.00 0.00 3.96 0.00
0.00 36.63
11:14:08 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:09 AM all 50.25 0.00 11.44 1.00 0.00 2.99 0.00
0.00 34.33
11:14:09 AM 0 61.00 0.00 0.00 2.00 0.00 0.00 0.00
0.00 37.00
11:14:09 AM 1 39.60 0.00 22.77 0.00 0.00 5.94 0.00
0.00 31.68
11:14:09 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:10 AM all 47.76 0.00 3.48 0.00 0.00 2.99 0.00
0.00 45.77
11:14:10 AM 0 41.00 0.00 1.00 0.00 0.00 0.00 0.00
0.00 58.00
11:14:10 AM 1 54.46 0.00 5.94 0.00 0.00 5.94 0.00
0.00 33.66
11:14:10 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:11 AM all 46.04 0.00 11.39 0.00 0.00 4.46 0.00
0.00 38.12
11:14:11 AM 0 49.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 51.00
11:14:11 AM 1 43.14 0.00 22.55 0.00 0.00 8.82 0.00
0.00 25.49
11:14:11 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:12 AM all 47.52 0.00 2.97 0.00 0.00 2.97 0.00
0.00 46.53
11:14:12 AM 0 15.00 0.00 5.00 0.00 0.00 0.00 0.00
0.00 80.00
11:14:12 AM 1 79.41 0.00 0.98 0.00 0.00 5.88 0.00
0.00 13.73
11:14:12 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:13 AM all 48.76 0.00 7.46 0.00 0.00 1.99 0.00
0.00 41.79
11:14:13 AM 0 55.00 0.00 1.00 0.00 0.00 0.00 0.00
0.00 44.00
11:14:13 AM 1 42.57 0.00 13.86 0.00 0.00 3.96 0.00
0.00 39.60
11:14:13 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:14 AM all 49.00 0.00 12.00 1.00 0.00 2.00 0.00
0.00 36.00
11:14:14 AM 0 58.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 42.00
11:14:14 AM 1 40.00 0.00 24.00 2.00 0.00 4.00 0.00
0.00 30.00
11:14:14 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:15 AM all 48.51 0.00 5.94 0.00 0.00 2.48 0.00
0.00 43.07
11:14:15 AM 0 59.00 0.00 1.00 0.00 0.00 0.00 0.00
0.00 40.00
11:14:15 AM 1 38.24 0.00 10.78 0.00 0.00 4.90 0.00
0.00 46.08
11:14:15 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:16 AM all 48.26 0.00 13.43 0.00 0.00 2.49 0.00
0.00 35.82
11:14:16 AM 0 61.00 0.00 1.00 0.00 0.00 0.00 0.00
0.00 38.00
11:14:16 AM 1 35.64 0.00 25.74 0.00 0.00 4.95 0.00
0.00 33.66
11:14:16 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:17 AM all 48.26 0.00 11.94 0.00 0.00 2.49 0.00
0.00 37.31
11:14:17 AM 0 53.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 47.00
11:14:17 AM 1 43.56 0.00 23.76 0.00 0.00 4.95 0.00
0.00 27.72
11:14:17 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:18 AM all 47.76 0.00 2.99 0.00 0.00 2.49 0.00
0.00 46.77
11:14:18 AM 0 19.00 0.00 1.00 0.00 0.00 0.00 0.00
0.00 80.00
11:14:18 AM 1 76.24 0.00 4.95 0.00 0.00 4.95 0.00
0.00 13.86
11:14:18 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:19 AM all 48.00 0.00 0.50 1.00 0.00 1.50 0.00
0.00 49.00
11:14:19 AM 0 0.00 0.00 0.00 2.00 0.00 0.00 0.00
0.00 98.00
11:14:19 AM 1 96.00 0.00 1.00 0.00 0.00 3.00 0.00
0.00 0.00
11:14:19 AM CPU %usr %nice %sys %iowait %irq %soft %steal
%guest %idle
11:14:20 AM all 47.50 0.00 0.00 0.00 0.00 2.50 0.00
0.00 50.00
11:14:20 AM 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 100.00
11:14:20 AM 1 95.00 0.00 0.00 0.00 0.00 5.00 0.00
0.00 0.00
root@ilrsh01:~#
-----
> > @@ -128,28 +288,35 @@ static int ram_save_block(QEMUFile *f)
> > current_addr +
> TARGET_PAGE_SIZE,
> > MIGRATION_DIRTY_FLAG);
> >
> > - p = block->host + offset;
> > + if (arch_mig_state.use_xbrle) {
> > + p = qemu_mallocz(TARGET_PAGE_SIZE);
>
> qemu_malloc()
corrected to qemu_malloc() (see patch v4)
>
> > +static uint8_t count_hash_bits(uint64_t v)
> > +{
> > + uint8_t bits = 0;
> > +
> > + while (!(v & 1)) {
> > + v = v >> 1;
> > + bits++;
> > + }
> > + return bits;
> > +}
>
> See ffs(3). ffsll() does what you need.
using ctz64() (see patch v4)
>
> > +static uint8_t xor_buf[TARGET_PAGE_SIZE];
> > +static uint8_t xbzrle_buf[TARGET_PAGE_SIZE * 2];
>
> Do these need to be static globals? It should be fine to
> define them as
> local variables inside the functions that need them, there is enough
> stack space.
placed on stack (see patch v4)
>
> > +
> > +int xbzrle_encode(uint8_t *xbzrle, const uint8_t *old,
> const uint8_t *curr,
> > + const size_t max_compressed_len)
> > +{
> > + int compressed_len;
> > +
> > + xor_encode_word(xor_buf, old, curr);
> > + compressed_len = rle_encode((uint64_t *)xor_buf,
> > + sizeof(xor_buf)/sizeof(uint64_t), xbzrle_buf,
> > + sizeof(xbzrle_buf));
> > + if (compressed_len > max_compressed_len) {
> > + return -1;
> > + }
> > + memcpy(xbzrle, xbzrle_buf, compressed_len);
>
> Why the intermediate xbrzle_buf buffer and why the memcpy()?
xbzrle encoding may take up to 150% in a rare worst case scenario - to avoid
having to check during each xbzrle iteration or alternatively adding a loop
that checks for overflow potential during the xbzrle encoding I use the
xbzrle_buf as working area. memcpy() is a factor faster than xbzrle so it's
slow-down is in-significant.
>
> return rle_encode((uint64_t *)xor_buf, sizeof(xor_buf) /
> sizeof(uint64_t),
> xbzrle, max_compressed_len);
>
> Stefan
>
