Re: [Qemu-devel] [PATCH 1/2] migration: Implement a multiple compress threads feature to accelerate live migration

2014-11-02 Thread Li, Liang Z
This patch is a RFC version. -Original Message- From: Li, Liang Z Sent: Friday, October 31, 2014 1:40 PM To: qemu-devel@nongnu.org Cc: lcapitul...@redhat.com; ebl...@redhat.com; arm...@redhat.com; Li, Liang Z Subject: [PATCH 1/2] migration: Implement a multiple compress threads feature

Re: [Qemu-devel] [PATCH 2/2] docs: Add a document about multiple compression threads

2014-11-02 Thread Li, Liang Z
This is a RFC version. -Original Message- From: Li, Liang Z Sent: Friday, October 31, 2014 1:40 PM To: qemu-devel@nongnu.org Cc: lcapitul...@redhat.com; ebl...@redhat.com; arm...@redhat.com; Li, Liang Z Subject: [PATCH 2/2] docs: Add a document about multiple compression threads From

Re: [Qemu-devel] [v5 09/12] migration: Make compression co-work with xbzrle

2015-02-11 Thread Li, Liang Z
> Liang Li wrote: > > Now, multiple thread compression can co-work with xbzrle. when xbzrle > > is on, multiple thread compression will only work at the first round > > of RAM data sync. > > > > Signed-off-by: Liang Li > > Signed-off-by: Yang Zhang > > Reviewed-by: Dr.David Alan Gilbert > > Dr

Re: [Qemu-devel] [v5 05/12] arch_init: Alloc and free data struct for compression

2015-02-11 Thread Li, Liang Z
Hi Juan, Have you reviewed the 04 patch of the patch series? I didn't see the reply email. Liang > -Original Message- > From: Juan Quintela [mailto:quint...@redhat.com] > Sent: Wednesday, February 11, 2015 5:03 PM > To: Li, Liang Z > Cc: qemu-devel@nongnu.or

Re: [Qemu-devel] [v5 02/12] migration: Add the framework of multi-thread compression

2015-02-11 Thread Li, Liang Z
> Reviewing patch 8, I found that we need to fix some things here. > > > +static int ram_save_compressed_page(QEMUFile *f, RAMBlock *block, > > +ram_addr_t offset, bool > > +last_stage) { > > +int bytes_sent = -1; > > + > > +/* To be done*/ > > + > > +

Re: [Qemu-devel] [v5 08/12] migration: Add the core code of multi-thread compression

2015-02-11 Thread Li, Liang Z
> -Original Message- > From: Juan Quintela [mailto:quint...@redhat.com] > Sent: Wednesday, February 11, 2015 7:45 PM > To: Li, Liang Z > Cc: qemu-devel@nongnu.org; ebl...@redhat.com; amit.s...@redhat.com; > lcapitul...@redhat.com; arm...@redhat.com; dgilb...@redhat.co

Re: [Qemu-devel] [v5 09/12] migration: Make compression co-work with xbzrle

2015-02-12 Thread Li, Liang Z
> >> Drop this patch and just give an error when trying to set xbzrle and > >> compression? User have to pick one and only one, no second guess > him/her? > >> > > > > Live migration can benefit from compression co-work with xbzrle. You > > know, xbzrle transfer the raw RAM pages to destination in

Re: [Qemu-devel] [PATCH 0/6] migration: differentiate between pages and bytes

2015-02-12 Thread Li, Liang Z
> (Li special edition) > > Current migration code returns number of bytes transferred and from there > we decide if we.have sent something or not. Problem, we need two results: > number of pages written, and number of bytes written (depending on > compression, zero pages, etc, it is not possible

Re: [Qemu-devel] [PATCH 0/6] migration: differentiate between pages and bytes

2015-02-25 Thread Li, Liang Z
> -Original Message- > From: Dr. David Alan Gilbert [mailto:dgilb...@redhat.com] > Sent: Tuesday, February 17, 2015 10:24 PM > To: Juan Quintela > Cc: qemu-devel@nongnu.org; Li, Liang Z > Subject: Re: [Qemu-devel] [PATCH 0/6] migration: differentiate between > pages

Re: [Qemu-devel] [PATCH v3 0/13] migration: Add a new feature to do live migration

2015-01-24 Thread Li, Liang Z
Thanks Dave & Eric for spending time to review my patches and giving the valuable comments, I will refine my patches in the later version according to your suggestions. > * Liang Li (liang.z...@intel.com) wrote: > > This feature can help to reduce the data transferred about 60%, and > > the migr

Re: [Qemu-devel] [v3 04/13] qemu-file: Add tow function will be used in migration

2015-01-24 Thread Li, Liang Z
> > +size_t migrate_qemu_add_compression_data(QEMUFile *f, > > +const uint8_t *p, size_t size, int level) > > It's an odd name, QEMUFile is only used by migration anyway; maybe > qemufile_add_compression_data ? > > > +{ > > +size_t blen = IO_BUF_SIZE - f->buf_index - sizeof(int); > >

Re: [Qemu-devel] [v3 05/13] arch_init: alloc and free data struct in multi-thread compression

2015-01-24 Thread Li, Liang Z
> > typedef struct compress_param compress_param; > > > > +enum { > > +DONE, > > +START, > > +}; > > + > > Do you really need any more than a 'bool busy' ? Good ideal. > > struct decompress_param { > > /* To be done */ > > }; > > typedef struct decompress_param decompress_param;

Re: [Qemu-devel] [v3 08/13] migration: Add the core code of multi-thread compresion

2015-01-24 Thread Li, Liang Z
> > - > > +/* When starting the process of a new block, the first page of > > + * the block should be sent out before other pages in the same > > + * block, and all the pages in last block should have been sent > > + * out, keeping this order is important. > > Why? Is this just be

Re: [Qemu-devel] [v3 12/13] migration: Add command to set migration parameter

2015-01-24 Thread Li, Liang Z
> * Liang Li (liang.z...@intel.com) wrote: > > Add the qmp and hmp commands to tune the parameters used in live > > migration. > > If I understand correctly on the destination side we need to set the number > of decompression threads very early on an incoming migration - I'm not clear > how earl

Re: [Qemu-devel] [v4 12/13] migration: Add command to set migration parameter

2015-02-03 Thread Li, Liang Z
> > +++ b/migration/migration.c > > @@ -66,9 +66,12 @@ MigrationState *migrate_get_current(void) > > .bandwidth_limit = MAX_THROTTLE, > > .xbzrle_cache_size = DEFAULT_MIGRATE_CACHE_SIZE, > > .mbps = -1, > > -.compress_thread_count = > DEFAULT_MIGRATE_COMPRESS_THR

Re: [Qemu-devel] [PATCH] migration: flush the bdrv before stopping VM

2015-03-17 Thread Li, Liang Z
> This needs further review/changes on the block layer. > > First explanation, why I think this don't fix the full problem. > Whith this patch, we fix the problem where we have a dirty block layer but > basically nothing dirtying the memory on the guest (we are moving the 20 > seconds from max_dow

Re: [Qemu-devel] [PATCH] migration: flush the bdrv before stopping VM

2015-03-18 Thread Li, Liang Z
> > > First explanation, why I think this don't fix the full problem. > > > Whith this patch, we fix the problem where we have a dirty block > > > layer but basically nothing dirtying the memory on the guest (we are > > > moving the 20 seconds from max_downtime for the blocklayer flush), > > > to 2

Re: [Qemu-devel] [v5 12/12] migration: Add commands to set and query parameter

2015-03-18 Thread Li, Liang Z
> > +# > > +# Migration parameter information > > +# > > +# @compress-level: compression level > > +# > > +# @compress-threads: compression thread count # # > > +@decompress-threads: decompression thread count # # Since: 2.3 ## { > > +'union': 'MigrationParameterStatus', > > + 'base': 'MigrationPa

Re: [Qemu-devel] [PATCH] migration: flush the bdrv before stopping VM

2015-03-19 Thread Li, Liang Z
> * Li, Liang Z (liang.z...@intel.com) wrote: > > > > > First explanation, why I think this don't fix the full problem. > > > > > Whith this patch, we fix the problem where we have a dirty block > > > > > layer but basically nothing dirtying th

Re: [Qemu-devel] [v5 12/12] migration: Add commands to set and query parameter

2015-03-19 Thread Li, Liang Z
> >> The command takes a list of key-value pairs. Looks like this > >> (example stolen from your patch to qmp-commands.hx): > >> > >> { "execute": "migrate-set-parameters", > >> "arguments": { "parameters": > >> [ { "parameter": "compress-level", "value": 1 } > >> ]

Re: [Qemu-devel] [v5 12/12] migration: Add commands to set and query parameter

2015-03-19 Thread Li, Liang Z
> > >> The command takes a list of key-value pairs. Looks like this > > >> (example stolen from your patch to qmp-commands.hx): > > >> > > >> { "execute": "migrate-set-parameters", > > >> "arguments": { "parameters": > > >> [ { "parameter": "compress-level", "value":

Re: [Qemu-devel] [PATCH] migration: flush the bdrv before stopping VM

2015-03-20 Thread Li, Liang Z
> >> > Right now, we don't have an interface to detect that cases and got > >> > back to the iterative stage. > >> > >> How about go back to the iterative stage when detect that the > >> pending_size is larger Than max_size, like this: > >> > >> +/* do flush here is aimed to shorten

Re: [Qemu-devel] [PATCH] migration: flush the bdrv before stopping VM

2015-03-25 Thread Li, Liang Z
> > > Then, how to deal with this issue in 2.3, leave it here? or make an > > > incomplete fix like I do above? > > > > I think it is better to leave it here for 2.3. With a patch like this > > one, we improve in one load and we got worse in a different load > > (depens a lot in the ratio of dirtyi

Re: [Qemu-devel] [v6 08/14] migration: Add the core code of multi-thread compression

2015-03-25 Thread Li, Liang Z
> > --- a/arch_init.c > > +++ b/arch_init.c > > @@ -355,12 +355,33 @@ static DecompressParam *decomp_param; static > > QemuThread *decompress_threads; static uint8_t > *compressed_data_buf; > > > > +static int do_compress_ram_page(CompressParam *param); > > + > > static void *do_data_compress(vo

Re: [Qemu-devel] [v6 10/14] migration: Add the core code for decompression

2015-03-25 Thread Li, Liang Z
> > +qemu_mutex_lock(¶m->mutex); > > +while (!param->start && !quit_decomp_thread) { > > start protected by param->mutex. > > > > +qemu_cond_wait(¶m->cond, ¶m->mutex); > > +pagesize = TARGET_PAGE_SIZE; > > +if (!quit_decomp_thread) { > > +

Re: [Qemu-devel] [v6 08/14] migration: Add the core code of multi-thread compression

2015-03-26 Thread Li, Liang Z
> >> > --- a/arch_init.c > >> > +++ b/arch_init.c > >> > @@ -355,12 +355,33 @@ static DecompressParam *decomp_param; > static > >> > QemuThread *decompress_threads; static uint8_t > >> *compressed_data_buf; > >> > > >> > +static int do_compress_ram_page(CompressParam *param); > >> > + > >> > stat

Re: [Qemu-devel] [v6 08/14] migration: Add the core code of multi-thread compression

2015-03-27 Thread Li, Liang Z
> - > > 1 file changed, 177 insertions(+), 7 deletions(-) > > > > diff --git a/arch_init.c b/arch_init.c index 48cae22..9f63c0f 100644 > > --- a/arch_init.c > > +++ b/arch_init.c > > @@ -355,12 +355,33 @@ static DecompressParam *decomp_param; static > > QemuThread *decompress_threads; static uin

Re: [Qemu-devel] [PATCH v6 0/14] migration: Add a new feature to do live migration

2015-04-01 Thread Li, Liang Z
Hi guys, Any more comments about this patch serials? Especially the last three patches about the qmp and hmp interfaces. Markus & Eric, could you help to take a look? Sorry for missing Markus in the CC lists. Liang > -Original Message- > From: Li, Lian

Re: [Qemu-devel] [PATCH v3 0/13] migration: Add a new feature to do live migration

2014-12-23 Thread Li, Liang Z
> -Original Message- > From: Li, Liang Z > Sent: Friday, December 12, 2014 9:29 AM > To: qemu-devel@nongnu.org > Cc: quint...@redhat.com; lcapitul...@redhat.com; ebl...@redhat.com; > arm...@redhat.com; Zhang, Yang Z; dgilb...@redhat.com; Li, Liang Z > Subject: [PAT

[Qemu-devel] about the re-attach more than one pci devices failed

2014-12-30 Thread Li, Liang Z
Hi Paolo, We have found a bug in all the xen-4.4 and xen-4.5-rcx, the bug can be reproduced by the following steps: Use the 'xl pci-attach $DomU $BDF' command to attach more then one PCI devices to the guest, then detach the devices with 'xl pci-detach $DomU $BDF', after that, re-attach these PC

Re: [Qemu-devel] [PATCH v3 0/13] migration: Add a new feature to do live migration

2015-01-06 Thread Li, Liang Z
Hi All, Have you returned from vacation? If so, could you help to review my patches I have submitted three weeks ago? Thanks a lot. Liang > -Original Message- > From: Li, Liang Z > Sent: Friday, December 12, 2014 9:29 AM > To: qemu-devel@nongnu.org > Cc: quin

Re: [Qemu-devel] [Xen-devel] [PATCH] xen-pt: Fix PCI devices re-attach failed

2015-01-12 Thread Li, Liang Z
> > > Use the 'xl pci-attach $DomU $BDF' command to attach more than one > > > PCI devices to the guest, then detach the devices with 'xl > > > pci-detach $DomU $BDF', after that, re-attach these PCI devices > > > again, an error message will be reported like following: > > > > > > libxl: error: li

Re: [Qemu-devel] [v2 1/2] docs: Add a doc about multiple compression threads

2014-11-06 Thread Li, Liang Z
>OK, some high level questions: > > 1) How does the performance compare to running a separate compressor process > in the stream rather than embedding it in the qemu? > I have not do the test, so I don't know the performance. Maybe I can do it later. > 2) Since you're looking at different comp

Re: [Qemu-devel] [v2 2/2] migration: Implement multiple compression threads

2014-11-20 Thread Li, Liang Z
> > +static void migrate_put_be32(MigBuf *f, unsigned int v) > > +{ > > +migrate_put_byte(f, v >> 24); > > +migrate_put_byte(f, v >> 16); > > +migrate_put_byte(f, v >> 8); > > +migrate_put_byte(f, v); > > +} > > + > > +static void migrate_put_be64(MigBuf *f, uint64_t v) > > +{ > > +

Re: [Qemu-devel] [v2 2/2] migration: Implement multiple compression threads

2014-11-20 Thread Li, Liang Z
> > +int migrate_compress_threads(void) > > +{ > > +MigrationState *s; > > + > > +s = migrate_get_current(); > > + > > +return s->compress_thread_count; > > +} > > + > > int migrate_use_xbzrle(void) > > { > > MigrationState *s; > > @@ -697,4 +795,5 @@ void migrate_fd_connect(Migr

Re: [Qemu-devel] [v2 2/2] migration: Implement multiple compression threads

2014-11-21 Thread Li, Liang Z
> hmm, multiple thread compression can't co-work with xbzrle. xbzrle need > guarantee the cache at src is same to dest. But I dont see that below: > > +/* XBZRLE overflow or normal page */ > +if (bytes_sent == -1) { > +bytes_sent = migrate_save_block_hdr(¶m->migbuf, block, > +

Re: [Qemu-devel] [v2 2/2] migration: Implement multiple compression threads

2014-11-23 Thread Li, Liang Z
> > # @auto-converge: If enabled, QEMU will automatically throttle down the > > guest > > # to speed up convergence of RAM migration. (since 1.6) > > # > > # Since: 1.2 > > ## > > { 'enum': 'MigrationCapability', > > - 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-block

Re: [Qemu-devel] [v2 2/2] migration: Implement multiple compression threads

2014-12-07 Thread Li, Liang Z
>> # >> # Since: 1.2 >> ## >> { 'enum': 'MigrationCapability', >> - 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks'] >> } >> + 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks', >>+ 'compress'] } >> > >I'll repeat what I said on v1 (but this time, with som

Re: [Qemu-devel] [v2 2/2] migration: Implement multiple compression threads

2014-12-10 Thread Li, Liang Z
>>> ## >>> { 'enum': 'MigrationCapability', >>> - 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks'] >>> } >>> + 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks', >>>+ 'compress'] } >>> >> >>I'll repeat what I said on v1 (but this time, with some links to back

Re: [Qemu-devel] [v7 09/14] migration: Make compression co-work with xbzrle

2015-04-08 Thread Li, Liang Z
> > Now, multiple thread compression can co-work with xbzrle. when xbzrle > > is on, multiple thread compression will only work at the first round > > of RAM data sync. > > > > Signed-off-by: Liang Li > > Signed-off-by: Yang Zhang > > Reviewed-by: Dr.David Alan Gilbert > > Reviewed-by: Juan Qui

Re: [Qemu-devel] [v7 10/14] migration: Add the core code for decompression

2015-04-08 Thread Li, Liang Z
> > @@ -889,7 +889,6 @@ static inline void > start_compression(CompressParam *param) > > qemu_mutex_unlock(¶m->mutex); } > > > > - > > static uint64_t bytes_transferred; > > > > static void flush_compressed_data(QEMUFile *f) @@ -1458,8 +1457,28 > @@ > > void ram_handle_compressed(void *host

Re: [Qemu-devel] [v7 13/14] migration: Add qmp commands to set and query parameters

2015-04-13 Thread Li, Liang Z
> Eric, can you review this and the following patch? I think they are correct, > but > I preffer someone more versed on QMP to review them. > > Thanks, Juan. Hi Juan & Eric, Since the latest QEMU is 2.3.0-rc2, is it possible to merge this serial of patches to QEMU 2.3? If it can't not,

Re: [Qemu-devel] [v7 12/14] migration: Use an array instead of 3 parameters

2015-04-15 Thread Li, Liang Z
> Eric Blake wrote: > > On 04/08/2015 12:20 AM, Liang Li wrote: > >> Put the three parameters related to multiple thread (de)compression > >> into an int array, and use an enum type to index the parameter. > >> > >> Signed-off-by: Liang Li > >> Signed-off-by: Yang Zhang > >> --- > >> include/mi

Re: [Qemu-devel] [PATCH] xen-pt: Fix bug cause PCI devices re-attach failed

2015-04-15 Thread Li, Liang Z
> On 13/04/2015 16:12, Liang Li wrote: > > 2. Do the attach and detach operation with a time interval. eg. 10s. > > > > The error message will not disappear if retry, in this case, it's > > a bug. > > > > In the 'xen_pt_region_add' and 'xen_pt_region_del', we should only > > care about

Re: [Qemu-devel] [PATCH v6 00/47] Postcopy implementation

2015-04-27 Thread Li, Liang Z
Hi David, I have tired your v6 postcopy patches and found it doesn't work. When I tried to start the postcopy in live migration, some errors were printed. I just did the following things: On destination side, started the qemu like this: /root/vt-sync/post_copy_v6_qemu.git/x86_64-softmmu/qemu-

Re: [Qemu-devel] [PATCH v6 00/47] Postcopy implementation

2015-04-29 Thread Li, Liang Z
> * Li, Liang Z (liang.z...@intel.com) wrote: > > Hi David, > > > > I have tired your v6 postcopy patches and found it doesn't work. When > > I tried to start the postcopy in live migration, some errors were printed. I > just did the following things: > >

Re: [Qemu-devel] [v5 08/12] migration: Add the core code of multi-thread compression

2015-03-01 Thread Li, Liang Z
> I thihnk this would make the code work, but not the locking. You are using > here: > > quit_comp_thread: global, and not completely clear what protects it > comp_done_lock: global > comp_done_cond: global > > param[i].busy: I would suggest renaming to pending work > param[i].mutex: > param[i]

Re: [Qemu-devel] [PATCH 0/6] migration: differentiate between pages and bytes

2015-03-08 Thread Li, Liang Z
> Current migration code returns number of bytes transferred and from there > we decide if we.have sent something or not. Problem, we need two results: > number of pages written, and number of bytes written (depending on > compression, zero pages, etc, it is not possible to derive one value from t

Re: [Qemu-devel] [PULL 07/11] save_block_hdr: we can recalculate the cont parameter here

2015-03-16 Thread Li, Liang Z
Hi Juan This patch will make my work more difficult, as we discussed before, after modification, I have to use a lock before I can reuse the function save_block_hdr in my compression threads, this will lead to low efficiency. Could help to revert this patch? Liang > -Original Message

Re: [Qemu-devel] [v2 0/4] Fix long vm downtime during live migration

2015-10-09 Thread Li, Liang Z
> > Some cleanup operations take long time during the pause and copy > > stage, especially with the KVM patch 3ea3b7fa9af067, do these > > operations after the completion of live migration can help to reduce VM > downtime. > > > > Ony the first patch changes the behavior, the rest 3 patches are for

Re: [Qemu-devel] [RFC 0/2] Reduce the VM downtime about 300us

2015-10-28 Thread Li, Liang Z
Hi mg, Have you tested this patch? Can it fix the kvm-clock issue? Liang > -Original Message- > From: Marcin Gibuła [mailto:m.gib...@beyond.pl] > Sent: Wednesday, August 26, 2015 3:26 AM > To: Li, Liang Z; qemu-devel@nongnu.org > Cc: pbonz...@redhat.com; mtosa...@redha

Re: [Qemu-devel] [RESEND 1/2] kvmclock: use a light weight interface to update env->tsc.

2015-11-02 Thread Li, Liang Z
> This can be simplified a bit: > > int kvm_get_tsc(CPUState *cs) > { > X86CPU *cpu = X86_CPU(cs); > CPUX86State *env = &cpu->env; > struct { > struct kvm_msrs info; > struct kvm_msr_entry entries[1]; > } msr_data; > int ret; > > if (env->tsc_valid) { >

Re: [Qemu-devel] assert during internal snapshot

2015-11-08 Thread Li, Liang Z
> Hello, All! > > This commit > > commit 94f5a43704129ca4995aa3385303c5ae225bde42 > Author: Liang Li > Date: Mon Nov 2 15:37:00 2015 +0800 > > migration: defer migration_end & blk_mig_cleanup > > Because of the patch 3ea3b7fa9af067982f34b of kvm, which introduces a > lazy coll

Re: [Qemu-devel] assert during internal snapshot

2015-11-08 Thread Li, Liang Z
> migration: defer migration_end & blk_mig_cleanup > > Because of the patch 3ea3b7fa9af067982f34b of kvm, which introduces a > lazy collapsing of small sptes into large sptes mechanism, now > migration_end() is a time consuming operation because it calls > memroy_global_di

Re: [Qemu-devel] assert during internal snapshot

2015-11-08 Thread Li, Liang Z
> -Original Message- > From: Denis V. Lunev [mailto:d...@openvz.org] > Sent: Saturday, November 07, 2015 11:20 PM > To: Li, Liang Z; Paolo Bonzini; Juan Quintela; Amit Shah > Cc: QEMU > Subject: assert during internal snapshot > > Hello, All! >

Re: [Qemu-devel] [PATCH 1/1] migration: qemu_savevm_state_cleanup becomes mandatory operation

2015-11-08 Thread Li, Liang Z
> since commit > commit 94f5a43704129ca4995aa3385303c5ae225bde42 > Author: Liang Li > Date: Mon Nov 2 15:37:00 2015 +0800 > > migration: defer migration_end & blk_mig_cleanup > > when actual .cleanup callbacks calling was removed from complete operations. > > Signed-off-by: De

Re: [Qemu-devel] [PATCH 1/1] migration: qemu_savevm_state_cleanup becomes mandatory operation

2015-11-08 Thread Li, Liang Z
> On 11/09/2015 08:10 AM, Li, Liang Z wrote: > >> since commit > >> commit 94f5a43704129ca4995aa3385303c5ae225bde42 > >> Author: Liang Li > >> Date: Mon Nov 2 15:37:00 2015 +0800 > >> > >> migration: defer migra

Re: [Qemu-devel] [PATCH 1/1] migration: qemu_savevm_state_cleanup becomes mandatory operation

2015-11-09 Thread Li, Liang Z
> >>> Yes, you are right. Thanks a lot. > >>> > >>> BTW, can this patch fix the regression you reported? > >>> > >>> Reviewed-by: Liang Li > >>> > >> yes > > Great. You'd better change the commit message to make it more clear. > > > > Liang > argh.. you are right... > > This problem has appeared

Re: [Qemu-devel] [v2 0/2] add avx2 instruction optimization

2015-11-09 Thread Li, Liang Z
> Rather than trying to cater to multiple assembly instruction implementations > ourselves, have you tried taking the ideas in this earlier thread? > https://lists.gnu.org/archive/html/qemu-devel/2015-10/msg05298.html > > Ideally, libc's memcmp() will already be using the most efficient assembly >

Re: [Qemu-devel] [v2 0/2] add avx2 instruction optimization

2015-11-10 Thread Li, Liang Z
> > Eric, thanks for you information. I didn't notice that discussion before. > > > > > > I rewrite the buffer_find_nonzero_offset() with the 'bool memeqzero4_paolo > length' > > then write a test program to check a large amount of zero pages, and > > use the 'time' to recode the time takes by diff

Re: [Qemu-devel] [v2 0/2] add avx2 instruction optimization

2015-11-10 Thread Li, Liang Z
> On 10/11/2015 10:26, Li, Liang Z wrote: > > I don't know Paolo's opinion about how to deal with the SSE2 > > Intrinsics, he is the author. From my personal view, now that we have > > found a better way, why to use such low level SSE2/AVX2 Intrinsics. > > I t

Re: [Qemu-devel] [v2 0/2] add avx2 instruction optimization

2015-11-10 Thread Li, Liang Z
> On 10/11/2015 10:41, Li, Liang Z wrote: > >> On 10/11/2015 10:26, Li, Liang Z wrote: > >>> I don't know Paolo's opinion about how to deal with the SSE2 > >>> Intrinsics, he is the author. From my personal view, now that we > >>> ha

Re: [Qemu-devel] [v2 0/2] add avx2 instruction optimization

2015-11-10 Thread Li, Liang Z
> On 10/11/2015 10:56, Li, Liang Z wrote: > > > I agree that your patch can be dropped, but go ahead and submit your > > > improvements! > > > > You mean I do this work? > > If you are busy, I can do this. > > It's not that I'm busy, it'

Re: [Qemu-devel] [v2 0/2] add avx2 instruction optimization

2015-11-11 Thread Li, Liang Z
> > On 10/11/2015 10:26, Li, Liang Z wrote: > > I don't know Paolo's opinion about how to deal with the SSE2 > > Intrinsics, he is the author. From my personal view, now that we have > > found a better way, why to use such low level SSE2/AVX2 Intrinsics. >

Re: [Qemu-devel] [v2 0/2] add avx2 instruction optimization

2015-11-12 Thread Li, Liang Z
> On 12/11/2015 03:49, Li, Liang Z wrote: > > I am very surprised about the live migration performance result when > > I use your ' memeqzero4_paolo' instead of these SSE2 Intrinsics to > > check the zero pages. > > What code were you using? Remember

Re: [Qemu-devel] [v2 0/2] add avx2 instruction optimization

2015-11-12 Thread Li, Liang Z
> >>> I am very surprised about the live migration performance result > >>> when I use your ' memeqzero4_paolo' instead of these SSE2 Intrinsics > >>> to check the zero pages. > >> > >> What code were you using? Remember I suggested using only unsigned > >> long checks, like > >> > >>unsigned

Re: [Qemu-devel] [v2 0/2] add avx2 instruction optimization

2015-11-12 Thread Li, Liang Z
> On 12/11/2015 10:40, Li, Liang Z wrote: > > I migrate a 8GB RAM Idle guest, I think most of it's pages are zero pages. > > > > I use your new code: > > - > > unsigned long *p = ... > > if (p[0] |

Re: [Qemu-devel] [v2 1/2] cutils: add avx2 instruction optimization

2015-11-12 Thread Li, Liang Z
> > The main issue here is that you are not testing whether the compiler supports > gnu_indirect_function. > > I suggest that you start by moving the functions to util/buffer-zero.c > > Then the structure should be something like > > #ifdef CONFIG_HAVE_AVX2 > #include > #endif > > ... define

Re: [Qemu-devel] [v2 0/2] add avx2 instruction optimization

2015-11-12 Thread Li, Liang Z
> >> > > >> > I use your new code: > >> > - > >> > unsigned long *p = ... > >> > if (p[0] || p[1] || p[2] || p[3] > >> > || memcmp(p+4, p, size - 4 * sizeof(unsigned long)) != 0) > >> > return BUFFER_NOT_ZERO; > >> > else > >> >

Re: [Qemu-devel] [v2 1/2] cutils: add avx2 instruction optimization

2015-11-12 Thread Li, Liang Z
> > This patch use the ifunc mechanism to select the proper function when > > running, for platform supports AVX2, excute the AVX2 instructions, > > else, excute the original code. > > > > Signed-off-by: Liang Li > > --- > > include/qemu-common.h | 28 +++-- > > util/Makefile.objs

Re: [Qemu-devel] [TCG only][Migration Bug? ] Occasionally, the content of VM's memory is inconsistent between Source and Destination of migration

2015-12-03 Thread Li, Liang Z
> > Thanks for describing how to reproduce the bug. > If some pages are not transferred to destination then it is a bug, so we need > to know what the problem is, notice that the problem can be that TCG is not > marking dirty some page, that Migration code "forgets" about that page, or > anything

Re: [Qemu-devel] [PATCH 0/2] fix the flaws of qemu_put_compression_data

2015-12-04 Thread Li, Liang Z
> Hi > > We are in hard freeze. My understanding is that this are "optimizations" that > can wait for 2.6: > - my understanding from commit from message one and from quick look at > the code is that this change is not needed for current users, is that > correct? > - we avoid a copy at the begi

Re: [Qemu-devel] [PATCH 1/2] qemu-file: fix flaws of qemu_put_compression_data

2015-12-04 Thread Li, Liang Z
> > There are some flaws in qemu_put_compression_data, this patch tries to > > fix it. Now it can be used by other code. > > > > Signed-off-by: Liang Li > > --- > > migration/qemu-file.c | 10 +- > > 1 file changed, 9 insertions(+), 1 deletion(-) > > > > diff --git a/migration/qemu-file.c

Re: [Qemu-devel] [PATCH 1/2] qemu-file: fix flaws of qemu_put_compression_data

2015-12-04 Thread Li, Liang Z
> On (Fri) 04 Dec 2015 [11:53:09], Liang Li wrote: > > There are some flaws in qemu_put_compression_data, this patch tries to > > fix it. Now it can be used by other code. > > Can you please write a better description here? What are the flaws? > What is being fixed? What other users, and how is

Re: [Qemu-devel] [PATCH 1/2] qemu-file: fix flaws of qemu_put_compression_data

2015-12-06 Thread Li, Liang Z
> >> - blen could still be smaller that compressBound(size), you need to > >> recheck > >> - blen could have changed, but you don't take that in account for the > >> following caller. > >> > >> So, I think code has a bug? > > > > Yes, there is a bug, I should consider the case QEMUFile with emp

Re: [Qemu-devel] [PATCH] xen_pt: fix failure of attaching & detaching a PCI device to VM repeatedly

2015-12-06 Thread Li, Liang Z
> > Add pci = [ '$VF_BDF', '$VF_BDF', '$VF_BDF'] in > > This is a bit confusing: it is not actually correct to assign the same > device, even > an SR_IOV VF, multiple times, so these must be all different. More like: > > pci = [ '$VF_BDF1', '$VF_BDF2', '$VF_BDF3'] > > > > hvm guest configurati

Re: [Qemu-devel] [PATCH 1/2] qemu-file: fix flaws of qemu_put_compression_data

2015-12-06 Thread Li, Liang Z
> On Fri, Dec 04, 2015 at 11:52:07AM +0800, Liang Li wrote: > > There are some flaws in qemu_put_compression_data, this patch tries to > > fix it. Now it can be used by other code. > > > > Signed-off-by: Liang Li > > --- > > migration/qemu-file.c | 10 +- > > 1 file changed, 9 insertions(

Re: [Qemu-devel] [v3 3/3] configure: add options to config avx2

2015-12-08 Thread Li, Liang Z
> On 8 December 2015 at 12:08, Liang Li wrote: > > Add the '--enable-avx2' & '--disable-avx2' option so as to config the > > AVX2 instruction optimization. > > > > If '--disable-avx2' is not set, configure will detect if the compiler > > can support AVX2 option, if yes, AVX2 optimization is eabled

Re: [Qemu-devel] [v3 1/3] cutils: add avx2 instruction optimization

2015-12-09 Thread Li, Liang Z
> On 12/08/2015 04:08 AM, Liang Li wrote: > > +++ b/util/buffer-zero-avx2.c > > @@ -0,0 +1,54 @@ > > +#include "qemu-common.h" > > + > > +#if defined CONFIG_IFUNC && defined CONFIG_AVX2 #include > > + > > +#define AVX2_VECTYPE__m256i > > +#define AVX2_SPLAT(p) _mm256_set1_epi8(*(p)) >

Re: [Qemu-devel] [v3 1/3] cutils: add avx2 instruction optimization

2015-12-09 Thread Li, Liang Z
> On 12/09/2015 01:32 AM, Li, Liang Z wrote: > > I think you means the ' __attribute__((target("avx2")))', I have tried this > way, the issue here is: > > without the ' -mavx2' option for gcc, there are compiling error: > > '__m256i u

Re: [Qemu-devel] [v3 1/3] cutils: add avx2 instruction optimization

2015-12-10 Thread Li, Liang Z
> >> without the ' -mavx2' option for gcc, there are compiling error: > >> '__m256i undeclared', the __attribute__((target("avx2"))) can't solve > >> this issue. Any idea? > > > > You're right that you can't use the normal __m256i, as it doesn't get > > declared. > > It should be declared. *in

Re: [Qemu-devel] [PATCH 6/6] multithread decompression: Avoid one copy

2015-12-15 Thread Li, Liang Z
> From: "Dr. David Alan Gilbert" > > qemu_get_buffer does a copy, we can avoid the memcpy, and we can then > remove the extra buffer. > > Signed-off-by: Dr. David Alan Gilbert > --- > migration/ram.c | 11 +++ > 1 file changed, 3 insertions(+), 8 deletions(-) > > diff --git a/migratio

[Qemu-devel] about the 'mptable' issue

2016-01-14 Thread Li, Liang Z
Hi Kevin, We just found when starting QEMU with the '-smp 20 -no-acpi' option, the centos6.6 or rhel 7.2 guest failed to boot. By debugging, I found it's your seabios patch, the commit id '9ee2e26255661a' caused the failure. I don't know what issue your patch tried to fix, assuming it's the r

Re: [Qemu-devel] about the 'mptable' issue

2016-01-14 Thread Li, Liang Z
Correct something. The actual parameter for QEMU in our test case is: 'qemu-systerm-x86_64 -enable-kvm -smp 20 -m 2048 -no-acpi -monitor stdio -drive file=/mnt/centos6u6.qcow, if=none, id=foo -device virtio-blk-pci, drive=foo' if there is no virtio-blk device: ' qemu-systerm-x86_64 -enable

Re: [Qemu-devel] about the 'mptable' issue

2016-01-14 Thread Li, Liang Z
> On Thu, Jan 14, 2016 at 10:36:07AM +0000, Li, Liang Z wrote: > > Correct something. > > The actual parameter for QEMU in our test case is: > > 'qemu-systerm-x86_64 -enable-kvm -smp 20 -m 2048 -no-acpi -monitor > > stdio -drive file=/mnt/centos6u6.qcow, if

Re: [Qemu-devel] [PATCH] migration: not send zero page header in ram bulk stage

2016-01-15 Thread Li, Liang Z
> It seems that this patch is incorrect, if the no-zero pages are zeroed again > during !ram_bulk_stage, we didn't send the new zeroed page, there will be > an error. > If not in ram_bulk_stage, still send the header, could you explain why it's wrong? Liang > > For guest just uses a small port

Re: [Qemu-devel] [PATCH] migration: not send zero page header in ram bulk stage

2016-01-16 Thread Li, Liang Z
> On 15/01/2016 10:48, Liang Li wrote: > > Now that VM's RAM pages are initialized to zero, (VM's RAM is allcated > > with the mmap() and MAP_ANONYMOUS option, or mmap() without > MAP_SHARED > > if hugetlbfs is used.) so there is no need to send the zero page > > header to destination. > > > > For

Re: [Qemu-devel] [PATCH] migration: not send zero page header in ram bulk stage

2016-01-16 Thread Li, Liang Z
> * Liang Li (liang.z...@intel.com) wrote: > > Now that VM's RAM pages are initialized to zero, (VM's RAM is allcated > > with the mmap() and MAP_ANONYMOUS option, or mmap() without > MAP_SHARED > > if hugetlbfs is used.) so there is no need to send the zero page > > header to destination. > > > >

Re: [Qemu-devel] [PATCH] migration: not send zero page header in ram bulk stage

2016-01-18 Thread Li, Liang Z
> On 2016/1/15 18:24, Li, Liang Z wrote: > >> It seems that this patch is incorrect, if the no-zero pages are > >> zeroed again during !ram_bulk_stage, we didn't send the new zeroed > >> page, there will be an error. > >> > > > > If not in

Re: [Qemu-devel] [PATCH] migration: not send zero page header in ram bulk stage

2016-01-18 Thread Li, Liang Z
> > Not yet, I saw Dave's comment's, it will beak post copy, it's not hard to > > fix > this. > > A more important thing is Paolo's comments, I don't know in which case > this patch will break LM. Do you have any idea about this? > > Hope that QEMU don't write data to the block 'pc.ram'. > > > >

Re: [Qemu-devel] [PATCH] migration: not send zero page header in ram bulk stage

2016-01-18 Thread Li, Liang Z
> Actually, someone has done like that before and cause a migration bug, See > commit f1c72795af573b24a7da5eb52375c9aba8a37972, and the fixing patch is > commit 9ef051e5536b6368a1076046ec6c4ec4ac12b5c6 > Revert "migration: do not sent zero pages in bulk stage" Thanks for your information, I didn't

Re: [Qemu-devel] [PATCH] migration: not send zero page header in ram bulk stage

2016-01-20 Thread Li, Liang Z
This patch will break LM. > > > > Which portion of the VM's RAM pages will be written by QEMU? Do you > know some exact information? > > I can't wait for Paolo's response. > > It is basically anything that uses rom_add_file_fixed or rom_add_blob_fixed > with an address that points into RAM. > > P

Re: [Qemu-devel] [PATCH v4 1/2] configure: detect ifunc and avx2 attribute

2016-01-20 Thread Li, Liang Z
> On 20/01/2016 10:05, Liang Li wrote: > > Detect if the compiler can support the ifun and avx2, if so, set > > CONFIG_AVX2_OPT which will be used to turn on the avx2 instruction > > optimization. > > > > Signed-off-by: Liang Li > > --- > > configure | 20 > > 1 file changed,

Re: [Qemu-devel] [PATCH v5 0/2] add avx2 instruction optimization

2016-02-21 Thread Li, Liang Z
> On 27/01/2016 08:33, Liang Li wrote: > > buffer_find_nonzero_offset() is a hot function during live migration. > > Now it use SSE2 instructions for optimization. For platform supports > > AVX2 instructions, use the AVX2 instructions for optimization can help > > to improve the performance of zero

Re: [Qemu-devel] [PATCH v5 0/2] add avx2 instruction optimization

2016-02-21 Thread Li, Liang Z
> Not sure; I could take it from the migration tree if no one objects. > > Amit Thanks, Amit. If rework is needed, just let me know. Liang

Re: [Qemu-devel] [PATCH RESEND v2 1/2] qemu-file: Fix qemu_put_compression_data flaw

2016-02-23 Thread Li, Liang Z
Ping ... Liang > -Original Message- > From: Li, Liang Z > Sent: Friday, January 15, 2016 6:06 PM > To: qemu-devel@nongnu.org > Cc: quint...@redhat.com; amit.s...@redhat.com; dgilb...@redhat.com; > zhang.zhanghaili...@huawei.com; Li, Liang Z > Subject: [PATCH RES

Re: [Qemu-devel] [PULL 0/5] migration pull

2016-02-23 Thread Li, Liang Z
> Cc: qemu list; Dr. David Alan Gilbert; Juan Quintela > Subject: Re: [Qemu-devel] [PULL 0/5] migration pull > > On 23 February 2016 at 07:30, Amit Shah wrote: > > The following changes since commit > 8eb779e4223a18db9838a49ece1bc72cfdfb7761: > > > > Merge remote-tracking branch 'remotes/kevin/

Re: [Qemu-devel] [PULL 0/5] migration pull

2016-02-23 Thread Li, Liang Z
> On 23 February 2016 at 09:48, Paolo Bonzini wrote: > > On 23/02/2016 10:09, Peter Maydell wrote: > >> Hi. I'm afraid this doesn't compile for x86-64 Linux: > > > > What compiler is this, and does the following compile with no > > particular extra options? > > > > #pragma GCC target("avx2") > > #

Re: [Qemu-devel] [PATCH RESEND v2 1/2] qemu-file: Fix qemu_put_compression_data flaw

2016-02-23 Thread Li, Liang Z
On %D, %SN wrote: %Q %C Liang > -Original Message- > From: qemu-devel-bounces+liang.z.li=intel@nongnu.org [mailto:qemu- > devel-bounces+liang.z.li=intel@nongnu.org] On Behalf Of Juan Quintela > Sent: Tuesday, February 23, 2016 5:57 PM > To: Li, Liang

Re: [Qemu-devel] [PULL 0/5] migration pull

2016-02-24 Thread Li, Liang Z
> > > On 23/02/2016 12:25, Peter Maydell wrote: > > On 23 February 2016 at 10:43, Peter Maydell > wrote: > >> That code fragment you suggest compiles fine normally, but not if I > >> add -save-temps: > >> > >> $ cat /tmp/zz9.c > >> #pragma GCC target("avx2") > >> #include > >> __m256i foo; > >>

  1   2   3   4   >