Re: [Xen-devel] [PATCH v13 00/26] COarse-grain LOck-stepping Virtual Machines for Non-stop Service

[Wen Congyang]

On 03/30/2016 10:50 PM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v13 00/26] COarse-grain LOck-stepping Virtual 
> Machines for Non-stop Service"):
>> This patchset implemented the COLO feature for Xen.
>> For detail/install/use of COLO feature, refer to:
>> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
>>
>> You can get the codes from here:
>> https://github.com/Pating/xen/tree/changlox/colo_v13
> 
> I fetched the branches `colo_v13' and `colo_v13_fixup' and it seems
> that I can get the proper versions of v13.1 from the latter.  I have
> acked them accordingly.
> 
> Can you confirm that that branch is what you intend for upstream ?
> 
> If so, I have a question about it:
> 
> There are two patches in it which have not been posted as part of your
> series and are marked as "[DO NOT MERGE]".  (Thanks for your admirably
> clear marking, btw.)
> 
> They are
>   [DONT MERGE] don't create default ioreq server
>   [DONT MERGE] tools/libxc: support to resume uncooperative HVM guests
> 
> 
> The latter patch "support to resume uncooperative HVM guests" seems to
> have been posted a number of times and AFAICT most recently as
>   [PATCH v8 05/13] tools/libxc: support to resume uncooperative HVM guests
> on the 18th of February.
> 
> Is that not required for COLO ?  We need to sort this out, I think.

Yes, it is required for COLO.

> 
> 
> What is the status of the default ioreq server patch ?  Why is it in
> your git branch ?

I have reported this bug last year:
http://lists.xenproject.org/archives/html/xen-devel/2015-12/msg02850.html

This patch is just a temporary patch.

Thanks
Wen Congyang

> 
> 
> Thanks,
> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 0/3] COLO: only build on Linux

[Wen Congyang]

On 04/06/2016 04:05 AM, Wei Liu wrote:
> COLO depends on netlink which is only available on Linux. This series cleans 
> up
> COLO code and make it only build on Linux. This should fix FreeBSD build.
> 
> Congyang and Changlong, please review this series as soon as possible. I also
> have a question why COLO doesn't use libnl? It is using Linux header directly.

It is implemented by Yang one year ago. I don't know the reason. We will 
investigate
how to use libnl.

Thanks
Wen Congyang

> 
> Roger, can you run functional tests on FreeBSD? I don't have FreeBSD Dom0.
> 
> Wei.
> 
> Wei Liu (3):
>   libxl: colo: rearrange things in header files
>   libxl: colo: move netlink related stuff to libxl_colo_proxy.c
>   libxl: colo: only build COLO on Linux
> 
>  tools/libxl/Makefile   |  5 
>  tools/libxl/libxl_colo.h   | 52 +--
>  tools/libxl/libxl_colo_proxy.c | 13 +
>  tools/libxl/libxl_internal.h   | 36 
>  tools/libxl/libxl_no_colo.c| 62 
> ++
>  5 files changed, 117 insertions(+), 51 deletions(-)
>  create mode 100644 tools/libxl/libxl_no_colo.c
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 0/3] COLO: only build on Linux

[Wen Congyang]

On 04/06/2016 06:17 PM, Wei Liu wrote:
> On Wed, Apr 06, 2016 at 09:24:26AM +0800, Wen Congyang wrote:
>> On 04/06/2016 04:05 AM, Wei Liu wrote:
>>> COLO depends on netlink which is only available on Linux. This series 
>>> cleans up
>>> COLO code and make it only build on Linux. This should fix FreeBSD build.
>>>
>>> Congyang and Changlong, please review this series as soon as possible. I 
>>> also
>>> have a question why COLO doesn't use libnl? It is using Linux header 
>>> directly.
>>
>> It is implemented by Yang one year ago. I don't know the reason. We will 
>> investigate
>> how to use libnl.
>>
> 
> It's easy -- change inclusion of linux/netlink.h to netlink/netlink.h. I
> already have a series to do that -- this series is actually my second
> attempt to fix COLO.
> 
> If you can confirm COLO should depend on libnl, I will post my patches.

I think COLO should depend on libnl.

Thanks
Wen Congyang

> 
> Wei.
> 
>> Thanks
>> Wen Congyang
>>
>>>
>>> Roger, can you run functional tests on FreeBSD? I don't have FreeBSD Dom0.
>>>
>>> Wei.
>>>
>>> Wei Liu (3):
>>>   libxl: colo: rearrange things in header files
>>>   libxl: colo: move netlink related stuff to libxl_colo_proxy.c
>>>   libxl: colo: only build COLO on Linux
>>>
>>>  tools/libxl/Makefile   |  5 
>>>  tools/libxl/libxl_colo.h   | 52 +--
>>>  tools/libxl/libxl_colo_proxy.c | 13 +
>>>  tools/libxl/libxl_internal.h   | 36 
>>>  tools/libxl/libxl_no_colo.c| 62 
>>> ++
>>>  5 files changed, 117 insertions(+), 51 deletions(-)
>>>  create mode 100644 tools/libxl/libxl_no_colo.c
>>>
>>
>>
>>
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH for-4.7] tools/libxl: Fix legacy migration following COLO backchannel breakage

[Wen Congyang]

On 04/15/2016 03:54 AM, Andrew Cooper wrote:
> c/s f5d947bf1b "tools/libxl: add back channel support to read stream"
> made a bogus adjustment to libxl__stream_read_start(), including
> removing the comment hinting at what was going on, which breaks
> conversion of a legacy migration stream.
> 
> Symptoms look like:
> 
>   root@anonymi:~ # xl migrate domU host
>   migration target: Ready to receive domain.
>   Saving to migration stream new xl format (info 0x1/0x0/2677)
>   xc: error: error polling suspend notification channel: -1: Internal error
>   Loading new save file  (new xl fmt info 
> 0x1/0x0/2677)
>Savefile contains xl domain config in JSON format
>   Parsing config from 
>   libxl: error: libxl_stream_read.c:327:stream_header_done: Invalid ident: 
> expected 0x4c6962786c466d74, got 0x01f00f00
>   libxl: error: libxl_utils.c:430:libxl_read_exactly: file/stream truncated 
> reading ipc msg header from domain 1 save/restore helper stdout pipe
> 
> The adjustment is not required for backchannel support (as there is no
> interaction between back channels and legacy conversion), and caused
> stream->fd to be latched in the datacopier before legacy conversion
> substitutes it for the fd which is the output of the conversion script.
> 
> This causes libxl to consume data from the legacy stream rather than the
> v2 stream, and for the conversion script to encounter an error as the
> legacy stream appears to skip ahead.
> 
> Undo the adjustments to libxl__stream_read_start(), and introduce a
> better description of what is going on.  Introduce some extra assertions
> to try and catch similar breakage in the future.
> 
> Reported-by: Olaf Hering 
> Signed-off-by: Andrew Cooper 

Reviewed-by: Wen Congyang 

> ---
> CC: Ian Jackson 
> CC: Wei Liu 
> CC: Olaf Hering 
> CC: Yang Hongyang 
> CC: Wen Congyang 
> CC: Changlong Xie 
> ---
>  tools/libxl/libxl_stream_read.c | 33 -
>  1 file changed, 24 insertions(+), 9 deletions(-)
> 
> diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
> index 9659051..89c2f21 100644
> --- a/tools/libxl/libxl_stream_read.c
> +++ b/tools/libxl/libxl_stream_read.c
> @@ -234,16 +234,16 @@ void libxl__stream_read_start(libxl__egc *egc,
>  stream->running = true;
>  stream->phase   = SRS_PHASE_NORMAL;
>  
> -dc->ao   = stream->ao;
> -dc->copywhat = "restore v2 stream";
> -dc->readfd = stream->fd;
> -dc->writefd  = -1;
> -
> -if (stream->back_channel)
> -return;
> -
>  if (stream->legacy) {
> -/* Convert the legacy stream. */
> +/*
> + * Convert the legacy stream.
> + *
> + * This results in a fork()/exec() of conversion helper script.  It 
> is
> + * passed the exiting stream->fd as an input, and returns the
> + * transformed stream via a new pipe.  The fd of this new pipe then
> + * replaces stream->fd, to make the rest of the stream read code
> + * agnostic to whether legacy conversion is happening or not.
> + */
>  libxl__conversion_helper_state *chs = &stream->chs;
>  
>  chs->legacy_fd = stream->fd;
> @@ -258,10 +258,25 @@ void libxl__stream_read_start(libxl__egc *egc,
>  goto err;
>  }
>  
> +/* There should be no interaction of COLO backchannels and legacy
> + * stream conversion. */
> +assert(!stream->back_channel);
> +
> +/* Confirm *dc is still zeroed out, while we shuffle stream->fd. */
> +assert(dc->ao == NULL);
>  assert(stream->chs.v2_carefd);
>  stream->fd = libxl__carefd_fd(stream->chs.v2_carefd);
>  stream->dcs->libxc_fd = stream->fd;
>  }
> +/* stream->fd is now a v2 stream. */
> +
> +dc->ao   = stream->ao;
> +dc->copywhat = "restore v2 stream";
> +dc->readfd   = stream->fd;
> +dc->writefd  = -1;
> +
> +if (stream->back_channel)
> +return;
>  
>  /* Start reading the stream header. */
>  rc = setup_read(stream, "stream header",
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Error migrating VM to secondary host using COLO replication

[Wen Congyang]

On 11/10/2016 01:31 AM, Sadi wrote:
> Hello again,
> 
> Looking at the primary host's syslog, the arptables command from 
> xen/etc/scripts/colo-proxy-setup has failed.
> 
> Here's the log:
> 
> Nov  9 14:43:39 colop colop: /etc/xen/scripts/colo-proxy-setup: setup 
> XENBUS_PATH=
> Nov  9 14:43:39 colop kernel: [  302.825788] u32 classifier
> Nov  9 14:43:39 colop kernel: [  302.825791] Actions configured
> Nov  9 14:43:39 colop kernel: [  302.835407] Mirror/redirect action on
> Nov  9 14:43:39 colop kernel: [  302.919605] ip6_tables: (C) 2000-2006 
> Netfilter Core Team
> Nov  9 14:43:39 colop kernel: [  302.941511] arp_tables: (C) 2002 David S. 
> Miller
> Nov  9 14:43:39 colop colop: /etc/xen/scripts/colo-proxy-setup: arptables -I 
> INPUT -i eth1 -j MARK --set-mark 1 failed
> Nov  9 14:43:39 colop colop: /etc/xen/scripts/colo-proxy-setup: Writing 
> /hotplug-status connected to xenstore.
> Nov  9 14:43:39 colop colop: /etc/xen/scripts/colo-proxy-setup: Successful 
> colo-proxy-setup setup for vif2.0-emu. mode = primary  vifname: vif2.0-emu, 
> index: 1, forwarddev: eth1.
> 
> It's ok for the --set-mark argument to have value equal '1' , not a hex?

This log is very useful, we will investigate it.

Thanks
Wen Congyang

> 
> The i got running the command is:
> root@colop:~# arptables -I INPUT -i eth1 -j MARK --set-mark 1
> Bad argument `1'
> 
> Thanks, Sadi.
> 
> 
> 
> On Tue, Nov 8, 2016 at 6:53 PM, Sadi  <mailto:sadi...@gmail.com>> wrote:
> 
> Hi,
> 
> Apparently vif2.0-emu was already binded with br0 when "brctl addif br0 
> vif2.0-emu" failed.
> 
> root@colob-HP-Compaq-6005-Pro-MT-PC:~# brctl addif br0 vif2.0-emu
> device vif2.0-emu is already a member of a bridge; can't enslave it to 
> bridge br0.
> root@colob-HP-Compaq-6005-Pro-MT-PC:~# brctl show
> bridge name bridge id   STP enabled interfaces
> br0 8000.001a3fc46255   no  eth0
> vif2.0
> vif2.0-emu
> br1 8000.   no
> 
> About the iptables, it seems like SECCOLO target can't be recognised.
> 
> root@colob-HP-Compaq-6005-Pro-MT-PC:~# iptables -t mangle -D PREROUTING 
> -m physdev --physdev-in vif2.0-emu -j SECCOLO --index 1
> iptables: No chain/target/match by that name.
> 
> Here is my active modules matching colo:
> 
> root@colob-HP-Compaq-6005-Pro-MT-PC:~# lsmod | grep -i colo
> xt_SECCOLO 16384  1
> nf_conntrack_colo  16384  2 xt_SECCOLO
> x_tables   36864  8 
> xt_physdev,ip6table_mangle,ip_tables,xt_SECCOLO,xt_tcpudp,iptable_filter,iptable_mangle,ip6_tables
> nf_conntrack  106496  4 
> xt_SECCOLO,nf_nat,nf_conntrack_colo,nf_conntrack_ipv4
> 
> So i was looking in the iptables and this really looks like the source of 
> the problem. 
> 
> Sadi.
> 
> On Tue, Nov 8, 2016 at 5:57 PM, Konrad Rzeszutek Wilk 
> mailto:konrad.w...@oracle.com>> wrote:
> 
> > entered forwarding state
> > Nov  7 18:10:30 colob NetworkManager[907]:   (vif2.0-emu): 
> enslaved
> > to br0
> > Nov  7 18:10:30 colob root: /etc/xen/scripts/colo-proxy-setup: 
> brctl addif
> > br0 vif2.0-emu failed
> 
> 
> How come this failed?
> 
> > Nov  7 18:10:30 colob root: /etc/xen/scripts/colo-proxy-setup: 
> iptables -t
> > mangle -D PREROUTING -m physdev --physdev-in vif2.0-emu -j SECCOLO 
> --index
> > 1 failed
> 
> Ah b/c of this. Are there any errors of what exactly failed?
> 
> 
> 
> 
> -- 
> Sadi.
> 
> 
> 
> 
> -- 
> Sadi.




___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 1/3] Don't create default ioreq server

[Wen Congyang]

At 2016/11/30 20:25, Andrew Cooper wrote:

On 30/11/16 09:47, Zhang Chen wrote:

The ioreq server make colo run failed.

Signed-off-by: Zhang Chen 
Signed-off-by: Wen Congyang 


Nack.

You can simply "fix" a COLO issue by breaking a much more common usecase.


Yes, this patch is wrong. It is only in my git tree to let COLO work.




What actually breaks in the COLO case here?


I have reported this BUG last year:
https://lists.xenproject.org/archives/html/xen-devel/2015-12/msg02850.html


Thanks
Wen Congyang



~Andrew


---
  xen/arch/x86/hvm/hvm.c | 11 ---
  1 file changed, 11 deletions(-)

diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c
index 25dc759..8522852 100644
--- a/xen/arch/x86/hvm/hvm.c
+++ b/xen/arch/x86/hvm/hvm.c
@@ -5339,17 +5339,6 @@ static int hvmop_get_param(
  case HVM_PARAM_IOREQ_PFN:
  case HVM_PARAM_BUFIOREQ_PFN:
  case HVM_PARAM_BUFIOREQ_EVTCHN:
-{
-domid_t domid;
-
-/* May need to create server. */
-domid = d->arch.hvm_domain.params[HVM_PARAM_DM_DOMAIN];
-rc = hvm_create_ioreq_server(d, domid, 1,
- HVM_IOREQSRV_BUFIOREQ_LEGACY, NULL);
-if ( rc != 0 && rc != -EEXIST )
-goto out;
-}
-/*FALLTHRU*/
  default:
  a.value = d->arch.hvm_domain.params[a.index];
  break;



___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel



___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] Error migrating VM to secondary host using COLO replication

[Wen Congyang]

On 10/28/2016 06:56 AM, Sadi wrote:
> Hello,
> 
> I've been trying to set COLO replication to work but i'm stuck on a problem 
> when migrating de primary VM to secondary host.
> 
> I have been following the instructions from this wiki
> 
> - http://wiki.xenproject.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
> 
> and this mail thread
> 
>  - 
> http://xen.markmail.org/search/?q=COLO#query:COLO+page:1+mid:fb7wrn62vbks4unn+state:results
> 
> I'm anexing the steps i took setting the environment before facing this 
> problem when executing 'xl remus' command:

Sorry for the late reply.

> 
>>migration target: Ready to receive domain.
>>Saving to migration stream new xl format (info 0x3/0x0/2840)
>>Loading new save file  (new xl fmt info 
>>0x3/0x0/2840)
>>Savefile contains xl domain config in JSON format
>>Parsing config from 
>>xc: info: Saving domain 2, type x86 HVM
>>xc: info: Found x86 HVM domain from Xen 4.7
>>xc: info: Restoring domain
>>xc: Frames iteration 0 of 5: 1045504/1045504  100%
>>xc: Domain now suspended: 0/00%
>>libxl: error: libxl_qmp.c:702:libxl__qmp_initialize: Connection error: No 
>>such file or directory
>>libxl: error: libxl_colo_restore.c:817:colo_restore_setup_cds_done: COLO: 
>>failed to setup device >for guest with domid 1

According to the log, we cannot connect to the secondary qemu. Can you give the 
secondry qemu log?

Thanks
Wen Congyang

>>xc: error: Restore failed (38 = Function not implemented): Internal error
>>libxl: info: libxl_colo_restore.c:320:libxl__colo_restore_teardown: colo fails
>>libxl: error: libxl_stream_read.c:852:libxl__xc_domain_restore_done: 
>>restoring domain: Function >not implemented
>>libxl: info: libxl_colo_restore.c:320:libxl__colo_restore_teardown: colo fails
> 
> I'm hoping that someone could provide with directions.
> 
> Thanks for your time and sory for bad english (not native language).
> 
> 
> Sadi.
> 
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> https://lists.xen.org/xen-devel
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq page only one time

[Wen Congyang]

Cc: Paul Durrant

On 06/10/2015 03:44 PM, Andrew Cooper wrote:
> On 10/06/2015 06:26, Yang Hongyang wrote:
>>
>>
>> On 06/09/2015 03:30 PM, Andrew Cooper wrote:
>>> On 09/06/2015 01:59, Yang Hongyang wrote:
>>>>
>>>>
>>>> On 06/08/2015 06:15 PM, Andrew Cooper wrote:
>>>>> On 08/06/15 10:58, Yang Hongyang wrote:
>>>>>>
>>>>>>
>>>>>> On 06/08/2015 05:46 PM, Andrew Cooper wrote:
>>>>>>> On 08/06/15 04:43, Yang Hongyang wrote:
>>>>>>>> ioreq page contains evtchn which will be set when we resume the
>>>>>>>> secondary vm the first time. The hypervisor will check if the
>>>>>>>> evtchn is corrupted, so we cannot zero the ioreq page more
>>>>>>>> than one time.
>>>>>>>>
>>>>>>>> The ioreq->state is always STATE_IOREQ_NONE after the vm is
>>>>>>>> suspended, so it is OK if we only zero it one time.
>>>>>>>>
>>>>>>>> Signed-off-by: Yang Hongyang 
>>>>>>>> Signed-off-by: Wen congyang 
>>>>>>>> CC: Andrew Cooper 
>>>>>>>
>>>>>>> The issue here is that we are running the restore algorithm over a
>>>>>>> domain which has already been running in Xen for a while.  This is a
>>>>>>> brand new usecase, as far as I am aware.
>>>>>>
>>>>>> Exactly.
>>>>>>
>>>>>>>
>>>>>>> Does the qemu process associated with this domain get frozen
>>>>>>> while the
>>>>>>> secondary is being reset, or does the process get destroyed and
>>>>>>> recreated.
>>>>>>
>>>>>> What do you mean by reset? do you mean secondary is suspended at
>>>>>> checkpoint?
>>>>>
>>>>> Well - at the point that the buffered records are being processed, we
>>>>> are in the process of resetting the state of the secondary to match
>>>>> the
>>>>> primary.
>>>>
>>>> Yes, at this point, the qemu process associated with this domain is
>>>> frozen.
>>>> the suspend callback will call libxl__qmp_stop(vm_stop() in qemu) to
>>>> pause
>>>> qemu. After we processed all records, qemu will be restored with the
>>>> received
>>>> state, that's why we add a libxl__qmp_restore(qemu_load_vmstate() in
>>>> qemu)
>>>> api to restore qemu with received state. Currently in libxl, qemu only
>>>> start
>>>> with the received state, there's no api to load received state while
>>>> qemu is
>>>> running for a while.
>>>
>>> Now I consider this more, it is absolutely wrong to not zero the page
>>> here.  The event channel in the page is not guaranteed to be the same
>>> between the primary and secondary,
>>
>> That's why we don't zero it on secondary.
> 
> I think you missed my point.  Apologies for the double negative.   It
> must, under all circumstances, be zeroed at this point, for safety reasons.
> 
> The page in question is subject to logdirty just like any other guest
> pages, which means that if the guest writes to it naturally (i.e. not a
> Xen or Qemu write, both of whom have magic mappings which are not
> subject to logdirty), it will be transmitted in the stream.  As the
> event channel could be different, the lack of zeroing it at this point
> means that the event channel would be wrong as opposed to simply
> missing.  This is a worse position to be in.

The guest should not access this page. I am not sure if the guest can
access the ioreq page.

But in the exceptional case, the ioreq page is dirtied, and is copied to
the secondary vm. The ioreq page will contain a wrong event channel, the
hypervisor will check it: if the event channel is wrong, the guest will
be crashed.

> 
>>
>>> and we don't want to unexpectedly
>>> find a pending/in-flight ioreq.
>>
>> ioreq->state is always STATE_IOREQ_NONE after the vm is suspended, there
>> should be no pending/in-flight ioreq at checkpoint.
> 
> In the common case perhaps, but we must consider the exceptional case. 
> The exceptional case here is some corruption which happens to appear as
> an in-flight ioreq.

If the state is STATE_IOREQ_NONE, it may be hypervisor's bug. If the hypervisor
has a bug, anything can happen. I think we should trust the hypervisor.

> 
>>
>>>
>>> Either qemu needs to take care of re-initialising the event channels
>>> back to appropriate values, or Xen should tolerate the channels
>>> disappearing.
> 
> I still stand by this statement.  I believe it is the only safe way of
> solving the issue you have discovered.

Add a new qemu monitor command to update ioreq page?

Thanks
Wen Congyang

> 
> ~Andrew
> .
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq page only one time

[Wen Congyang]

On 06/10/2015 06:40 PM, Paul Durrant wrote:
>> -Original Message-
>> From: Wen Congyang [mailto:we...@cn.fujitsu.com]
>> Sent: 10 June 2015 10:06
>> To: Andrew Cooper; Yang Hongyang; xen-devel@lists.xen.org; Paul Durrant
>> Cc: Wei Liu; Ian Campbell; yunhong.ji...@intel.com; Eddie Dong;
>> guijianf...@cn.fujitsu.com; rshri...@cs.ubc.ca; Ian Jackson
>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq
>> page only one time
>>
>> Cc: Paul Durrant
>>
>> On 06/10/2015 03:44 PM, Andrew Cooper wrote:
>>> On 10/06/2015 06:26, Yang Hongyang wrote:
>>>>
>>>>
>>>> On 06/09/2015 03:30 PM, Andrew Cooper wrote:
>>>>> On 09/06/2015 01:59, Yang Hongyang wrote:
>>>>>>
>>>>>>
>>>>>> On 06/08/2015 06:15 PM, Andrew Cooper wrote:
>>>>>>> On 08/06/15 10:58, Yang Hongyang wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 06/08/2015 05:46 PM, Andrew Cooper wrote:
>>>>>>>>> On 08/06/15 04:43, Yang Hongyang wrote:
>>>>>>>>>> ioreq page contains evtchn which will be set when we resume the
>>>>>>>>>> secondary vm the first time. The hypervisor will check if the
>>>>>>>>>> evtchn is corrupted, so we cannot zero the ioreq page more
>>>>>>>>>> than one time.
>>>>>>>>>>
>>>>>>>>>> The ioreq->state is always STATE_IOREQ_NONE after the vm is
>>>>>>>>>> suspended, so it is OK if we only zero it one time.
>>>>>>>>>>
>>>>>>>>>> Signed-off-by: Yang Hongyang 
>>>>>>>>>> Signed-off-by: Wen congyang 
>>>>>>>>>> CC: Andrew Cooper 
>>>>>>>>>
>>>>>>>>> The issue here is that we are running the restore algorithm over a
>>>>>>>>> domain which has already been running in Xen for a while.  This is a
>>>>>>>>> brand new usecase, as far as I am aware.
>>>>>>>>
>>>>>>>> Exactly.
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Does the qemu process associated with this domain get frozen
>>>>>>>>> while the
>>>>>>>>> secondary is being reset, or does the process get destroyed and
>>>>>>>>> recreated.
>>>>>>>>
>>>>>>>> What do you mean by reset? do you mean secondary is suspended
>> at
>>>>>>>> checkpoint?
>>>>>>>
>>>>>>> Well - at the point that the buffered records are being processed, we
>>>>>>> are in the process of resetting the state of the secondary to match
>>>>>>> the
>>>>>>> primary.
>>>>>>
>>>>>> Yes, at this point, the qemu process associated with this domain is
>>>>>> frozen.
>>>>>> the suspend callback will call libxl__qmp_stop(vm_stop() in qemu) to
>>>>>> pause
>>>>>> qemu. After we processed all records, qemu will be restored with the
>>>>>> received
>>>>>> state, that's why we add a libxl__qmp_restore(qemu_load_vmstate()
>> in
>>>>>> qemu)
>>>>>> api to restore qemu with received state. Currently in libxl, qemu only
>>>>>> start
>>>>>> with the received state, there's no api to load received state while
>>>>>> qemu is
>>>>>> running for a while.
>>>>>
>>>>> Now I consider this more, it is absolutely wrong to not zero the page
>>>>> here.  The event channel in the page is not guaranteed to be the same
>>>>> between the primary and secondary,
>>>>
>>>> That's why we don't zero it on secondary.
>>>
>>> I think you missed my point.  Apologies for the double negative.   It
>>> must, under all circumstances, be zeroed at this point, for safety reasons.
>>>
>>> The page in question is subject to logdirty just like any other guest
>>> pages, which means that if the guest writes to it naturally (i.e. not a
>>> Xen or Qemu write, both of whom have magic mappings which are not
>>> subject to logdirty), it wi

Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq page only one time

[Wen Congyang]

On 06/10/2015 06:58 PM, Paul Durrant wrote:
>> -Original Message-
>> From: Wen Congyang [mailto:we...@cn.fujitsu.com]
>> Sent: 10 June 2015 11:55
>> To: Paul Durrant; Andrew Cooper; Yang Hongyang; xen-devel@lists.xen.org
>> Cc: Wei Liu; Ian Campbell; yunhong.ji...@intel.com; Eddie Dong;
>> guijianf...@cn.fujitsu.com; rshri...@cs.ubc.ca; Ian Jackson
>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq
>> page only one time
>>
>> On 06/10/2015 06:40 PM, Paul Durrant wrote:
>>>> -Original Message-
>>>> From: Wen Congyang [mailto:we...@cn.fujitsu.com]
>>>> Sent: 10 June 2015 10:06
>>>> To: Andrew Cooper; Yang Hongyang; xen-devel@lists.xen.org; Paul
>> Durrant
>>>> Cc: Wei Liu; Ian Campbell; yunhong.ji...@intel.com; Eddie Dong;
>>>> guijianf...@cn.fujitsu.com; rshri...@cs.ubc.ca; Ian Jackson
>>>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero
>> ioreq
>>>> page only one time
>>>>
>>>> Cc: Paul Durrant
>>>>
>>>> On 06/10/2015 03:44 PM, Andrew Cooper wrote:
>>>>> On 10/06/2015 06:26, Yang Hongyang wrote:
>>>>>>
>>>>>>
>>>>>> On 06/09/2015 03:30 PM, Andrew Cooper wrote:
>>>>>>> On 09/06/2015 01:59, Yang Hongyang wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 06/08/2015 06:15 PM, Andrew Cooper wrote:
>>>>>>>>> On 08/06/15 10:58, Yang Hongyang wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 06/08/2015 05:46 PM, Andrew Cooper wrote:
>>>>>>>>>>> On 08/06/15 04:43, Yang Hongyang wrote:
>>>>>>>>>>>> ioreq page contains evtchn which will be set when we resume
>> the
>>>>>>>>>>>> secondary vm the first time. The hypervisor will check if the
>>>>>>>>>>>> evtchn is corrupted, so we cannot zero the ioreq page more
>>>>>>>>>>>> than one time.
>>>>>>>>>>>>
>>>>>>>>>>>> The ioreq->state is always STATE_IOREQ_NONE after the vm is
>>>>>>>>>>>> suspended, so it is OK if we only zero it one time.
>>>>>>>>>>>>
>>>>>>>>>>>> Signed-off-by: Yang Hongyang 
>>>>>>>>>>>> Signed-off-by: Wen congyang 
>>>>>>>>>>>> CC: Andrew Cooper 
>>>>>>>>>>>
>>>>>>>>>>> The issue here is that we are running the restore algorithm over
>> a
>>>>>>>>>>> domain which has already been running in Xen for a while.  This
>> is a
>>>>>>>>>>> brand new usecase, as far as I am aware.
>>>>>>>>>>
>>>>>>>>>> Exactly.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Does the qemu process associated with this domain get frozen
>>>>>>>>>>> while the
>>>>>>>>>>> secondary is being reset, or does the process get destroyed and
>>>>>>>>>>> recreated.
>>>>>>>>>>
>>>>>>>>>> What do you mean by reset? do you mean secondary is
>> suspended
>>>> at
>>>>>>>>>> checkpoint?
>>>>>>>>>
>>>>>>>>> Well - at the point that the buffered records are being processed,
>> we
>>>>>>>>> are in the process of resetting the state of the secondary to match
>>>>>>>>> the
>>>>>>>>> primary.
>>>>>>>>
>>>>>>>> Yes, at this point, the qemu process associated with this domain is
>>>>>>>> frozen.
>>>>>>>> the suspend callback will call libxl__qmp_stop(vm_stop() in qemu)
>> to
>>>>>>>> pause
>>>>>>>> qemu. After we processed all records, qemu will be restored with
>> the
>>>>>>>> received
>>>>>>>> state, that's why we add a
>> libxl__qmp_restore(qemu_load_vmstate()
>>>> in
>>>>>&g

Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq page only one time

[Wen Congyang]

On 06/10/2015 07:47 PM, Paul Durrant wrote:
>> -Original Message-
>> From: xen-devel-boun...@lists.xen.org [mailto:xen-devel-
>> boun...@lists.xen.org] On Behalf Of Wen Congyang
>> Sent: 10 June 2015 12:38
>> To: Paul Durrant; Andrew Cooper; Yang Hongyang; xen-devel@lists.xen.org
>> Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
>> yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian Jackson
>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq
>> page only one time
>>
>> On 06/10/2015 06:58 PM, Paul Durrant wrote:
>>>> -Original Message-
>>>> From: Wen Congyang [mailto:we...@cn.fujitsu.com]
>>>> Sent: 10 June 2015 11:55
>>>> To: Paul Durrant; Andrew Cooper; Yang Hongyang; xen-
>> de...@lists.xen.org
>>>> Cc: Wei Liu; Ian Campbell; yunhong.ji...@intel.com; Eddie Dong;
>>>> guijianf...@cn.fujitsu.com; rshri...@cs.ubc.ca; Ian Jackson
>>>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero
>> ioreq
>>>> page only one time
>>>>
>>>> On 06/10/2015 06:40 PM, Paul Durrant wrote:
>>>>>> -Original Message-
>>>>>> From: Wen Congyang [mailto:we...@cn.fujitsu.com]
>>>>>> Sent: 10 June 2015 10:06
>>>>>> To: Andrew Cooper; Yang Hongyang; xen-devel@lists.xen.org; Paul
>>>> Durrant
>>>>>> Cc: Wei Liu; Ian Campbell; yunhong.ji...@intel.com; Eddie Dong;
>>>>>> guijianf...@cn.fujitsu.com; rshri...@cs.ubc.ca; Ian Jackson
>>>>>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero
>>>> ioreq
>>>>>> page only one time
>>>>>>
>>>>>> Cc: Paul Durrant
>>>>>>
>>>>>> On 06/10/2015 03:44 PM, Andrew Cooper wrote:
>>>>>>> On 10/06/2015 06:26, Yang Hongyang wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 06/09/2015 03:30 PM, Andrew Cooper wrote:
>>>>>>>>> On 09/06/2015 01:59, Yang Hongyang wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 06/08/2015 06:15 PM, Andrew Cooper wrote:
>>>>>>>>>>> On 08/06/15 10:58, Yang Hongyang wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 06/08/2015 05:46 PM, Andrew Cooper wrote:
>>>>>>>>>>>>> On 08/06/15 04:43, Yang Hongyang wrote:
>>>>>>>>>>>>>> ioreq page contains evtchn which will be set when we
>> resume
>>>> the
>>>>>>>>>>>>>> secondary vm the first time. The hypervisor will check if the
>>>>>>>>>>>>>> evtchn is corrupted, so we cannot zero the ioreq page more
>>>>>>>>>>>>>> than one time.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The ioreq->state is always STATE_IOREQ_NONE after the vm
>> is
>>>>>>>>>>>>>> suspended, so it is OK if we only zero it one time.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Signed-off-by: Yang Hongyang 
>>>>>>>>>>>>>> Signed-off-by: Wen congyang 
>>>>>>>>>>>>>> CC: Andrew Cooper 
>>>>>>>>>>>>>
>>>>>>>>>>>>> The issue here is that we are running the restore algorithm
>> over
>>>> a
>>>>>>>>>>>>> domain which has already been running in Xen for a while.
>> This
>>>> is a
>>>>>>>>>>>>> brand new usecase, as far as I am aware.
>>>>>>>>>>>>
>>>>>>>>>>>> Exactly.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Does the qemu process associated with this domain get
>> frozen
>>>>>>>>>>>>> while the
>>>>>>>>>>>>> secondary is being reset, or does the process get destroyed
>> and
>>>>>>>>>>>>> recreated.
>>>>>>>>>

Re: [Xen-devel] [PATCH v2 COLOPre 02/13] tools/libxc: support to resume uncooperative HVM guests

[Wen Congyang]

On 06/10/2015 11:18 PM, Ian Campbell wrote:
> On Mon, 2015-06-08 at 11:43 +0800, Yang Hongyang wrote:
>> From: Wen Congyang 
>>
>> For PVHVM, the hypercall return code is 0, and it can be resumed
>> in a new domain context.
>> we suspend PVHVM and resume it is like this:
>> 1. suspend it via evtchn
>> 2. modifty the return code to 1
>> 3. the guest know that the suspend is cancelled, we will use fast path
>>to resume it.
>>
>> Under COLO, we will update the guest's state(modify memory, cpu's registers,
>> device status...). In this case, we cannot use the fast path to resume it.
>> Keep the return code 0, and use a slow path to resume the guest. We have
>> updated the guest state, so we call it a new domain context.
>>
>> For HVM, the hypercall is a NOP.
> 
> This doesn't match my reading of domain_resume on the Xen side, which is
> the ultimate effect of this hypercall. It seems to unpause the domain
> (and all vcpus) regardless of the domain type, including PVHVM vs HVM
> (which isn't something Xen is generally aware of anyway).
> 
> I also can't really follow the stuff about PVHVM vs HVM vs uncooperative
> guests, and I certainly can't see where the PVHVM vs HVM distinction is
> made in this patch.

Sorry for my mistake. I read the codes again:

1. suspend
a. PVHVM and PV: we use the same way to suspend the guest(send the suspend
   request to the guest)
b. pure HVM: we call xc_domain_shutdown(..., SHUTDOWN_suspend) to suspend
   the guest
c. ???: suspending the guest via XenBus control node
I don't know we will goto c in which case.

2. Resume:
a. fast path
   In this case, we don't change the guest's state.
   PV: modify the return code to 1, and than call the domctl: 
XEN_DOMCTL_resumedomain
   PVHVM: same with PV
   HVM: do nothing in modify_returncode, and than call the domctl: 
XEN_DOMCTL_resumedomain
b. slow
   In this case, we have changed the guest's state.
   PV: update start info, and reset all secondary CPU states. Than call the
   domctl: XEN_DOMCTL_resumedomain
   PVHVM and HVM can not be resumed.

For PVHVM, in my test, only call the domctl: XEN_DOMCTL_resumedomain
can work. I am not sure if we should update start info and reset all secondary 
CPU
states.

For pure HVM guest, in my test, only call the domctl: XEN_DOMCTL_resumedomain 
can
work.

So we can call libxl__domain_resume(..., 1) if we don't change the guest state, 
otherwise
call libxl__domain_resume(..., 0).

Any suggestion is welcomed.

Thanks
Wen Congyang


> 
> Ian.
> 
> 
>>
>> Signed-off-by: Wen Congyang 
>> Signed-off-by: Yang Hongyang 
>> ---
>>  tools/libxc/xc_resume.c | 22 ++
>>  1 file changed, 18 insertions(+), 4 deletions(-)
>>
>> diff --git a/tools/libxc/xc_resume.c b/tools/libxc/xc_resume.c
>> index e67bebd..bd82334 100644
>> --- a/tools/libxc/xc_resume.c
>> +++ b/tools/libxc/xc_resume.c
>> @@ -109,6 +109,23 @@ static int xc_domain_resume_cooperative(xc_interface 
>> *xch, uint32_t domid)
>>  return do_domctl(xch, &domctl);
>>  }
>>  
>> +static int xc_domain_resume_hvm(xc_interface *xch, uint32_t domid)
>> +{
>> +DECLARE_DOMCTL;
>> +
>> +/*
>> + * If it is PVHVM, the hypercall return code is 0, because this
>> + * is not a fast path resume, we do not modify_returncode as in
>> + * xc_domain_resume_cooperative.
>> + * (resuming it in a new domain context)
>> + *
>> + * If it is a HVM, the hypercall is a NOP.
>> + */
>> +domctl.cmd = XEN_DOMCTL_resumedomain;
>> +domctl.domain = domid;
>> +return do_domctl(xch, &domctl);
>> +}
>> +
>>  static int xc_domain_resume_any(xc_interface *xch, uint32_t domid)
>>  {
>>  DECLARE_DOMCTL;
>> @@ -138,10 +155,7 @@ static int xc_domain_resume_any(xc_interface *xch, 
>> uint32_t domid)
>>   */
>>  #if defined(__i386__) || defined(__x86_64__)
>>  if ( info.hvm )
>> -{
>> -ERROR("Cannot resume uncooperative HVM guests");
>> -return rc;
>> -}
>> +return xc_domain_resume_hvm(xch, domid);
>>  
>>  if ( xc_domain_get_guest_width(xch, domid, &dinfo->guest_width) != 0 )
>>  {
> 
> 
> .
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq page only one time

[Wen Congyang]

On 06/11/2015 04:32 PM, Paul Durrant wrote:
>> -Original Message-
>> From: Wen Congyang [mailto:we...@cn.fujitsu.com]
>> Sent: 11 June 2015 02:14
>> To: Paul Durrant; Andrew Cooper; Yang Hongyang; xen-devel@lists.xen.org
>> Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
>> yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian Jackson
>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq
>> page only one time
>>
>> On 06/10/2015 07:47 PM, Paul Durrant wrote:
>>>> -Original Message-
>>>> From: xen-devel-boun...@lists.xen.org [mailto:xen-devel-
>>>> boun...@lists.xen.org] On Behalf Of Wen Congyang
>>>> Sent: 10 June 2015 12:38
>>>> To: Paul Durrant; Andrew Cooper; Yang Hongyang; xen-
>> de...@lists.xen.org
>>>> Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
>>>> yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian Jackson
>>>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero
>> ioreq
>>>> page only one time
>>>>
>>>> On 06/10/2015 06:58 PM, Paul Durrant wrote:
>>>>>> -Original Message-
>>>>>> From: Wen Congyang [mailto:we...@cn.fujitsu.com]
>>>>>> Sent: 10 June 2015 11:55
>>>>>> To: Paul Durrant; Andrew Cooper; Yang Hongyang; xen-
>>>> de...@lists.xen.org
>>>>>> Cc: Wei Liu; Ian Campbell; yunhong.ji...@intel.com; Eddie Dong;
>>>>>> guijianf...@cn.fujitsu.com; rshri...@cs.ubc.ca; Ian Jackson
>>>>>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero
>>>> ioreq
>>>>>> page only one time
>>>>>>
>>>>>> On 06/10/2015 06:40 PM, Paul Durrant wrote:
>>>>>>>> -Original Message-
>>>>>>>> From: Wen Congyang [mailto:we...@cn.fujitsu.com]
>>>>>>>> Sent: 10 June 2015 10:06
>>>>>>>> To: Andrew Cooper; Yang Hongyang; xen-devel@lists.xen.org; Paul
>>>>>> Durrant
>>>>>>>> Cc: Wei Liu; Ian Campbell; yunhong.ji...@intel.com; Eddie Dong;
>>>>>>>> guijianf...@cn.fujitsu.com; rshri...@cs.ubc.ca; Ian Jackson
>>>>>>>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore:
>> zero
>>>>>> ioreq
>>>>>>>> page only one time
>>>>>>>>
>>>>>>>> Cc: Paul Durrant
>>>>>>>>
>>>>>>>> On 06/10/2015 03:44 PM, Andrew Cooper wrote:
>>>>>>>>> On 10/06/2015 06:26, Yang Hongyang wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 06/09/2015 03:30 PM, Andrew Cooper wrote:
>>>>>>>>>>> On 09/06/2015 01:59, Yang Hongyang wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 06/08/2015 06:15 PM, Andrew Cooper wrote:
>>>>>>>>>>>>> On 08/06/15 10:58, Yang Hongyang wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 06/08/2015 05:46 PM, Andrew Cooper wrote:
>>>>>>>>>>>>>>> On 08/06/15 04:43, Yang Hongyang wrote:
>>>>>>>>>>>>>>>> ioreq page contains evtchn which will be set when we
>>>> resume
>>>>>> the
>>>>>>>>>>>>>>>> secondary vm the first time. The hypervisor will check if
>> the
>>>>>>>>>>>>>>>> evtchn is corrupted, so we cannot zero the ioreq page
>> more
>>>>>>>>>>>>>>>> than one time.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> The ioreq->state is always STATE_IOREQ_NONE after the
>> vm
>>>> is
>>>>>>>>>>>>>>>> suspended, so it is OK if we only zero it one time.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Signed-off-by: Yang Hongyang 
>>>>>>>>>>>>>>>> Signed-off-by: Wen congyang 
>>>>>>>>>>>>>

Re: [Xen-devel] [PATCH v2 COLOPre 02/13] tools/libxc: support to resume uncooperative HVM guests

[Wen Congyang]

On 06/11/2015 04:44 PM, Ian Campbell wrote:
> On Thu, 2015-06-11 at 10:42 +0800, Wen Congyang wrote:
>> On 06/10/2015 11:18 PM, Ian Campbell wrote:
>>> On Mon, 2015-06-08 at 11:43 +0800, Yang Hongyang wrote:
>>>> From: Wen Congyang 
>>>>
>>>> For PVHVM, the hypercall return code is 0, and it can be resumed
>>>> in a new domain context.
>>>> we suspend PVHVM and resume it is like this:
>>>> 1. suspend it via evtchn
>>>> 2. modifty the return code to 1
>>>> 3. the guest know that the suspend is cancelled, we will use fast path
>>>>to resume it.
>>>>
>>>> Under COLO, we will update the guest's state(modify memory, cpu's 
>>>> registers,
>>>> device status...). In this case, we cannot use the fast path to resume it.
>>>> Keep the return code 0, and use a slow path to resume the guest. We have
>>>> updated the guest state, so we call it a new domain context.
>>>>
>>>> For HVM, the hypercall is a NOP.
>>>
>>> This doesn't match my reading of domain_resume on the Xen side, which is
>>> the ultimate effect of this hypercall. It seems to unpause the domain
>>> (and all vcpus) regardless of the domain type, including PVHVM vs HVM
>>> (which isn't something Xen is generally aware of anyway).
>>>
>>> I also can't really follow the stuff about PVHVM vs HVM vs uncooperative
>>> guests, and I certainly can't see where the PVHVM vs HVM distinction is
>>> made in this patch.
>>
>> Sorry for my mistake. I read the codes again:
>>
>> 1. suspend
>> a. PVHVM and PV: we use the same way to suspend the guest(send the suspend
>>request to the guest)
>> b. pure HVM: we call xc_domain_shutdown(..., SHUTDOWN_suspend) to suspend
>>the guest
>> c. ???: suspending the guest via XenBus control node
> 
> AFAIK c is another option under a, it depends on whether the guest
> supports evtchn or not, if not then the xenstore variant will be used.

I remember it now. IIRC, the behavior in the guest are the same. Is it right?

Thanks
Wen Congyang

> 
>> I don't know we will goto c in which case.
>>
>> 2. Resume:
>> a. fast path
>>In this case, we don't change the guest's state.
>>PV: modify the return code to 1, and than call the domctl: 
>> XEN_DOMCTL_resumedomain
>>PVHVM: same with PV
>>HVM: do nothing in modify_returncode, and than call the domctl: 
>> XEN_DOMCTL_resumedomain
>> b. slow
>>In this case, we have changed the guest's state.
>>PV: update start info, and reset all secondary CPU states. Than call the
>>domctl: XEN_DOMCTL_resumedomain
>>PVHVM and HVM can not be resumed.
>>
>> For PVHVM, in my test, only call the domctl: XEN_DOMCTL_resumedomain
>> can work. I am not sure if we should update start info and reset all 
>> secondary CPU
>> states.
>>
>> For pure HVM guest, in my test, only call the domctl: 
>> XEN_DOMCTL_resumedomain can
>> work.
>>
>> So we can call libxl__domain_resume(..., 1) if we don't change the guest 
>> state, otherwise
>> call libxl__domain_resume(..., 0).
>>
>> Any suggestion is welcomed.
>>
>> Thanks
>> Wen Congyang
>>
>>
>>>
>>> Ian.
>>>
>>>
>>>>
>>>> Signed-off-by: Wen Congyang 
>>>> Signed-off-by: Yang Hongyang 
>>>> ---
>>>>  tools/libxc/xc_resume.c | 22 ++
>>>>  1 file changed, 18 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/tools/libxc/xc_resume.c b/tools/libxc/xc_resume.c
>>>> index e67bebd..bd82334 100644
>>>> --- a/tools/libxc/xc_resume.c
>>>> +++ b/tools/libxc/xc_resume.c
>>>> @@ -109,6 +109,23 @@ static int xc_domain_resume_cooperative(xc_interface 
>>>> *xch, uint32_t domid)
>>>>  return do_domctl(xch, &domctl);
>>>>  }
>>>>  
>>>> +static int xc_domain_resume_hvm(xc_interface *xch, uint32_t domid)
>>>> +{
>>>> +DECLARE_DOMCTL;
>>>> +
>>>> +/*
>>>> + * If it is PVHVM, the hypercall return code is 0, because this
>>>> + * is not a fast path resume, we do not modify_returncode as in
>>>> + * xc_domain_resume_cooperative.
>>>> + * (resuming it in a new domain context)
>>>> + *
>>>> + * If it is a HVM, the hypercall is a NOP.
>>>> + */
>>>> +domctl.cmd = XEN_DOMCTL_resumedomain;
>>>> +domctl.domain = domid;
>>>> +return do_domctl(xch, &domctl);
>>>> +}
>>>> +
>>>>  static int xc_domain_resume_any(xc_interface *xch, uint32_t domid)
>>>>  {
>>>>  DECLARE_DOMCTL;
>>>> @@ -138,10 +155,7 @@ static int xc_domain_resume_any(xc_interface *xch, 
>>>> uint32_t domid)
>>>>   */
>>>>  #if defined(__i386__) || defined(__x86_64__)
>>>>  if ( info.hvm )
>>>> -{
>>>> -ERROR("Cannot resume uncooperative HVM guests");
>>>> -return rc;
>>>> -}
>>>> +return xc_domain_resume_hvm(xch, domid);
>>>>  
>>>>  if ( xc_domain_get_guest_width(xch, domid, &dinfo->guest_width) != 0 )
>>>>  {
>>>
>>>
>>> .
>>>
>>
> 
> 
> .
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2 COLOPre 06/13] tools/libxl: Introduce a new internal API libxl__domain_unpause()

[Wen Congyang]

On 06/11/2015 04:43 PM, Ian Campbell wrote:
> On Thu, 2015-06-11 at 10:21 +0800, Yang Hongyang wrote:
>>
>> On 06/10/2015 11:37 PM, Ian Campbell wrote:
>>> On Mon, 2015-06-08 at 11:43 +0800, Yang Hongyang wrote:
>>>> From: Wen Congyang 
>>>>
>>>> The guest is paused after libxl_domain_create_restore().
>>>> Secondary vm is running in colo mode. So we need to unpause
>>>> the guest. The current API libxl_domain_unpause() is
>>>> not an internal API. Introduce a new API to support it.
>>>> No functional change.
>>>
>>> In general there is nothing wrong with using a public function
>>> internally. Is there some special consideration here?
>>
>> It's just that we thought it's better to use internal functions for
>> internal purpose.
>> Most the public functions take ctx as the first param, the internal functions
>> take gc/egc as the first param(although we can get ctx from gcs and call
>> public functions when needed).
>> If it doesn't matter, we can drop this patch.
> 
> It doesn't matter so you can, yes.

If the public API creates a new AO, it is safe to call it directly?

Thanks
Wen Congyang

> 
>>
>>>
>>>>
>>>> Signed-off-by: Wen Congyang 
>>>> Signed-off-by: Yang Hongyang 
>>>> ---
>>>>   tools/libxl/libxl.c  | 20 ++--
>>>>   tools/libxl/libxl_internal.h |  1 +
>>>>   2 files changed, 15 insertions(+), 6 deletions(-)
>>>>
>>>> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
>>>> index ba2da92..d5691dc 100644
>>>> --- a/tools/libxl/libxl.c
>>>> +++ b/tools/libxl/libxl.c
>>>> @@ -933,9 +933,8 @@ out:
>>>>   return AO_INPROGRESS;
>>>>   }
>>>>
>>>> -int libxl_domain_unpause(libxl_ctx *ctx, uint32_t domid)
>>>> +int libxl__domain_unpause(libxl__gc *gc, uint32_t domid)
>>>>   {
>>>> -GC_INIT(ctx);
>>>>   char *path;
>>>>   char *state;
>>>>   int ret, rc = 0;
>>>> @@ -947,7 +946,7 @@ int libxl_domain_unpause(libxl_ctx *ctx, uint32_t 
>>>> domid)
>>>>   }
>>>>
>>>>   if (type == LIBXL_DOMAIN_TYPE_HVM) {
>>>> -uint32_t dm_domid = libxl_get_stubdom_id(ctx, domid);
>>>> +uint32_t dm_domid = libxl_get_stubdom_id(CTX, domid);
>>>>
>>>>   path = libxl__device_model_xs_path(gc, dm_domid, domid, 
>>>> "/state");
>>>>   state = libxl__xs_read(gc, XBT_NULL, path);
>>>> @@ -957,12 +956,21 @@ int libxl_domain_unpause(libxl_ctx *ctx, uint32_t 
>>>> domid)
>>>>NULL, NULL, NULL);
>>>>   }
>>>>   }
>>>> -ret = xc_domain_unpause(ctx->xch, domid);
>>>> -if (ret<0) {
>>>> -LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "unpausing domain %d", 
>>>> domid);
>>>> +
>>>> +ret = xc_domain_unpause(CTX->xch, domid);
>>>> +if (ret < 0) {
>>>> +LIBXL__LOG_ERRNO(CTX, LIBXL__LOG_ERROR, "unpausing domain %d", 
>>>> domid);
>>>>   rc = ERROR_FAIL;
>>>>   }
>>>>out:
>>>> +return rc;
>>>> +}
>>>> +
>>>> +int libxl_domain_unpause(libxl_ctx *ctx, uint32_t domid)
>>>> +{
>>>> +GC_INIT(ctx);
>>>> +int rc = libxl__domain_unpause(gc, domid);
>>>> +
>>>>   GC_FREE;
>>>>   return rc;
>>>>   }
>>>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>>>> index 20364c6..366470f 100644
>>>> --- a/tools/libxl/libxl_internal.h
>>>> +++ b/tools/libxl/libxl_internal.h
>>>> @@ -1044,6 +1044,7 @@ _hidden int libxl__domain_restore(libxl__gc *gc, 
>>>> uint32_t domid);
>>>>   _hidden int libxl__domain_resume(libxl__gc *gc, uint32_t domid,
>>>>int suspend_cancel);
>>>>   _hidden int libxl__domain_s3_resume(libxl__gc *gc, int domid);
>>>> +_hidden int libxl__domain_unpause(libxl__gc *gc, uint32_t domid);
>>>>
>>>>   /* returns 0 or 1, or a libxl error code */
>>>>   _hidden int libxl__domain_pvcontrol_available(libxl__gc *gc, uint32_t 
>>>> domid);
>>>
>>>
>>> .
>>>
>>
> 
> 
> .
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2 COLOPre 06/13] tools/libxl: Introduce a new internal API libxl__domain_unpause()

[Wen Congyang]

On 06/11/2015 05:42 PM, Ian Campbell wrote:
> On Thu, 2015-06-11 at 17:09 +0800, Wen Congyang wrote:
>> On 06/11/2015 04:43 PM, Ian Campbell wrote:
>>> On Thu, 2015-06-11 at 10:21 +0800, Yang Hongyang wrote:
>>>>
>>>> On 06/10/2015 11:37 PM, Ian Campbell wrote:
>>>>> On Mon, 2015-06-08 at 11:43 +0800, Yang Hongyang wrote:
>>>>>> From: Wen Congyang 
>>>>>>
>>>>>> The guest is paused after libxl_domain_create_restore().
>>>>>> Secondary vm is running in colo mode. So we need to unpause
>>>>>> the guest. The current API libxl_domain_unpause() is
>>>>>> not an internal API. Introduce a new API to support it.
>>>>>> No functional change.
>>>>>
>>>>> In general there is nothing wrong with using a public function
>>>>> internally. Is there some special consideration here?
>>>>
>>>> It's just that we thought it's better to use internal functions for
>>>> internal purpose.
>>>> Most the public functions take ctx as the first param, the internal 
>>>> functions
>>>> take gc/egc as the first param(although we can get ctx from gcs and call
>>>> public functions when needed).
>>>> If it doesn't matter, we can drop this patch.
>>>
>>> It doesn't matter so you can, yes.
>>
>> If the public API creates a new AO, it is safe to call it directly?
> 
> A public function which takes an ao_how is, I believe, an exception to
> this rule and should be annotated with LIBXL_EXTERNAL_CALLERS_ONLY to
> prevent accidents.
> 
> I don't think libxl_domain_unpause is such a function though.

OK, we will drop this patch.

Thanks
Wen Congyang

> 
> Ian.
> 
> 
> .
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq page only one time

[Wen Congyang]

On 06/11/2015 06:20 PM, Paul Durrant wrote:
>> -Original Message-
>> From: Wen Congyang [mailto:we...@cn.fujitsu.com]
>> Sent: 11 June 2015 09:48
>> To: Paul Durrant; Andrew Cooper; Yang Hongyang; xen-devel@lists.xen.org
>> Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
>> yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian Jackson
>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq
>> page only one time
>>
>> On 06/11/2015 04:32 PM, Paul Durrant wrote:
>>>> -Original Message-
>>>> From: Wen Congyang [mailto:we...@cn.fujitsu.com]
>>>> Sent: 11 June 2015 02:14
>>>> To: Paul Durrant; Andrew Cooper; Yang Hongyang; xen-
>> de...@lists.xen.org
>>>> Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
>>>> yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian Jackson
>>>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero
>> ioreq
>>>> page only one time
>>>>
>>>> On 06/10/2015 07:47 PM, Paul Durrant wrote:
>>>>>> -Original Message-
>>>>>> From: xen-devel-boun...@lists.xen.org [mailto:xen-devel-
>>>>>> boun...@lists.xen.org] On Behalf Of Wen Congyang
>>>>>> Sent: 10 June 2015 12:38
>>>>>> To: Paul Durrant; Andrew Cooper; Yang Hongyang; xen-
>>>> de...@lists.xen.org
>>>>>> Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
>>>>>> yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian Jackson
>>>>>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero
>>>> ioreq
>>>>>> page only one time
>>>>>>
>>>>>> On 06/10/2015 06:58 PM, Paul Durrant wrote:
>>>>>>>> -Original Message-
>>>>>>>> From: Wen Congyang [mailto:we...@cn.fujitsu.com]
>>>>>>>> Sent: 10 June 2015 11:55
>>>>>>>> To: Paul Durrant; Andrew Cooper; Yang Hongyang; xen-
>>>>>> de...@lists.xen.org
>>>>>>>> Cc: Wei Liu; Ian Campbell; yunhong.ji...@intel.com; Eddie Dong;
>>>>>>>> guijianf...@cn.fujitsu.com; rshri...@cs.ubc.ca; Ian Jackson
>>>>>>>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore:
>> zero
>>>>>> ioreq
>>>>>>>> page only one time
>>>>>>>>
>>>>>>>> On 06/10/2015 06:40 PM, Paul Durrant wrote:
>>>>>>>>>> -Original Message-
>>>>>>>>>> From: Wen Congyang [mailto:we...@cn.fujitsu.com]
>>>>>>>>>> Sent: 10 June 2015 10:06
>>>>>>>>>> To: Andrew Cooper; Yang Hongyang; xen-devel@lists.xen.org;
>> Paul
>>>>>>>> Durrant
>>>>>>>>>> Cc: Wei Liu; Ian Campbell; yunhong.ji...@intel.com; Eddie Dong;
>>>>>>>>>> guijianf...@cn.fujitsu.com; rshri...@cs.ubc.ca; Ian Jackson
>>>>>>>>>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore:
>>>> zero
>>>>>>>> ioreq
>>>>>>>>>> page only one time
>>>>>>>>>>
>>>>>>>>>> Cc: Paul Durrant
>>>>>>>>>>
>>>>>>>>>> On 06/10/2015 03:44 PM, Andrew Cooper wrote:
>>>>>>>>>>> On 10/06/2015 06:26, Yang Hongyang wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 06/09/2015 03:30 PM, Andrew Cooper wrote:
>>>>>>>>>>>>> On 09/06/2015 01:59, Yang Hongyang wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 06/08/2015 06:15 PM, Andrew Cooper wrote:
>>>>>>>>>>>>>>> On 08/06/15 10:58, Yang Hongyang wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 06/08/2015 05:46 PM, Andrew Cooper wrote:
>>>>>>>>>>>>>>>>> On 08/06/15 04:43, Yang Hongyang wrote:
>>>>>>>>>>>>>>>>>> ioreq pa

Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq page only one time

[Wen Congyang]

On 06/11/2015 09:25 PM, Paul Durrant wrote:
>> -Original Message-
>> From: Yang Hongyang [mailto:yan...@cn.fujitsu.com]
>> Sent: 11 June 2015 13:59
>> To: Paul Durrant; Wen Congyang; Andrew Cooper; xen-devel@lists.xen.org
>> Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
>> yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian Jackson
>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq
>> page only one time
>>
>>
>>
>> On 06/11/2015 06:20 PM, Paul Durrant wrote:
>>>> -Original Message-
>>>> From: Wen Congyang [mailto:we...@cn.fujitsu.com]
>>>> Sent: 11 June 2015 09:48
>>>> To: Paul Durrant; Andrew Cooper; Yang Hongyang; xen-
>> de...@lists.xen.org
>>>> Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
>> [...]
>>>
>>>>
>>>> In our implementation, we don't start a new emulator. The codes can
>> work,
>>>> but some bugs may be not triggered.
>>>>
>>>
>>> How do you reconcile the incoming QEMU save record with the running
>> emulator state?
>>
>> We introduce a qmp command "xen-load-devices-
>> state"(libxl__qmp_restore) which
>> can restore the emulator state. The step of resotre emulator state at a
>> checkpoint is:
>>
>> 1. libxl__qmp_stop-> vm_stop() in qemu
>> 2. libxl__qmp_restore -> load_vmstate() in qemu
>> 3. libxl__qmp_resume  -> vm_start() in qemu
>>
> 
> Ok, that sounds like the ideal time to hook back into Xen by creating a new 
> ioreq server.

I have some questions about ioreq server:
1. If we use old version xen and newest version qemu, is it OK? Is default
   ioreq server created when the guest is created. xen_create_ioreq_server() 
does
   nothing, and xen_get_ioreq_server_info() will get the default ioreq server 
information.
   Is it right?
2. Why we create a default ioreq server when getting the hvm param if there is 
already a
   not default ioreq server?
3. In the far end, we will clear the ioreq page, and this ioreq page is used 
for default
   ioreq server, is it right?

Thanks
Wen Congyang

> 
>   Paul
> 
>>>
>>>Paul
>>>
>>>> Thanks
>>>> Wen Congyang
>>>>
>>>>>
>>>>>Paul
>>>>>
>>>>>>
>>>>>> Thanks
>>>>>> Wen Congyang
>>>>>>
>>>>>>>
>>>>>>>Paul
>>>>>>>
>>>>>>>> We will set to the guest to a new state, the old state should be
>>>> dropped.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Wen Congyang
>>>>>>>>
>>>>>>>>>
>>>>>>>>>Paul
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>> Wen Congyang
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Paul
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Wen Congyang
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ~Andrew
>>>>>>>>>>>>> .
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> .
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> .
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ___
>>>>>>>> Xen-devel mailing list
>>>>>>>> Xen-devel@lists.xen.org
>>>>>>>> http://lists.xen.org/xen-devel
>>>>>>>
>>>>>>> ___
>>>>>>> Xen-devel mailing list
>>>>>>> Xen-devel@lists.xen.org
>>>>>>> http://lists.xen.org/xen-devel
>>>>>>> .
>>>>>>>
>>>>>
>>>>> .
>>>>>
>>>
>>>
>>> ___
>>> Xen-devel mailing list
>>> Xen-devel@lists.xen.org
>>> http://lists.xen.org/xen-devel
>>> .
>>>
>>
>> --
>> Thanks,
>> Yang.
> .
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq page only one time

[Wen Congyang]

On 06/12/2015 03:41 PM, Paul Durrant wrote:
>> -Original Message-
>> From: Wen Congyang [mailto:we...@cn.fujitsu.com]
>> Sent: 12 June 2015 04:22
>> To: Paul Durrant; Yang Hongyang; Andrew Cooper; xen-devel@lists.xen.org
>> Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
>> yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian Jackson
>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq
>> page only one time
>>
>> On 06/11/2015 09:25 PM, Paul Durrant wrote:
>>>> -Original Message-
>>>> From: Yang Hongyang [mailto:yan...@cn.fujitsu.com]
>>>> Sent: 11 June 2015 13:59
>>>> To: Paul Durrant; Wen Congyang; Andrew Cooper; xen-
>> de...@lists.xen.org
>>>> Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
>>>> yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian Jackson
>>>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero
>> ioreq
>>>> page only one time
>>>>
>>>>
>>>>
>>>> On 06/11/2015 06:20 PM, Paul Durrant wrote:
>>>>>> -Original Message-
>>>>>> From: Wen Congyang [mailto:we...@cn.fujitsu.com]
>>>>>> Sent: 11 June 2015 09:48
>>>>>> To: Paul Durrant; Andrew Cooper; Yang Hongyang; xen-
>>>> de...@lists.xen.org
>>>>>> Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
>>>> [...]
>>>>>
>>>>>>
>>>>>> In our implementation, we don't start a new emulator. The codes can
>>>> work,
>>>>>> but some bugs may be not triggered.
>>>>>>
>>>>>
>>>>> How do you reconcile the incoming QEMU save record with the running
>>>> emulator state?
>>>>
>>>> We introduce a qmp command "xen-load-devices-
>>>> state"(libxl__qmp_restore) which
>>>> can restore the emulator state. The step of resotre emulator state at a
>>>> checkpoint is:
>>>>
>>>> 1. libxl__qmp_stop-> vm_stop() in qemu
>>>> 2. libxl__qmp_restore -> load_vmstate() in qemu
>>>> 3. libxl__qmp_resume  -> vm_start() in qemu
>>>>
>>>
>>> Ok, that sounds like the ideal time to hook back into Xen by creating a new
>> ioreq server.
>>
>> I have some questions about ioreq server:
>> 1. If we use old version xen and newest version qemu, is it OK? Is default
>>ioreq server created when the guest is created. xen_create_ioreq_server()
>> does
>>nothing, and xen_get_ioreq_server_info() will get the default ioreq server
>> information.
>>Is it right?
> 
> No. It's not compatible in that direction. A new Xen will work with an old 
> QEMU but not the other way round.
> 
>> 2. Why we create a default ioreq server when getting the hvm param if there
>> is already a
>>not default ioreq server?
> 
> If something reads the 'legacy' HVM params then that is Xen's trigger to 
> create the default server. Any 'new' emulator should be using the ioreq 
> server hypercalls so the default server will not be needed.

If there are two ioreq servers: default ioreq server, and a ioreq server 
created by emulator. The guest can work it correctly in
this case? Is there any application(not emulator) that uses the libxenctrl 
directly?

Thanks
Wen Congyang

> 
>> 3. In the far end, we will clear the ioreq page, and this ioreq page is used 
>> for
>> default
>>ioreq server, is it right?
> 
> Yes, AFAIK it's only the 'magic' pages that get cleared at the far end - and 
> that includes the default server pages. Other ioreq servers will have their 
> pages cleared on re-insertion to the P2M at the source end when the server is 
> disabled.
> 
>   Paul
> 
>>
>> Thanks
>> Wen Congyang
>>
>>>
>>>   Paul
>>>
>>>>>
>>>>>Paul
>>>>>
>>>>>> Thanks
>>>>>> Wen Congyang
>>>>>>
>>>>>>>
>>>>>>>Paul
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Wen Congyang
>>>>>>>>
>>>>>>>>>
>>>>>>>>>Paul
>>>>>>>>>
>>>>>>>>>> We will set to the guest to a new state, the old st

Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq page only one time

[Wen Congyang]

On 06/12/2015 06:54 PM, Paul Durrant wrote:
>> -Original Message-
>> From: Wen Congyang [mailto:we...@cn.fujitsu.com]
>> Sent: 12 June 2015 11:26
>> To: Paul Durrant; Yang Hongyang; Andrew Cooper; xen-devel@lists.xen.org
>> Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
>> yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian Jackson
>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq
>> page only one time
>>
>> On 06/12/2015 03:41 PM, Paul Durrant wrote:
>>>> -Original Message-
>>>> From: Wen Congyang [mailto:we...@cn.fujitsu.com]
>>>> Sent: 12 June 2015 04:22
>>>> To: Paul Durrant; Yang Hongyang; Andrew Cooper; xen-
>> de...@lists.xen.org
>>>> Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
>>>> yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian Jackson
>>>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero
>> ioreq
>>>> page only one time
>>>>
>>>> On 06/11/2015 09:25 PM, Paul Durrant wrote:
>>>>>> -Original Message-
>>>>>> From: Yang Hongyang [mailto:yan...@cn.fujitsu.com]
>>>>>> Sent: 11 June 2015 13:59
>>>>>> To: Paul Durrant; Wen Congyang; Andrew Cooper; xen-
>>>> de...@lists.xen.org
>>>>>> Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
>>>>>> yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian Jackson
>>>>>> Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero
>>>> ioreq
>>>>>> page only one time
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 06/11/2015 06:20 PM, Paul Durrant wrote:
>>>>>>>> -Original Message-
>>>>>>>> From: Wen Congyang [mailto:we...@cn.fujitsu.com]
>>>>>>>> Sent: 11 June 2015 09:48
>>>>>>>> To: Paul Durrant; Andrew Cooper; Yang Hongyang; xen-
>>>>>> de...@lists.xen.org
>>>>>>>> Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
>>>>>> [...]
>>>>>>>
>>>>>>>>
>>>>>>>> In our implementation, we don't start a new emulator. The codes
>> can
>>>>>> work,
>>>>>>>> but some bugs may be not triggered.
>>>>>>>>
>>>>>>>
>>>>>>> How do you reconcile the incoming QEMU save record with the
>> running
>>>>>> emulator state?
>>>>>>
>>>>>> We introduce a qmp command "xen-load-devices-
>>>>>> state"(libxl__qmp_restore) which
>>>>>> can restore the emulator state. The step of resotre emulator state at a
>>>>>> checkpoint is:
>>>>>>
>>>>>> 1. libxl__qmp_stop-> vm_stop() in qemu
>>>>>> 2. libxl__qmp_restore -> load_vmstate() in qemu
>>>>>> 3. libxl__qmp_resume  -> vm_start() in qemu
>>>>>>
>>>>>
>>>>> Ok, that sounds like the ideal time to hook back into Xen by creating a
>> new
>>>> ioreq server.
>>>>
>>>> I have some questions about ioreq server:
>>>> 1. If we use old version xen and newest version qemu, is it OK? Is default
>>>>ioreq server created when the guest is created.
>> xen_create_ioreq_server()
>>>> does
>>>>nothing, and xen_get_ioreq_server_info() will get the default ioreq
>> server
>>>> information.
>>>>Is it right?
>>>
>>> No. It's not compatible in that direction. A new Xen will work with an old
>> QEMU but not the other way round.

If the xen is newest, and qemu is old, how is the default ioreq server created
for the emulator? Will the old qemu call xc_get_hvm_param(), and the hypervisor
then creates a default ioreq server?
get is a 'readonly' operation, and creating the default ioreq server in it is
very strange.

>>>
>>>> 2. Why we create a default ioreq server when getting the hvm param if
>> there
>>>> is already a
>>>>not default ioreq server?
>>>
>>> If something reads the 'legacy' HVM params then that is Xen's trigger to
>> create the default server. Any 'new' emulator should be using the ioreq
>> server hypercalls so the default server will 

Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq page only one time

[Wen Congyang]

At 2015/6/12 19:48, Paul Durrant Wrote:

-Original Message-
From: Wen Congyang [mailto:we...@cn.fujitsu.com]
Sent: 12 June 2015 12:10
To: Paul Durrant; Yang Hongyang; Andrew Cooper; xen-devel@lists.xen.org
Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian Jackson
Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq
page only one time

On 06/12/2015 06:54 PM, Paul Durrant wrote:

-Original Message-
From: Wen Congyang [mailto:we...@cn.fujitsu.com]
Sent: 12 June 2015 11:26
To: Paul Durrant; Yang Hongyang; Andrew Cooper; xen-

de...@lists.xen.org

Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian Jackson
Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero

ioreq

page only one time

On 06/12/2015 03:41 PM, Paul Durrant wrote:

-Original Message-
From: Wen Congyang [mailto:we...@cn.fujitsu.com]
Sent: 12 June 2015 04:22
To: Paul Durrant; Yang Hongyang; Andrew Cooper; xen-

de...@lists.xen.org

Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian Jackson
Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero

ioreq

page only one time

On 06/11/2015 09:25 PM, Paul Durrant wrote:

-Original Message-
From: Yang Hongyang [mailto:yan...@cn.fujitsu.com]
Sent: 11 June 2015 13:59
To: Paul Durrant; Wen Congyang; Andrew Cooper; xen-

de...@lists.xen.org

Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian

Jackson

Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore:

zero

ioreq

page only one time



On 06/11/2015 06:20 PM, Paul Durrant wrote:

-Original Message-
From: Wen Congyang [mailto:we...@cn.fujitsu.com]
Sent: 11 June 2015 09:48
To: Paul Durrant; Andrew Cooper; Yang Hongyang; xen-

de...@lists.xen.org

Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;

[...]




In our implementation, we don't start a new emulator. The codes

can

work,

but some bugs may be not triggered.



How do you reconcile the incoming QEMU save record with the

running

emulator state?

We introduce a qmp command "xen-load-devices-
state"(libxl__qmp_restore) which
can restore the emulator state. The step of resotre emulator state at

a

checkpoint is:

1. libxl__qmp_stop-> vm_stop() in qemu
2. libxl__qmp_restore -> load_vmstate() in qemu
3. libxl__qmp_resume  -> vm_start() in qemu



Ok, that sounds like the ideal time to hook back into Xen by creating a

new

ioreq server.

I have some questions about ioreq server:
1. If we use old version xen and newest version qemu, is it OK? Is

default

ioreq server created when the guest is created.

xen_create_ioreq_server()

does
nothing, and xen_get_ioreq_server_info() will get the default ioreq

server

information.
Is it right?


No. It's not compatible in that direction. A new Xen will work with an old

QEMU but not the other way round.


If the xen is newest, and qemu is old, how is the default ioreq server created
for the emulator? Will the old qemu call xc_get_hvm_param(), and the
hypervisor
then creates a default ioreq server?
get is a 'readonly' operation, and creating the default ioreq server in it is
very strange.



Might be strange, but that's the way it's done.


If I only want to get the default ioreq server pfn, and don't want to 
create the default

ioreq server, how can I do it?

Thanks
Wen Congyang






2. Why we create a default ioreq server when getting the hvm param if

there

is already a
not default ioreq server?


If something reads the 'legacy' HVM params then that is Xen's trigger to

create the default server. Any 'new' emulator should be using the ioreq
server hypercalls so the default server will not be needed.

If there are two ioreq servers: default ioreq server, and a ioreq server
created by emulator. The guest can work it correctly in
this case?


You mean a secondary emulator? Yes, that's why there is the notion of

default ioreq server... to allow a secondary emulator to be used even when
an old QEMU is in use.

No, only one emulator. Can we run more than one emulator for one hvm
guest? How
to do it?



Yes, more than one emulator can run. There's nothing in libxl to do it, but we 
do it in XenServer using XAPI.




Is there any application(not emulator) that uses the libxenctrl
directly?



What do you mean by application? Toolstacks may use libxenctrl.


For example: libvirt. I know it uses libxl now. Is there any similar application
which uses libxenctrl.


Well, XAPI uses it for one.

   Paul



Thanks
Wen Congyang



   Paul


Thanks
Wen Congyang




3. In the far end, we will clear the ioreq page, and this ioreq page is

used

for

default
ioreq server, is it right?


Yes, A

Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq page only one time

[Wen Congyang]

At 2015/6/12 23:31, Paul Durrant Wrote:

-Original Message-
From: Wen Congyang [mailto:ghost...@gmail.com]
Sent: 12 June 2015 16:05
To: Paul Durrant; Wen Congyang; Yang Hongyang; Andrew Cooper; xen-
de...@lists.xen.org
Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian Jackson
Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero ioreq
page only one time

At 2015/6/12 19:48, Paul Durrant Wrote:

-Original Message-
From: Wen Congyang [mailto:we...@cn.fujitsu.com]
Sent: 12 June 2015 12:10
To: Paul Durrant; Yang Hongyang; Andrew Cooper; xen-

de...@lists.xen.org

Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian Jackson
Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero

ioreq

page only one time

On 06/12/2015 06:54 PM, Paul Durrant wrote:

-Original Message-
From: Wen Congyang [mailto:we...@cn.fujitsu.com]
Sent: 12 June 2015 11:26
To: Paul Durrant; Yang Hongyang; Andrew Cooper; xen-

de...@lists.xen.org

Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian Jackson
Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore: zero

ioreq

page only one time

On 06/12/2015 03:41 PM, Paul Durrant wrote:

-Original Message-
From: Wen Congyang [mailto:we...@cn.fujitsu.com]
Sent: 12 June 2015 04:22
To: Paul Durrant; Yang Hongyang; Andrew Cooper; xen-

de...@lists.xen.org

Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian

Jackson

Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore:

zero

ioreq

page only one time

On 06/11/2015 09:25 PM, Paul Durrant wrote:

-Original Message-
From: Yang Hongyang [mailto:yan...@cn.fujitsu.com]
Sent: 11 June 2015 13:59
To: Paul Durrant; Wen Congyang; Andrew Cooper; xen-

de...@lists.xen.org

Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;
yunhong.ji...@intel.com; Eddie Dong; rshri...@cs.ubc.ca; Ian

Jackson

Subject: Re: [Xen-devel] [PATCH v2 COLOPre 03/13] libxc/restore:

zero

ioreq

page only one time



On 06/11/2015 06:20 PM, Paul Durrant wrote:

-Original Message-
From: Wen Congyang [mailto:we...@cn.fujitsu.com]
Sent: 11 June 2015 09:48
To: Paul Durrant; Andrew Cooper; Yang Hongyang; xen-

de...@lists.xen.org

Cc: Wei Liu; Ian Campbell; guijianf...@cn.fujitsu.com;

[...]




In our implementation, we don't start a new emulator. The

codes

can

work,

but some bugs may be not triggered.



How do you reconcile the incoming QEMU save record with the

running

emulator state?

We introduce a qmp command "xen-load-devices-
state"(libxl__qmp_restore) which
can restore the emulator state. The step of resotre emulator state

at

a

checkpoint is:

1. libxl__qmp_stop-> vm_stop() in qemu
2. libxl__qmp_restore -> load_vmstate() in qemu
3. libxl__qmp_resume  -> vm_start() in qemu



Ok, that sounds like the ideal time to hook back into Xen by

creating a

new

ioreq server.

I have some questions about ioreq server:
1. If we use old version xen and newest version qemu, is it OK? Is

default

 ioreq server created when the guest is created.

xen_create_ioreq_server()

does
 nothing, and xen_get_ioreq_server_info() will get the default

ioreq

server

information.
 Is it right?


No. It's not compatible in that direction. A new Xen will work with an

old

QEMU but not the other way round.


If the xen is newest, and qemu is old, how is the default ioreq server

created

for the emulator? Will the old qemu call xc_get_hvm_param(), and the
hypervisor
then creates a default ioreq server?
get is a 'readonly' operation, and creating the default ioreq server in it is
very strange.



Might be strange, but that's the way it's done.


If I only want to get the default ioreq server pfn, and don't want to
create the default
ioreq server, how can I do it?


At the moment you can't. If you read the pfns then you will instantiate the 
server, which I agree is a bit of a problem if you're only doing a domain save.


Yes, but we have to instantiate the server so old version qemu can work.
Can we add a new interface to just get the ioreq pfn, buffer ioreq 
pfn/evtchn?

For example HVM_PARAM_DEFAULT_IOREQ_PFN.

Thanks
Wen congyang



   Paul



Thanks
Wen Congyang






2. Why we create a default ioreq server when getting the hvm

param if

there

is already a
 not default ioreq server?


If something reads the 'legacy' HVM params then that is Xen's trigger

to

create the default server. Any 'new' emulator should be using the ioreq
server hypercalls so the default server will not be needed.

If there are two ioreq servers: default ioreq server, and a ioreq server
created by emulator. The guest can work it correctly in
this case?


Yo

Re: [Xen-devel] [PATCH v2 COLOPre 09/13] tools/libxl: Update libxl_save_msgs_gen.pl to support return data from xl to xc

[Wen Congyang]

On 06/16/2015 06:49 PM, Ian Campbell wrote:
> On Mon, 2015-06-08 at 11:43 +0800, Yang Hongyang wrote:
>> From: Wen Congyang 
>>
>>  Currently, all callbacks return an integer value or void. We cannot
>>  return some data to xc via callback. Update libxl_save_msgs_gen.pl
>>  to support this case.
> 
> (Odd two space indent from somewhere)
> 
> Can you be more specific about what kind of data you need to pass back
> here? Is it an opaque "void *" blob or is it potentially more
> structured? That is useful to know in case there might be a better way
> to achieve the goal.

The secondary vm's dirty bitmap.

Thanks
Wen Congyang

> 
> Ian.
> 
> .
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 20/27] tools/libxl: Infrastructure for writing a v2 stream

[Wen Congyang]

k_buf,
> +&toolstack_len, dss);
> +if (ret)
> +goto err;
> +
> +dc->copywhat = "toolstack record";
> +dc->writewhat = "save/migration stream";
> +dc->callback = toolstack_record_done;
> +
> +ret = libxl__datacopier_start(dc);
> +if (ret)
> +goto err;
> +
> +rec.length = toolstack_len;
> +
> +libxl__datacopier_prefixdata(egc, dc, &rec, sizeof(rec));
> +libxl__datacopier_prefixdata(egc, dc, toolstack_buf, toolstack_len);
> +
> +padding_len = ROUNDUP(rec.length, REC_ALIGN_ORDER) - rec.length;
> +if (padding_len)
> +libxl__datacopier_prefixdata(egc, dc, zero_padding, padding_len);
> +
> +free(toolstack_buf);
> +return;
> +
> + err:
> +assert(ret);
> +free(toolstack_buf);
> +stream_failed(egc, stream, ret);
> +}
> +
> +static void toolstack_record_done(libxl__egc *egc,
> +  libxl__datacopier_state *dc,
> +  int onwrite, int errnoval)
> +{
> +libxl__stream_write_state *stream = CONTAINER_OF(dc, *stream, dc);
> +libxl__domain_suspend_state *dss = CONTAINER_OF(stream, *dss, sws);
> +STATE_AO_GC(stream->ao);
> +int ret = 0;
> +
> +    if (onwrite || errnoval) {
> +ret = ERROR_FAIL;
> +goto err;
> +}
> +
> +if (dss->type == LIBXL_DOMAIN_TYPE_HVM)
> +write_emulator_record(egc, stream);
> +else
> +write_end_record(egc, stream);
> +
> +return;
> +
> + err:
> +assert(ret);
> +stream_failed(egc, stream, ret);
> +}
> +
> +static void write_emulator_record(libxl__egc *egc,
> +  libxl__stream_write_state *stream)
> +{
> +libxl__domain_suspend_state *dss = CONTAINER_OF(stream, *dss, sws);
> +libxl__datacopier_state *dc = &stream->dc;
> +STATE_AO_GC(stream->ao);
> +struct libxl_sr_rec_hdr rec = { REC_TYPE_EMULATOR_CONTEXT, 0 };
> +struct libxl_sr_emulator_hdr ehdr = { 0 };
> +struct stat st;
> +int ret = 0;
> +uint32_t qemu_state_len;
> +
> +assert(dss->type == LIBXL_DOMAIN_TYPE_HVM);
> +
> +/* Convenience aliases */
> +const char *const filename = dss->dm_savefile;
> +const uint32_t domid = dss->domid;
> +
> +switch(libxl__device_model_version_running(gc, domid)) {
> +case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL:
> +ehdr.id = EMULATOR_QEMU_TRADITIONAL;
> +break;
> +
> +case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN:
> +ehdr.id = EMULATOR_QEMU_UPSTREAM;
> +break;
> +
> +default:
> +ret = ERROR_FAIL;
> +goto err;
> +}
> +
> +ret = libxl__domain_suspend_device_model(gc, dss);

We have called it when suspending the guest. For migration, calling
this function twice is OK, but for remus, it will cause a problem:
the guest is running, and this device-model will be stopped by this
function.

Thanks
Wen Congyang

> +if (ret)
> +goto err;
> +
> +dc->readwhat = GCSPRINTF("qemu save file %s", filename);
> +dc->copywhat = "emulator record";
> +dc->writewhat = "save/migration stream";
> +dc->callback = emulator_body_done;
> +
> +dc->readfd = open(filename, O_RDONLY);
> +if (dc->readfd < 0) {
> +LOGE(ERROR, "unable to open %s", dc->readwhat);
> +goto err;
> +}
> +
> +if (fstat(dc->readfd, &st))
> +{
> +LOGE(ERROR, "unable to fstat %s", dc->readwhat);
> +goto err;
> +}
> +
> +if (!S_ISREG(st.st_mode)) {
> +LOG(ERROR, "%s is not a plain file!", dc->readwhat);
> +goto err;
> +}
> +
> +qemu_state_len = st.st_size;
> +rec.length = qemu_state_len + sizeof(ehdr);
> +
> +ret = libxl__datacopier_start(dc);
> +if (ret)
> +goto err;
> +
> +libxl__datacopier_prefixdata(egc, dc, &rec, sizeof(rec));
> +libxl__datacopier_prefixdata(egc, dc, &ehdr, sizeof(ehdr));
> +
> +stream->padding = ROUNDUP(qemu_state_len, REC_ALIGN_ORDER) - 
> qemu_state_len;
> +return;
> +
> + err:
> +assert(ret);
> +stream_failed(egc, stream, ret);
> +}
> +
> +static void emulator_body_done(libxl__egc *egc,
> +   libxl__datacopier_state *dc,
> +   int onwrite, int errnoval)
> +{
> +libxl__stream_write_state *stream = CONTAINER_OF(dc, *stream, dc);
> +STATE_AO_GC(stream->ao);
> +int r

Re: [Xen-devel] [PATCH 00/27] Libxl migration v2

[Wen Congyang]

On 06/15/2015 09:44 PM, Andrew Cooper wrote:
> This series adds support for the libxl migration v2 stream, and untangles the
> existing layering violations of the toolstack and qemu records.
> 
> At the end of the series, legacy migration is no longer used.
> 
> Note: Remus support is broken and (RFC) fixed in separate patches in this
> series.  It was too tangled to fix in a bisectable fashon.  Plain
> suspend/migrate/resume however is (should be) bisectable along the entire
> series.
> 
> There are a couple of outstanding questions:
> 
> 1) What to do about the toolstack/xenstore record.  It is currently by being
>passed around as a blob, but it might be better to split it out.
> 
> 2) What (if any) ABI/API qualifications are needed? (Particularly in reference
>to patch 21)
> 
> The Remus code is untested by me, but is hopefully in the correct ballpark.
> All other combinations of suspend/migrate/resume have been tested with PV and
> HVM guests (qemu-trad and qemu-upstream), including 32 -> 64 bit migration
> (which was the underlying bug causing us to write migration v2 in the first
> place).
> 
> There are some further improvements which could be made.  In particular, it
> appears that sending the toolstack record on each checkpoint is redundant, and
> there is certainly room for some more pruning of the legacy migration code.

Do you mean: libxl__toolstack_save is harmless, and it can be called when the
guest is running?

Thanks
Wen Congyang

> 
> Anyway, thoughts/comments welcome.  Please test!
> 
> ~Andrew
> 
> 
> Andrew Cooper (22):
>   tools/libxl: Fix libxl__ev_child_inuse() check for not-yet-initialised 
> children
>   tools/libxc: Always compile the compat qemu variables into xc_sr_context
>   tools/libxl: Stash all restore parameters in domain_create_state
>   tools/xl: Mandatory flag indicating the format of the migration stream
>   tools/libxl: Introduce ROUNDUP()
>   tools/libxl: Extra APIs for the save helper
>   tools/libxl: Pass restore_fd as a parameter to libxl__xc_domain_restore()
>   docs: Libxl migration v2 stream specification
>   tools/python: Libxc migration v2 infrastructure
>   tools/python: Libxl migration v2 infrastructure
>   tools/python: Verification utility for v2 stream spec compliance
>   tools/python: Conversion utility for legacy migration streams
>   tools/libxl: Support converting a legacy stream to a v2 stream
>   tools/libxl: Convert a legacy stream if needed
>   tools/libxc+libxl+xl: Restore v2 streams
>   tools/libxc+libxl+xl: Save v2 streams
>   docs/libxl: [RFC] Introduce CHECKPOINT_END to support migration v2 remus 
> streams
>   tools/libxl: [RFC] Write checkpoint records into the stream
>   tools/libx{c,l}: [RFC] Introduce restore_callbacks.checkpoint()
>   tools/libxl: [RFC] Handle checkpoint records in a libxl migration v2 stream
>   tools/libxc: Drop all XG_LIBXL_HVM_COMPAT code from libxc
>   tools/libxl: Drop all knowledge of toolstack callbacks
> 
> Ian Jackson (2):
>   libxl: cancellation: Preparations for save/restore cancellation
>   libxl: cancellation: Handle SIGTERM in save/restore helper
> 
> Ross Lagerwall (3):
>   tools/libxl: Migration v2 stream format
>   tools/libxl: Infrastructure for reading a libxl migration v2 stream
>   tools/libxl: Infrastructure for writing a v2 stream
> 
>  docs/specs/libxl-migration-stream.pandoc  |  218 
>  tools/libxc/Makefile  |2 -
>  tools/libxc/include/xenguest.h|3 +
>  tools/libxc/xc_sr_common.h|5 -
>  tools/libxc/xc_sr_restore.c   |   33 +-
>  tools/libxc/xc_sr_restore_x86_hvm.c   |  124 -
>  tools/libxc/xc_sr_save_x86_hvm.c  |   36 --
>  tools/libxl/Makefile  |2 +
>  tools/libxl/libxl_aoutils.c   |7 +
>  tools/libxl/libxl_convert_callout.c   |  146 ++
>  tools/libxl/libxl_create.c|   80 +--
>  tools/libxl/libxl_dom.c   |   61 +--
>  tools/libxl/libxl_internal.h  |  140 -
>  tools/libxl/libxl_save_callout.c  |   63 +--
>  tools/libxl/libxl_save_helper.c   |   95 ++--
>  tools/libxl/libxl_save_msgs_gen.pl|9 +-
>  tools/libxl/libxl_sr_stream_format.h  |   58 +++
>  tools/libxl/libxl_stream_read.c   |  663 
>  tools/libxl/libxl_stream_write.c  |  640 +++
>  tools/libxl/libxl_types.idl   |2 +
>  tools/libxl/xl_cmdimpl.c  |9 +-
>  tools/python/Makefile |4 +
>  tools/python/scripts/convert-legacy-str

Re: [Xen-devel] [PATCH 20/27] tools/libxl: Infrastructure for writing a v2 stream

[Wen Congyang]

k_buf,
> +&toolstack_len, dss);
> +if (ret)
> +goto err;
> +
> +dc->copywhat = "toolstack record";
> +dc->writewhat = "save/migration stream";
> +dc->callback = toolstack_record_done;
> +
> +ret = libxl__datacopier_start(dc);
> +if (ret)
> +goto err;
> +
> +rec.length = toolstack_len;
> +
> +libxl__datacopier_prefixdata(egc, dc, &rec, sizeof(rec));
> +libxl__datacopier_prefixdata(egc, dc, toolstack_buf, toolstack_len);
> +
> +padding_len = ROUNDUP(rec.length, REC_ALIGN_ORDER) - rec.length;
> +if (padding_len)
> +libxl__datacopier_prefixdata(egc, dc, zero_padding, padding_len);
> +
> +free(toolstack_buf);
> +return;
> +
> + err:
> +assert(ret);
> +free(toolstack_buf);
> +stream_failed(egc, stream, ret);
> +}
> +
> +static void toolstack_record_done(libxl__egc *egc,
> +  libxl__datacopier_state *dc,
> +  int onwrite, int errnoval)
> +{
> +libxl__stream_write_state *stream = CONTAINER_OF(dc, *stream, dc);
> +libxl__domain_suspend_state *dss = CONTAINER_OF(stream, *dss, sws);
> +STATE_AO_GC(stream->ao);
> +int ret = 0;
> +
> +if (onwrite || errnoval) {
> +ret = ERROR_FAIL;
> +goto err;
> +}
> +
> +if (dss->type == LIBXL_DOMAIN_TYPE_HVM)
> +write_emulator_record(egc, stream);
> +else
> +write_end_record(egc, stream);
> +
> +return;
> +
> + err:
> +assert(ret);
> +stream_failed(egc, stream, ret);
> +}
> +
> +static void write_emulator_record(libxl__egc *egc,
> +  libxl__stream_write_state *stream)
> +{
> +libxl__domain_suspend_state *dss = CONTAINER_OF(stream, *dss, sws);
> +libxl__datacopier_state *dc = &stream->dc;
> +STATE_AO_GC(stream->ao);
> +struct libxl_sr_rec_hdr rec = { REC_TYPE_EMULATOR_CONTEXT, 0 };
> +struct libxl_sr_emulator_hdr ehdr = { 0 };
> +struct stat st;
> +int ret = 0;
> +uint32_t qemu_state_len;
> +
> +assert(dss->type == LIBXL_DOMAIN_TYPE_HVM);
> +
> +/* Convenience aliases */
> +const char *const filename = dss->dm_savefile;
> +const uint32_t domid = dss->domid;
> +
> +switch(libxl__device_model_version_running(gc, domid)) {
> +case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL:
> +ehdr.id = EMULATOR_QEMU_TRADITIONAL;
> +break;
> +
> +case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN:
> +ehdr.id = EMULATOR_QEMU_UPSTREAM;
> +break;
> +
> +default:
> +ret = ERROR_FAIL;
> +goto err;
> +}
> +
> +ret = libxl__domain_suspend_device_model(gc, dss);
> +if (ret)
> +goto err;
> +
> +dc->readwhat = GCSPRINTF("qemu save file %s", filename);
> +dc->copywhat = "emulator record";
> +dc->writewhat = "save/migration stream";
> +dc->callback = emulator_body_done;
> +
> +dc->readfd = open(filename, O_RDONLY);
> +if (dc->readfd < 0) {
> +LOGE(ERROR, "unable to open %s", dc->readwhat);
> +goto err;
> +}
> +
> +if (fstat(dc->readfd, &st))
> +{
> +LOGE(ERROR, "unable to fstat %s", dc->readwhat);
> +goto err;
> +}
> +
> +if (!S_ISREG(st.st_mode)) {
> +LOG(ERROR, "%s is not a plain file!", dc->readwhat);
> +goto err;
> +}
> +
> +qemu_state_len = st.st_size;
> +rec.length = qemu_state_len + sizeof(ehdr);
> +
> +ret = libxl__datacopier_start(dc);
> +if (ret)
> +goto err;
> +
> +libxl__datacopier_prefixdata(egc, dc, &rec, sizeof(rec));
> +libxl__datacopier_prefixdata(egc, dc, &ehdr, sizeof(ehdr));
> +
> +stream->padding = ROUNDUP(qemu_state_len, REC_ALIGN_ORDER) - 
> qemu_state_len;
> +return;
> +
> + err:
> +assert(ret);
> +stream_failed(egc, stream, ret);
> +}
> +
> +static void emulator_body_done(libxl__egc *egc,
> +   libxl__datacopier_state *dc,
> +   int onwrite, int errnoval)
> +{
> +libxl__stream_write_state *stream = CONTAINER_OF(dc, *stream, dc);
> +STATE_AO_GC(stream->ao);
> +int ret = 0;

I think you forget to close dc->readfd here.

Thanks
Wen Congyang

> +
> +if (onwrite || errnoval) {
> +ret = ERROR_FAIL;
> +goto err;
> +}
> +
> +dc->readwhat = ""

Re: [Xen-devel] [PATCH 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream

[Wen Congyang]

eam);
> +}
> +}
> +
> +static void stream_done(libxl__egc *egc,
> +libxl__stream_read_state *stream)
> +{
> +libxl__domain_create_state *dcs = CONTAINER_OF(stream, *dcs, srs);
> +
> +assert(!stream->running);
> +
> +stream->completion_callback(egc, dcs, stream->rc);
> +}
> +
> +static void stream_header_done(libxl__egc *egc,
> +   libxl__datacopier_state *dc,
> +   int onwrite, int errnoval)
> +{
> +libxl__stream_read_state *stream = CONTAINER_OF(dc, *stream, dc);
> +libxl_sr_hdr *hdr = &stream->hdr;
> +STATE_AO_GC(dc->ao);
> +int ret = 0;
> +
> +if (onwrite || dc->used != stream->expected_len) {
> +ret = ERROR_FAIL;
> +LOG(ERROR, "write %d, err %d, expected %zu, got %zu",
> +onwrite, errnoval, stream->expected_len, dc->used);
> +goto err;
> +}
> +
> +hdr->ident   = be64toh(hdr->ident);
> +hdr->version = be32toh(hdr->version);
> +hdr->options = be32toh(hdr->options);
> +
> +if (hdr->ident != RESTORE_STREAM_IDENT) {
> +ret = ERROR_FAIL;
> +LOG(ERROR,
> +"Invalid ident: expected 0x%016"PRIx64", got 0x%016"PRIx64,
> +RESTORE_STREAM_IDENT, hdr->ident);
> +goto err;
> +}
> +if (hdr->version != RESTORE_STREAM_VERSION) {
> +ret = ERROR_FAIL;
> +LOG(ERROR, "Unexpected Version: expected %u, got %u",
> +RESTORE_STREAM_VERSION, hdr->version);
> +goto err;
> +}
> +if (hdr->options & RESTORE_OPT_BIG_ENDIAN) {
> +ret = ERROR_FAIL;
> +LOG(ERROR, "Unable to handle big endian streams");
> +goto err;

I think it is better to check if the host is big endian or not.
The source and target should be the same.

Thanks
Wen Congyang

> +}
> +
> +LOG(INFO, "Stream v%u%s", hdr->version,
> +hdr->options & RESTORE_OPT_LEGACY ? " (from legacy)" : "");
> +
> +libxl__stream_read_continue(egc, stream);
> +return;
> +
> + err:
> +assert(ret);
> +stream_failed(egc, stream, ret);
> +}
> +
> +static void record_header_done(libxl__egc *egc,
> +   libxl__datacopier_state *dc,
> +   int onwrite, int errnoval)
> +{
> +libxl__stream_read_state *stream = CONTAINER_OF(dc, *stream, dc);
> +libxl_sr_rec_hdr *rec_hdr = &stream->rec_hdr;
> +STATE_AO_GC(dc->ao);
> +int ret = 0;
> +
> +if (onwrite || dc->used != stream->expected_len) {
> +ret = ERROR_FAIL;
> +LOG(ERROR, "write %d, err %d, expected %zu, got %zu",
> +onwrite, errnoval, stream->expected_len, dc->used);
> +goto err;
> +}
> +
> +assert(stream->rec_body == NULL);
> +
> +/* No body? Process straight away. */
> +if (rec_hdr->length == 0) {
> +process_record(egc, stream);
> +return;
> +}
> +
> +/* Queue up reading the body. */
> +size_t bytes_to_read;
> +
> +switch (rec_hdr->type) {
> +/*
> + * Emulator records want to retain the blob in the pipe, for a 
> further
> + * datacopier call to move elsewhere.  Just read the emulator header.
> + */
> +case REC_TYPE_EMULATOR_CONTEXT:
> +bytes_to_read = sizeof(struct libxl_sr_emulator_hdr);
> +break;
> +
> +default:
> +bytes_to_read = rec_hdr->length;
> +break;
> +}
> +
> +bytes_to_read = ROUNDUP(bytes_to_read, REC_ALIGN_ORDER);
> +
> +dc->readwhat = "record body";
> +stream->rec_body = dc->readbuf = libxl__malloc(NOGC, bytes_to_read);
> +stream->expected_len = dc->bytes_to_read = bytes_to_read;
> +dc->used = 0;
> +dc->callback = record_body_done;
> +
> +ret = libxl__datacopier_start(dc);
> +if (ret)
> +goto err;
> +return;
> +
> + err:
> +assert(ret);
> +stream_failed(egc, stream, ret);
> +}
> +
> +static void record_body_done(libxl__egc *egc,
> + libxl__datacopier_state *dc,
> + int onwrite, int errnoval)
> +{
> +libxl__stream_read_state *stream = CONTAINER_OF(dc, *stream, dc);
> +STATE_AO_GC(dc->ao);
> +int ret = 0;
> +
> +if (onwrite || dc->used != stream->expected_len) {
> +ret = ERROR_FAIL;
> +LOG(ERROR, "write %d, err %d, expect

Re: [Xen-devel] [PATCH 22/27] docs/libxl: [RFC] Introduce CHECKPOINT_END to support migration v2 remus streams

[Wen Congyang]

On 06/15/2015 09:44 PM, Andrew Cooper wrote:
> In a remus senario, libxc will write a CHECKPOINT record, then hand ownership
> of the fd to libxl.  Libxl then writes any records required and finishes with
> a CHECKPOINT_END record, then hands ownership of the fd back to libxc.
> 
> Signed-off-by: Andrew Cooper 
> CC: Ian Campbell 
> CC: Ian Jackson 
> CC: Wei Liu 
> ---
>  docs/specs/libxl-migration-stream.pandoc |   15 ++-
>  tools/libxl/libxl_sr_stream_format.h |1 +
>  tools/python/xen/migration/libxl.py  |   11 +++
>  3 files changed, 26 insertions(+), 1 deletion(-)
> 
> diff --git a/docs/specs/libxl-migration-stream.pandoc 
> b/docs/specs/libxl-migration-stream.pandoc
> index 7235317..d41932a 100644
> --- a/docs/specs/libxl-migration-stream.pandoc
> +++ b/docs/specs/libxl-migration-stream.pandoc
> @@ -119,7 +119,9 @@ type 0x: END
>  
>   0x0003: EMULATOR_CONTEXT
>  
> - 0x0004 - 0x7FFF: Reserved for future _mandatory_
> + 0x0004: CHECKPOINT_END
> +
> + 0x0005 - 0x7FFF: Reserved for future _mandatory_
>   records.
>  
>   0x8000 - 0x: Reserved for future _optional_
> @@ -203,3 +205,14 @@ indexIndex of this emulator for the domain, 
> if multiple
>  
>  emulator_ctx Emulator context blob.
>  
> +
> +CHECKPOINT_END

CHECKPOINT\_END?

> +--
> +
> +A checkpoint end record marks the end of a checkpoint in the image.
> +
> + 0 1 2 3 4 5 6 7 octet
> ++-+
> +
> +The end record contains no fields; its body_length is 0.
> +
> diff --git a/tools/libxl/libxl_sr_stream_format.h 
> b/tools/libxl/libxl_sr_stream_format.h
> index 487f9e2..5dfa55f 100644
> --- a/tools/libxl/libxl_sr_stream_format.h
> +++ b/tools/libxl/libxl_sr_stream_format.h
> @@ -35,6 +35,7 @@
>  #define REC_TYPE_LIBXC_CONTEXT   0x0001U
>  #define REC_TYPE_XENSTORE_DATA   0x0002U
>  #define REC_TYPE_EMULATOR_CONTEXT0x0003U
> +#define REC_TYPE_CHECKPOINT_END  0x0004U
>  
>  typedef struct libxl_sr_emulator_hdr
>  {
> diff --git a/tools/python/xen/migration/libxl.py 
> b/tools/python/xen/migration/libxl.py
> index 4e1f4f8..415502e 100644
> --- a/tools/python/xen/migration/libxl.py
> +++ b/tools/python/xen/migration/libxl.py
> @@ -36,12 +36,14 @@ REC_TYPE_end  = 0x
>  REC_TYPE_libxc_context= 0x0001
>  REC_TYPE_xenstore_data= 0x0002
>  REC_TYPE_emulator_context = 0x0003
> +REC_TYPE_checkpoint_end   = 0x0004
>  
>  rec_type_to_str = {
>  REC_TYPE_end  : "End",
>  REC_TYPE_libxc_context: "Libxc context",
>  REC_TYPE_xenstore_data: "Xenstore data",
>  REC_TYPE_emulator_context : "Emulator context",
> +REC_TYPE_checkpoint_end   : "Checkpoint end",
>  }
>  
>  # emulator_context
> @@ -176,6 +178,13 @@ class VerifyLibxl(VerifyBase):
>  self.info("  Index %d, type %s" % (emu_idx, 
> emulator_id_to_str[emu_id]))
>  
>  
> +def verify_record_checkpoint_end(self, content):
> +""" Checkpoint end record """
> +
> +if len(content) != 0:
> +raise RecordError("Checkpoint end record with non-zero length")
> +
> +
>  record_verifiers = {
>  REC_TYPE_end:
>  VerifyLibxl.verify_record_end,
> @@ -185,4 +194,6 @@ record_verifiers = {
>  VerifyLibxl.verify_record_xenstore_data,
>  REC_TYPE_emulator_context:
>  VerifyLibxl.verify_record_emulator_context,
> +REC_TYPE_checkpoint_end:
> +VerifyLibxl.verify_record_checkpoint_end,
>  }
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream

[Wen Congyang]

l__stream_read_start()
> + * - Set up reading a stream from the start.
> + *
> + *  - libxl__stream_read_continue()
> + * - Set up reading the next record from a started stream.
> + *
> + * The principle loop functionality involves reading the stream header, then
> + * reading a record at time and acting upon it.  It follows the callbacks:
> + *
> + *  - stream_header_done()
> + *  - stream_record_header_done()
> + *  - stream_record_body_done()
> + *  - process_record()
> + *
> + * process_record() will choose the correct next action based upon the
> + * record.  Upon completion of the action, the next record header will be 
> read
> + * from the stream.
> + */
> +
> +static void stream_success(libxl__egc *egc,
> +   libxl__stream_read_state *stream);
> +static void stream_failed(libxl__egc *egc,
> +  libxl__stream_read_state *stream, int rc);
> +static void stream_done(libxl__egc *egc,
> +libxl__stream_read_state *stream);
> +
> +/* Event callbacks for main reading loop. */
> +static void stream_header_done(libxl__egc *egc,
> +   libxl__datacopier_state *dc,
> +   int onwrite, int errnoval);
> +static void record_header_done(libxl__egc *egc,
> +   libxl__datacopier_state *dc,
> +   int onwrite, int errnoval);
> +static void record_body_done(libxl__egc *egc,
> + libxl__datacopier_state *dc,
> + int onwrite, int errnoval);
> +static void process_record(libxl__egc *egc,
> +   libxl__stream_read_state *stream);
> +
> +/* Mini-event loop for splicing a emulator record out of the stream. */
> +static void read_emulator_body(libxl__egc *egc,
> +   libxl__stream_read_state *stream);
> +static void emulator_body_done(libxl__egc *egc,
> +   libxl__datacopier_state *dc,
> +   int onwrite, int errnoval);
> +static void emulator_padding_done(libxl__egc *egc,
> +  libxl__datacopier_state *dc,
> +  int onwrite, int errnoval);
> +
> +void libxl__stream_read_start(libxl__egc *egc,
> +  libxl__stream_read_state *stream)
> +{
> +libxl__datacopier_state *dc = &stream->dc;
> +int ret = 0;
> +
> +/* State initialisation. */
> +assert(!stream->running);
> +
> +memset(dc, 0, sizeof(*dc));
> +dc->ao = stream->ao;
> +dc->readfd = stream->fd;
> +dc->writefd = -1;
> +
> +/* Start reading the stream header. */
> +dc->readwhat = "stream header";
> +dc->readbuf = &stream->hdr;
> +stream->expected_len = dc->bytes_to_read = sizeof(stream->hdr);
> +dc->used = 0;
> +dc->callback = stream_header_done;
> +
> +ret = libxl__datacopier_start(dc);
> +if (ret)
> +goto err;
> +
> +stream->running = true;
> +assert(!ret);
> +return;
> +
> + err:
> +assert(ret);
> +stream_failed(egc, stream, ret);
> +}
> +
> +void libxl__stream_read_continue(libxl__egc *egc,
> + libxl__stream_read_state *stream)
> +{
> +libxl__datacopier_state *dc = &stream->dc;
> +int ret = 0;
> +
> +assert(stream->running);
> +
> +/* Read a record header. */
> +dc->readwhat = "record header";
> +dc->readbuf = &stream->rec_hdr;
> +stream->expected_len = dc->bytes_to_read = sizeof(stream->rec_hdr);
> +dc->used = 0;
> +dc->callback = record_header_done;
> +
> +ret = libxl__datacopier_start(dc);
> +if (ret)
> +goto err;
> +
> +assert(!ret);
> +return;
> +
> + err:
> +assert(ret);
> +stream_failed(egc, stream, ret);
> +}
> +
> +void libxl__stream_read_abort(libxl__egc *egc,
> +  libxl__stream_read_state *stream, int rc)
> +{
> +stream_failed(egc, stream, rc);
> +}
> +
> +static void stream_success(libxl__egc *egc, libxl__stream_read_state *stream)
> +{
> +stream->rc = 0;
> +stream->running = false;
> +
> +stream_done(egc, stream);
> +}
> +
> +static void stream_failed(libxl__egc *egc,
> +  libxl__stream_read_state *stream, int rc)
> +{
> +assert(rc);
> +stream->rc = rc;

I have a question: rc is always less than 0?

Thanks
Wen Congyang

> +
> +if (stream->running) {
> +

Re: [Xen-devel] [PATCH 25/27] tools/libxl: [RFC] Handle checkpoint records in a libxl migration v2 stream

[Wen Congyang]

  libxl__stream_read_state *stream)
>  {
> @@ -162,6 +166,35 @@ void libxl__stream_read_continue(libxl__egc *egc,
>  stream_failed(egc, stream, ret);
>  }
>  
> +void libxl__stream_read_start_checkpoint(libxl__egc *egc,
> + libxl__stream_read_state *stream)
> +{
> +libxl__datacopier_state *dc = &stream->dc;
> +int ret = 0;
> +
> +assert(stream->running);
> +assert(!stream->in_checkpoint);
> +stream->in_checkpoint = true;
> +

I think you can call libxl__stream_read_continue() here.

Thanks
Wen Congyang

> +/* Read a record header. */
> +dc->readwhat = "record header";
> +dc->readbuf = &stream->rec_hdr;
> +stream->expected_len = dc->bytes_to_read = sizeof(stream->rec_hdr);
> +dc->used = 0;
> +dc->callback = record_header_done;
> +
> +ret = libxl__datacopier_start(dc);
> +if (ret)
> +goto err;
> +
> +assert(!ret);
> +return;
> +
> + err:
> +assert(ret);
> +stream_failed(egc, stream, ret);
> +}
> +
>  void libxl__stream_read_abort(libxl__egc *egc,
>libxl__stream_read_state *stream, int rc)
>  {
> @@ -182,6 +215,15 @@ static void stream_failed(libxl__egc *egc,
>  assert(rc);
>  stream->rc = rc;
>  
> +/*
> + *If we are in a checkpoint, pass the failure to libxc, which will come
> + * back around to us via libxl__xc_domain_restore_done().
> + */
> +if (stream->in_checkpoint) {
> +checkpoint_done(egc, stream, rc);
> +return;
> +}
> +
>  if (stream->running) {
>  stream->running = false;
>  stream_done(egc, stream);
> @@ -194,6 +236,7 @@ static void stream_done(libxl__egc *egc,
>  libxl__domain_create_state *dcs = CONTAINER_OF(stream, *dcs, srs);
>  
>  assert(!stream->running);
> +assert(!stream->in_checkpoint);
>  
>  if (stream->v2_carefd)
>  libxl__carefd_close(stream->v2_carefd);
> @@ -452,6 +495,15 @@ static void process_record(libxl__egc *egc,
>  read_emulator_body(egc, stream);
>  break;
>  
> +case REC_TYPE_CHECKPOINT_END:
> +if (!stream->in_checkpoint) {
> +LOG(ERROR, "Unexpected CHECKPOINT_END record in stream");
> +ret = ERROR_FAIL;
> +goto err;
> +}
> +checkpoint_done(egc, stream, 0);
> +break;
> +
>  default:
>  LOG(ERROR, "Unrecognised record 0x%08x", rec_hdr->type);
>  ret = ERROR_FAIL;
> @@ -592,6 +644,16 @@ static void emulator_padding_done(libxl__egc *egc,
>  stream_failed(egc, stream, ret);
>  }
>  
> +static void checkpoint_done(libxl__egc *egc,
> +libxl__stream_read_state *stream, int rc)
> +{
> +libxl__domain_create_state *dcs = CONTAINER_OF(stream, *dcs, srs);
> +
> +assert(stream->in_checkpoint);
> +stream->in_checkpoint = false;
> +stream->checkpoint_callback(egc, dcs, rc);
> +}
> +
>  /*
>   * Local variables:
>   * mode: C
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream

[Wen Congyang]

eam);
> +}
> +}
> +
> +static void stream_done(libxl__egc *egc,
> +libxl__stream_read_state *stream)
> +{
> +libxl__domain_create_state *dcs = CONTAINER_OF(stream, *dcs, srs);
> +
> +assert(!stream->running);
> +
> +stream->completion_callback(egc, dcs, stream->rc);
> +}
> +
> +static void stream_header_done(libxl__egc *egc,
> +   libxl__datacopier_state *dc,
> +   int onwrite, int errnoval)
> +{
> +libxl__stream_read_state *stream = CONTAINER_OF(dc, *stream, dc);
> +libxl_sr_hdr *hdr = &stream->hdr;
> +STATE_AO_GC(dc->ao);
> +int ret = 0;
> +
> +if (onwrite || dc->used != stream->expected_len) {
> +ret = ERROR_FAIL;
> +LOG(ERROR, "write %d, err %d, expected %zu, got %zu",
> +onwrite, errnoval, stream->expected_len, dc->used);
> +goto err;
> +}
> +
> +hdr->ident   = be64toh(hdr->ident);
> +hdr->version = be32toh(hdr->version);
> +hdr->options = be32toh(hdr->options);
> +
> +if (hdr->ident != RESTORE_STREAM_IDENT) {
> +ret = ERROR_FAIL;
> +LOG(ERROR,
> +"Invalid ident: expected 0x%016"PRIx64", got 0x%016"PRIx64,
> +RESTORE_STREAM_IDENT, hdr->ident);
> +goto err;
> +}
> +if (hdr->version != RESTORE_STREAM_VERSION) {
> +ret = ERROR_FAIL;
> +LOG(ERROR, "Unexpected Version: expected %u, got %u",
> +RESTORE_STREAM_VERSION, hdr->version);
> +goto err;
> +}
> +if (hdr->options & RESTORE_OPT_BIG_ENDIAN) {
> +ret = ERROR_FAIL;
> +LOG(ERROR, "Unable to handle big endian streams");
> +goto err;
> +}
> +
> +LOG(INFO, "Stream v%u%s", hdr->version,
> +hdr->options & RESTORE_OPT_LEGACY ? " (from legacy)" : "");
> +
> +libxl__stream_read_continue(egc, stream);
> +return;
> +
> + err:
> +assert(ret);
> +stream_failed(egc, stream, ret);
> +}
> +
> +static void record_header_done(libxl__egc *egc,
> +   libxl__datacopier_state *dc,
> +   int onwrite, int errnoval)
> +{
> +libxl__stream_read_state *stream = CONTAINER_OF(dc, *stream, dc);
> +libxl_sr_rec_hdr *rec_hdr = &stream->rec_hdr;
> +STATE_AO_GC(dc->ao);
> +int ret = 0;
> +
> +if (onwrite || dc->used != stream->expected_len) {
> +ret = ERROR_FAIL;
> +LOG(ERROR, "write %d, err %d, expected %zu, got %zu",
> +onwrite, errnoval, stream->expected_len, dc->used);
> +goto err;
> +}
> +
> +assert(stream->rec_body == NULL);
> +
> +/* No body? Process straight away. */
> +if (rec_hdr->length == 0) {
> +process_record(egc, stream);
> +return;
> +}
> +
> +/* Queue up reading the body. */
> +size_t bytes_to_read;
> +
> +switch (rec_hdr->type) {
> +/*
> + * Emulator records want to retain the blob in the pipe, for a 
> further
> + * datacopier call to move elsewhere.  Just read the emulator header.
> + */

In this case, we should not call ROUNDUP().

> +case REC_TYPE_EMULATOR_CONTEXT:
> +bytes_to_read = sizeof(struct libxl_sr_emulator_hdr);
> +break;
> +
> +default:
> +bytes_to_read = rec_hdr->length;
> +break;
> +}
> +
> +bytes_to_read = ROUNDUP(bytes_to_read, REC_ALIGN_ORDER);

So, I think it is better to move ROUNDUP to default case.

Thanks
Wen Congyang

> +
> +dc->readwhat = "record body";
> +stream->rec_body = dc->readbuf = libxl__malloc(NOGC, bytes_to_read);
> +stream->expected_len = dc->bytes_to_read = bytes_to_read;
> +dc->used = 0;
> +dc->callback = record_body_done;
> +
> +ret = libxl__datacopier_start(dc);
> +if (ret)
> +goto err;
> +return;
> +
> + err:
> +assert(ret);
> +stream_failed(egc, stream, ret);
> +}
> +
> +static void record_body_done(libxl__egc *egc,
> + libxl__datacopier_state *dc,
> + int onwrite, int errnoval)
> +{
> +libxl__stream_read_state *stream = CONTAINER_OF(dc, *stream, dc);
> +STATE_AO_GC(dc->ao);
> +int ret = 0;
> +
> +if (onwrite || dc->used != stream->expected_len) {
> +ret = ERROR_FAIL;
> +LOG(ERROR, "write %d, err %d, expect

[Xen-devel] building error

[Wen Congyang]

When I try to build xen on fc21, I meet the following errors:
drivers/net/ath/ath9k/ath9k_ar9003_phy.c: In function ‘ar9003_hw_ani_control’:
drivers/net/ath/ath9k/ath9k_ar9003_phy.c:862:11: error: logical not is only 
applied to the left hand side of comparison [-Werror=logical-not-parentheses]
   if (!on != aniState->ofdmWeakSigDetectOff) {
   ^
drivers/net/ath/ath9k/ath9k_ar9003_phy.c:1016:14: error: logical not is only 
applied to the left hand side of comparison [-Werror=logical-not-parentheses]
   if (!is_on != aniState->mrcCCKOff) {
  ^
cc1: all warnings being treated as errors
bin/rules/drivers/net/ath/ath9k/ath9k_ar9003_phy.c.r:3: recipe for target 
'bin/ath9k_ar9003_phy.o' failed
make[7]: *** [bin/ath9k_ar9003_phy.o] Error 1
make[7]: *** Waiting for unfinished jobs
  [BUILD] bin/nvsvpd.o
drivers/net/ath/ath9k/ath9k_ar5008_phy.c: In function 
‘ar5008_hw_ani_control_old’:
drivers/net/ath/ath9k/ath9k_ar5008_phy.c:1144:11: error: logical not is only 
applied to the left hand side of comparison [-Werror=logical-not-parentheses]
   if (!on != aniState->ofdmWeakSigDetectOff) {
   ^
drivers/net/ath/ath9k/ath9k_ar5008_phy.c: In function 
‘ar5008_hw_ani_control_new’:
drivers/net/ath/ath9k/ath9k_ar5008_phy.c:1310:11: error: logical not is only 
applied to the left hand side of comparison [-Werror=logical-not-parentheses]
   if (!on != aniState->ofdmWeakSigDetectOff) {
   ^
cc1: all warnings being treated as errors
bin/rules/drivers/net/ath/ath9k/ath9k_ar5008_phy.c.r:3: recipe for target 
'bin/ath9k_ar5008_phy.o' failed
make[7]: *** [bin/ath9k_ar5008_phy.o] Error 1
make[7]: Leaving directory '/work/src/xen/tools/firmware/etherboot/ipxe/src'
Makefile:28: recipe for target 'ipxe/src/bin/rtl8139.rom' failed
make[6]: *** [ipxe/src/bin/rtl8139.rom] Error 2

I know my gcc has some problems, and will generate wrong bin file(We have 
discussed
it earlier).
But this building error is new problem. Do we have any plan to use newer ipxe?

I update ipxe to avoid the building error, and I meet the following error:
hw/virtio/virtio-rng.c: In function ‘virtio_rng_device_realize’:
hw/virtio/virtio-rng.c:152:31: error: logical not is only applied to the left 
hand side of comparison [-Werror=logical-not-parentheses]
 if (!vrng->conf.period_ms > 0) {
   ^
cc1: all warnings being treated as errors

It seems that qemu have some problems.

Thanks
Wen Congyang

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream

[Wen Congyang]

On 06/17/2015 05:50 PM, Andrew Cooper wrote:
> On 17/06/15 08:57, Wen Congyang wrote:
>>> +/* Queue up reading the body. */
>>>> +size_t bytes_to_read;
>>>> +
>>>> +switch (rec_hdr->type) {
>>>> +/*
>>>> + * Emulator records want to retain the blob in the pipe, for a 
>>>> further
>>>> + * datacopier call to move elsewhere.  Just read the emulator 
>>>> header.
>>>> + */
>> In this case, we should not call ROUNDUP().
>>
>>>> +case REC_TYPE_EMULATOR_CONTEXT:
>>>> +bytes_to_read = sizeof(struct libxl_sr_emulator_hdr);
>>>> +break;
>>>> +
>>>> +default:
>>>> +bytes_to_read = rec_hdr->length;
>>>> +break;
>>>> +}
>>>> +
>>>> +bytes_to_read = ROUNDUP(bytes_to_read, REC_ALIGN_ORDER);
>> So, I think it is better to move ROUNDUP to default case.
>>
>> Thanks
>> Wen Congyang
>>
> 
> sizeof(struct libxl_sr_emulator_hdr) is cunningly of the appropriate
> order already.

Yes

> 
> I suppose it is probably better to move the roundup into the default
> case and assert() appropriate alignment after the switch()

Do you mean the sub-header must be aligned

Thanks
Wen Congyang

> 
> ~Andrew
> .
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 20/27] tools/libxl: Infrastructure for writing a v2 stream

[Wen Congyang]

et up writing a $FOO
> + *  - $BAR_header(): A $BAR record header only
> + *  - $BAR_record(): A complete $BAR record with header and content
> + *
> + * The main loop for a plain VM writes:
> + *  - Stream header
> + *  - Libxc record
> + *  - Toolstack record
> + *  - if (hvm), Qemu record
> + *  - End record
> + */
> +
> +static const uint8_t zero_padding[1U << REC_ALIGN_ORDER] = { 0 };
> +
> +static void stream_success(libxl__egc *egc,
> +   libxl__stream_write_state *stream);
> +static void stream_failed(libxl__egc *egc,
> +  libxl__stream_write_state *stream, int ret);
> +static void stream_done(libxl__egc *egc,
> +libxl__stream_write_state *stream);
> +
> +static void check_stream_finished(libxl__egc *egc,
> +  libxl__domain_suspend_state *dcs,
> +  int rc, const char *what);
> +
> +/* Event callbacks for plain VM. */
> +static void stream_header_done(libxl__egc *egc,
> +   libxl__datacopier_state *dc,
> +   int onwrite, int errnoval);
> +static void libxc_header_done(libxl__egc *egc,
> +  libxl__datacopier_state *dc,
> +  int onwrite, int errnoval);
> +/* libxl__xc_domain_save_done() lives here, event-order wise. */
> +static void write_toolstack_record(libxl__egc *egc,
> +   libxl__stream_write_state *stream);
> +static void toolstack_record_done(libxl__egc *egc,
> +  libxl__datacopier_state *dc,
> +  int onwrite, int errnoval);
> +static void write_emulator_record(libxl__egc *egc,
> +  libxl__stream_write_state *stream);
> +static void emulator_body_done(libxl__egc *egc,
> +   libxl__datacopier_state *dc,
> +   int onwrite, int errnoval);
> +static void emulator_padding_done(libxl__egc *egc,
> +  libxl__datacopier_state *dc,
> +  int onwrite, int errnoval);
> +static void write_end_record(libxl__egc *egc,
> + libxl__stream_write_state *stream);
> +static void end_record_done(libxl__egc *egc,
> +libxl__datacopier_state *dc,
> +int onwrite, int errnoval);
> +
> +void libxl__stream_write_start(libxl__egc *egc,
> +   libxl__stream_write_state *stream)
> +{
> +libxl__datacopier_state *dc = &stream->dc;
> +STATE_AO_GC(stream->ao);
> +struct libxl_sr_hdr hdr = { 0 };
> +int ret = 0;
> +
> +assert(!stream->running);
> +stream->running = true;
> +
> +memset(dc, 0, sizeof(*dc));
> +dc->readwhat = "";
> +dc->copywhat = "suspend header";
> +dc->writewhat = "save/migration stream";
> +dc->ao = ao;
> +dc->readfd = -1;
> +dc->writefd = stream->fd;
> +dc->maxsz = INT_MAX;
> +dc->bytes_to_read = INT_MAX;
> +dc->callback = stream_header_done;
> +
> +ret = libxl__datacopier_start(dc);
> +if (ret)
> +goto err;
> +
> +hdr.ident   = htobe64(RESTORE_STREAM_IDENT);
> +hdr.version = htobe32(RESTORE_STREAM_VERSION);
> +hdr.options = htobe32(0);
> +
> +libxl__datacopier_prefixdata(egc, dc, &hdr, sizeof(hdr));
> +return;
> +
> + err:
> +assert(ret);
> +stream_failed(egc, stream, ret);
> +}
> +
> +void libxl__stream_write_abort(libxl__egc *egc,
> +   libxl__stream_write_state *stream, int rc)
> +{
> +stream_failed(egc, stream, rc);
> +}
> +
> +static void stream_success(libxl__egc *egc, libxl__stream_write_state 
> *stream)
> +{
> +stream->rc = 0;
> +stream->running = false;
> +
> +stream_done(egc, stream);
> +}
> +
> +static void stream_failed(libxl__egc *egc,
> +  libxl__stream_write_state *stream, int rc)
> +{
> +assert(rc);
> +stream->rc = rc;
> +
> +if (stream->running) {
> +stream->running = false;
> +stream_done(egc, stream);
> +}
> +}
> +
> +static void stream_done(libxl__egc *egc,
> +libxl__stream_write_state *stream)
> +{
> +libxl__domain_suspend_state *dss = CONTAINER_OF(stream, *dss, sws);

Sometimes, we need to back stream from restore to save. In this case,
there is no dss. So it is better to not use dss directly in stream_w

Re: [Xen-devel] [PATCH 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream

[Wen Congyang]

On 06/17/2015 06:15 PM, Ian Campbell wrote:
> On Wed, 2015-06-17 at 11:09 +0800, Wen Congyang wrote:
>>> +if (hdr->options & RESTORE_OPT_BIG_ENDIAN) {
>>> +ret = ERROR_FAIL;
>>> +LOG(ERROR, "Unable to handle big endian streams");
>>> +goto err;
>>
>> I think it is better to check if the host is big endian or not.
> 
> It's not, there are no big endian ports of Xen today. I think encoding
> that here is fine.

IIRC, arm supports big endian. Do we only use arm+little endian? If so,
it is OK now.

Thanks
Wen Congyang

> 
> 
> .
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 23/27] tools/libxl: [RFC] Write checkpoint records into the stream

[Wen Congyang]

u record
> + *  - Checkpoint end record
>   */
>  
>  static const uint8_t zero_padding[1U << REC_ALIGN_ORDER] = { 0 };
> @@ -81,6 +90,16 @@ static void end_record_done(libxl__egc *egc,
>  libxl__datacopier_state *dc,
>  int onwrite, int errnoval);
>  
> +/* Event callbacks unique to checkpointed streams. */
> +static void checkpoint_done(libxl__egc *egc,
> +libxl__stream_write_state *stream,
> +int rc);
> +static void write_checkpoint_end_record(libxl__egc *egc,
> +libxl__stream_write_state *stream);
> +static void checkpoint_end_record_done(libxl__egc *egc,
> +   libxl__datacopier_state *dc,
> +   int onwrite, int errnoval);
> +
>  void libxl__stream_write_start(libxl__egc *egc,
> libxl__stream_write_state *stream)
>  {
> @@ -119,6 +138,16 @@ void libxl__stream_write_start(libxl__egc *egc,
>  stream_failed(egc, stream, ret);
>  }
>  
> +void libxl__stream_write_start_checkpoint(libxl__egc *egc,
> +  libxl__stream_write_state *stream)
> +{
> +assert(stream->running);
> +assert(!stream->in_checkpoint);
> +stream->in_checkpoint = true;
> +
> +write_toolstack_record(egc, stream);
> +}
> +
>  void libxl__stream_write_abort(libxl__egc *egc,
> libxl__stream_write_state *stream, int rc)
>  {
> @@ -130,6 +159,7 @@ static void stream_success(libxl__egc *egc, 
> libxl__stream_write_state *stream)
>  stream->rc = 0;
>  stream->running = false;
>  
> +assert(!stream->in_checkpoint);
>  stream_done(egc, stream);
>  }
>  
> @@ -139,6 +169,15 @@ static void stream_failed(libxl__egc *egc,
>  assert(rc);
>  stream->rc = rc;
>  
> +/*
> + *If we are in a checkpoint, pass the failure to libxc, which will come
> + * back around to us via libxl__xc_domain_save_done().
> + */
> +if (stream->in_checkpoint) {

I think we should set running to false here too.

Thanks
Wen Congyang

> +checkpoint_done(egc, stream, rc);
> +return;
> +}
> +
>  if (stream->running) {
>  stream->running = false;
>  stream_done(egc, stream);
> @@ -151,6 +190,7 @@ static void stream_done(libxl__egc *egc,
>  libxl__domain_suspend_state *dss = CONTAINER_OF(stream, *dss, sws);
>  
>  assert(!stream->running);
> +assert(!stream->in_checkpoint);
>  
>  check_stream_finished(egc, dss, stream->rc, "stream");
>  }
> @@ -335,8 +375,12 @@ static void toolstack_record_done(libxl__egc *egc,
>  
>  if (dss->type == LIBXL_DOMAIN_TYPE_HVM)
>  write_emulator_record(egc, stream);
> -else
> -write_end_record(egc, stream);
> +else {
> +if (stream->in_checkpoint)
> +write_checkpoint_end_record(egc, stream);
> +else
> +write_end_record(egc, stream);
> +}
>  
>  return;
>  
> @@ -473,7 +517,10 @@ static void emulator_padding_done(libxl__egc *egc,
>  goto err;
>  }
>  
> -write_end_record(egc, stream);
> +if (stream->in_checkpoint)
> +write_checkpoint_end_record(egc, stream);
> +else
> +write_end_record(egc, stream);
>  return;
>  
>   err:
> @@ -526,6 +573,64 @@ static void end_record_done(libxl__egc *egc,
>  stream_failed(egc, stream, ret);
>  }
>  
> +static void checkpoint_done(libxl__egc *egc,
> +libxl__stream_write_state *stream,
> +int rc)
> +{
> +libxl__domain_suspend_state *dss = CONTAINER_OF(stream, *dss, sws);
> +
> +assert(stream->in_checkpoint);
> +stream->in_checkpoint = false;
> +stream->checkpoint_callback(egc, dss, rc);
> +}
> +
> +static void write_checkpoint_end_record(libxl__egc *egc,
> +libxl__stream_write_state *stream)
> +{
> +libxl__datacopier_state *dc = &stream->dc;
> +STATE_AO_GC(stream->ao);
> +struct libxl_sr_rec_hdr rec = { REC_TYPE_CHECKPOINT_END, 0 };
> +int ret = 0;
> +
> +assert(stream->in_checkpoint);
> +
> +dc->copywhat = "checkpoint record";
> +dc->writewhat = "save/migration stream";
> +dc->callback = checkpoint_end_record_done;
> +
> +ret = libxl__datacopier_start(dc);
> +if (ret)
> +goto err;
> +
> +libxl__datacopier_prefixdata(egc, dc, &rec, sizeof(rec));
> +return;
> +
> + err:
> +assert(ret);
> +stream_failed(egc, stream, ret);
> +}
> +
> +static void checkpoint_end_record_done(libxl__egc *egc,
> +   libxl__datacopier_state *dc,
> +   int onwrite, int errnoval)
> +{
> +libxl__stream_write_state *stream = CONTAINER_OF(dc, *stream, dc);
> +STATE_AO_GC(stream->ao);
> +int ret = 0;
> +
> +if (onwrite || errnoval) {
> +ret = ERROR_FAIL;
> +goto err;
> +}
> +
> +checkpoint_done(egc, stream, 0);
> +return;
> +
> + err:
> +assert(ret);
> +stream_failed(egc, stream, ret);
> +}
> +
>  /*
>   * Local variables:
>   * mode: C
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 01/31] tools/libxl: introduce libxl__domain_restore_device_model to load qemu state

[Wen Congyang]

On 02/25/2016 11:53 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:05AM +0800, Wen Congyang wrote:
>> In normal migration, the qemu state is passed to qemu as a parameter.
>> With COLO, secondary vm is running. So we will do the following steps
>> at every checkpoint:
>> 1. suspend both primary vm and secondary vm
>> 2. sync the state
>> 3. resume both primary vm and secondary vm
>> Primary will send qemu's state in step2, and secondary's qemu should
>> read it and restore the state before it is resumed. We can not pass
>> the state to qemu as a parameter because secondary QEMU already started
> 
> is already started.
> 
>> at this point, so we introduce libxl__domain_restore_device_model() to
>> do it. This API MUST be called before resuming secondary vm.
>>
> 
> The last sentence is of no relevancy of this function itself, so I would
> just remove it if this patch will be resent in its current form.
> 
> And some comments below.
> 
>> Signed-off-by: Yang Hongyang 
>> Signed-off-by: Wen Congyang 
>> Cc: Anthony Perard 
>> Reviewed-by: Konrad Rzeszutek Wilk 
>> ---
>>  tools/libxl/libxl_dom_save.c | 20 
>>  tools/libxl/libxl_internal.h |  4 
>>  tools/libxl/libxl_qmp.c  | 10 ++
>>  3 files changed, 34 insertions(+)
>>
>> diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
>> index 4eb7960..7d345f9 100644
>> --- a/tools/libxl/libxl_dom_save.c
>> +++ b/tools/libxl/libxl_dom_save.c
>> @@ -512,6 +512,26 @@ int 
>> libxl__restore_emulator_xenstore_data(libxl__domain_create_state *dcs,
>>  return rc;
>>  }
>>  
>> +int libxl__domain_restore_device_model(libxl__gc *gc, uint32_t domid,
>> +   const char *restore_file)
>> +{
>> +int rc;
>> +
>> +switch (libxl__device_model_version_running(gc, domid)) {
>> +case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL:
>> +/* Will never be supported. */
>> +rc = ERROR_INVAL;
>> +break;
>> +case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN:
>> +rc = libxl__qmp_restore(gc, domid, restore_file);
>> +break;
>> +default:
>> +rc = ERROR_INVAL;
>> +}
>> +
>> +return rc;
>> +}
>> +
> 
> The function name suggests that it should be universally applied to all
> device model cases and all places that involves restoring device model.
> This is not the case in this series.  I search for this function in the
> rest of this series, it seems to be only used by COLO (in patch #10).
> 
> It's also unclear where the other half libxl__domain_save_device_model
> is.  I don't think this is your problem, though.
> 
> Based on the two reasons above, I would say let's not have this function
> at all.  You can call libxl__qmp_restore in COLO code directly.
> 
> Does this sound plausible?
> 
> If you agree, this patch can be turned into introduction of
> libxl__qmp_restore.

Yes, it is only used by COLO. I will update it in the next version.

Thanks
Wen Congyang

> 
>>  /*
>>   * Local variables:
>>   * mode: C
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index a1aae97..5320e5e 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -1119,6 +1119,8 @@ _hidden int libxl__domain_rename(libxl__gc *gc, 
>> uint32_t domid,
>>   const char *old_name, const char *new_name,
>>   xs_transaction_t trans);
>>  
>> +_hidden int libxl__domain_restore_device_model(libxl__gc *gc, uint32_t 
>> domid,
>> +   const char *restore_file);
>>  _hidden int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t 
>> domid);
>>  
>>  _hidden const char *libxl__userdata_path(libxl__gc *gc, uint32_t domid,
>> @@ -1762,6 +1764,8 @@ _hidden int libxl__qmp_stop(libxl__gc *gc, int domid);
>>  _hidden int libxl__qmp_resume(libxl__gc *gc, int domid);
>>  /* Save current QEMU state into fd. */
>>  _hidden int libxl__qmp_save(libxl__gc *gc, int domid, const char *filename);
>> +/* Load current QEMU state from file. */
>> +_hidden int libxl__qmp_restore(libxl__gc *gc, int domid, const char 
>> *filename);
>>  /* Set dirty bitmap logging status */
>>  _hidden int libxl__qmp_set_global_dirty_log(libxl__gc *gc, int domid, bool 
>> enable);
>>  _hidden int libxl__qmp_insert_cdrom(libxl__gc *gc, int domid, const 
>>

Re: [Xen-devel] [PATCH v10 07/31] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams

[Wen Congyang]

On 02/25/2016 11:54 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:11AM +0800, Wen Congyang wrote:
>> It is the negotiation record for COLO.
>> Primary->Secondary:
>> control_id  0x: Secondary VM is out of sync, start a new 
>> checkpoint
>> Secondary->Primary:
>> 0x0001: Secondary VM is suspended
>> 0x0002: Secondary VM is ready
>> 0x0003: Secondary VM is resumed
>>
>> Signed-off-by: Wen Congyang 
>> Signed-off-by: Yang Hongyang 
>> ---
>>  docs/specs/libxl-migration-stream.pandoc | 25 +++--
>>  tools/libxl/libxl_sr_stream_format.h | 11 +++
>>  tools/python/xen/migration/libxl.py  |  9 +
>>  3 files changed, 43 insertions(+), 2 deletions(-)
>>
>> diff --git a/docs/specs/libxl-migration-stream.pandoc 
>> b/docs/specs/libxl-migration-stream.pandoc
>> index 2c97d86..5166d66 100644
>> --- a/docs/specs/libxl-migration-stream.pandoc
>> +++ b/docs/specs/libxl-migration-stream.pandoc
>> @@ -1,6 +1,6 @@
>>  % LibXenLight Domain Image Format
>>  % Andrew Cooper <>
> 
> You can add yours and Hongyang's name and email here as well.

OK

> 
>> -% Revision 1
>> +% Revision 2
>>  
>>  Introduction
>>  
>> @@ -119,7 +119,9 @@ type 0x: END
>>  
>>   0x0004: CHECKPOINT_END
>>  
>> - 0x0005 - 0x7FFF: Reserved for future _mandatory_
>> + 0x0005: CHECKPOINT_STATE
>> +
>> + 0x0006 - 0x7FFF: Reserved for future _mandatory_
>>   records.
>>  
>>   0x8000 - 0x: Reserved for future _optional_
>> @@ -249,6 +251,25 @@ A checkpoint end record marks the end of a checkpoint 
>> in the image.
>>  The end record contains no fields; its body_length is 0.
>>  
>>  
>> +CHECKPOINT\_STATE
>> +--
>> +
>> +A checkpoint state record contains the control information for checkpoint.
>> +
>> + 0 1 2 3 4 5 6 7 octet
>> ++++
>> +| control_id | padding|
>> ++++
>> +
>> +
>> +FieldDescription
>> + ---
>> +control_id   0x: Secondary VM is out of sync, start a new 
>> checkpoint
>> + 0x0001: Secondary VM is suspended
>> + 0x0002: Secondary VM is ready
>> + 0x0003: Secondary VM is resumed
> 
> You commit message seems to suggest 0 can only go from primary to
> secondary, while the other three only go from secondary to primary. You
> probably need to spell out the direction here (and later enforce this
> behaviour in code).

OK, will fix it in the next version.

Thanks
Wen Congyang

> 
> Wei.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 10/31] tools/libxl: add back channel support to write stream

[Wen Congyang]

On 02/25/2016 11:54 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:14AM +0800, Wen Congyang wrote:
>> Add back channel support to write stream. If the write stream is
>> a back channel stream, this means the write stream is used by
>> Secondary to send some records back.
>>
>> Signed-off-by: Yang Hongyang 
>> Signed-off-by: Wen Congyang 
>> ---
>>  tools/libxl/libxl_dom_save.c |  1 +
>>  tools/libxl/libxl_internal.h |  1 +
>>  tools/libxl/libxl_stream_write.c | 26 --
>>  3 files changed, 22 insertions(+), 6 deletions(-)
>>
>> diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
>> index 72b61c7..18946c5 100644
>> --- a/tools/libxl/libxl_dom_save.c
>> +++ b/tools/libxl/libxl_dom_save.c
>> @@ -404,6 +404,7 @@ void libxl__domain_save(libxl__egc *egc, 
>> libxl__domain_save_state *dss)
>>  dss->sws.ao  = dss->ao;
>>  dss->sws.dss = dss;
>>  dss->sws.fd  = dss->fd;
>> +dss->sws.back_channel = false;
>>  dss->sws.completion_callback = stream_done;
>>  
>>  libxl__stream_write_start(egc, &dss->sws);
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index 3d3e8e8..e02e554 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -3044,6 +3044,7 @@ struct libxl__stream_write_state {
>>  libxl__ao *ao;
>>  libxl__domain_save_state *dss;
>>  int fd;
>> +bool back_channel;
>>  void (*completion_callback)(libxl__egc *egc,
>>  libxl__stream_write_state *sws,
>>  int rc);
>> diff --git a/tools/libxl/libxl_stream_write.c 
>> b/tools/libxl/libxl_stream_write.c
>> index f6ea55d..5379126 100644
>> --- a/tools/libxl/libxl_stream_write.c
>> +++ b/tools/libxl/libxl_stream_write.c
>> @@ -49,6 +49,13 @@
>>   *  - if (hvm)
>>   *  - Emulator context record
>>   *  - Checkpoint end record
>> + *
>> + * For back channel stream:
>> + * - libxl__stream_write_start()
>> + *- Set up the stream to running state
>> + *
>> + * - Add a new API to write the record. When the record is written
>> + *   out, call stream->checkpoint_callback() to return.
> 
> What does this mean? Which new API?

The next patch introduces this API. The commits is very old.

I think I can merge these two patches into one patch.

> 
>>   */
>>  
>>  /* Success/error/cleanup handling. */
>> @@ -225,6 +232,15 @@ void libxl__stream_write_start(libxl__egc *egc,
>>  
>>  stream->running = true;
>>  
>> +dc->ao= ao;
>> +dc->readfd= -1;
>> +dc->copywhat  = "save v2 stream";
>> +dc->writefd   = stream->fd;
>> +dc->maxsz = -1;
>> +
>> +if (stream->back_channel)
>> +return;
>> +
> 
> There seems to be very subtle change of behaviour.
> 
> Before this patch, the dc->* are not set until the emulator check is
> done. With this path, it is possible in the normal case some of the
> fields are initialised in the error path.
> 
> I think this is OK given the callbacks in the upper layer and in
> the writer don't rely on those fields to clean up. Just one thing to
> note.
> 
> That said, I suggest you move all initialisation of dc->* in one place.

OK, I will do it.

> 
>>  if (dss->type == LIBXL_DOMAIN_TYPE_HVM) {
>>  stream->device_model_version =
>>  libxl__device_model_version_running(gc, dss->domid);
>> @@ -249,12 +265,7 @@ void libxl__stream_write_start(libxl__egc *egc,
>>  stream->emu_sub_hdr.index = 0;
>>  }
>>  
>> -dc->ao= ao;
>> -dc->readfd= -1;
>>  dc->writewhat = "stream header";
>> -dc->copywhat  = "save v2 stream";
>> -dc->writefd   = stream->fd;
>> -dc->maxsz = -1;
>>  dc->callback  = stream_header_done;
>>  
>>  rc = libxl__datacopier_start(dc);
>> @@ -279,6 +290,7 @@ void libxl__stream_write_start_checkpoint(libxl__egc 
>> *egc,
>>  {
>>  assert(stream->running);
>>  assert(!stream->in_checkpoint);
>> +assert(!stream->back_channel);
>>  stream->in_checkpoint = true;
>>  
>>  write_emulator_xenstore_record(egc, stream);
>> @@ -590,7 +602,9 @@ static void stream_done(libxl__egc *egc,
>>  libxl__carefd_clos

Re: [Xen-devel] [PATCH v10 12/31] tools/libxl: add back channel support to read stream

[Wen Congyang]

On 02/25/2016 11:54 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:16AM +0800, Wen Congyang wrote:
>> This is used by primay to read records sent by secondary.
>>
>> Signed-off-by: Yang Hongyang 
>> Signed-off-by: Wen Congyang 
>> ---
>>  tools/libxl/libxl_create.c  |  1 +
>>  tools/libxl/libxl_internal.h|  1 +
>>  tools/libxl/libxl_stream_read.c | 27 +++
>>  3 files changed, 25 insertions(+), 4 deletions(-)
>>
>> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
>> index 525bf85..fc746fb 100644
>> --- a/tools/libxl/libxl_create.c
>> +++ b/tools/libxl/libxl_create.c
>> @@ -1030,6 +1030,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
>>  dcs->srs.dcs = dcs;
>>  dcs->srs.fd = restore_fd;
>>  dcs->srs.legacy = (dcs->restore_params.stream_version == 1);
>> +dcs->srs.back_channel = false;
>>  dcs->srs.completion_callback = domcreate_stream_done;
>>  
>>  if (restore_fd >= 0) {
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index cdf6751..32957ca 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -3420,6 +3420,7 @@ struct libxl__stream_read_state {
>>  libxl__domain_create_state *dcs;
>>  int fd;
>>  bool legacy;
>> +bool back_channel;
>>  void (*completion_callback)(libxl__egc *egc,
>>  libxl__stream_read_state *srs,
>>  int rc);
>> diff --git a/tools/libxl/libxl_stream_read.c 
>> b/tools/libxl/libxl_stream_read.c
>> index f4781eb..02a2c46 100644
>> --- a/tools/libxl/libxl_stream_read.c
>> +++ b/tools/libxl/libxl_stream_read.c
>> @@ -118,6 +118,15 @@
>>   *record, and therefore the buffered state is inconsistent. In
>>   *libxl__xc_domain_restore_done(), we just complete the stream and
>>   *stream->completion_callback() will be called to resume the guest
>> + *
>> + * For back channel stream:
>> + * - libxl__stream_read_start()
>> + *- Set up the stream to running state
>> + *
>> + * - libxl__stream_read_continue()
>> + * - Set up reading the next record from a started stream.
>> + *   Add some codes to process_record() to handle the record.
>> + *   Then call stream->checkpoint_callback() to return.
>>   */
>>  
>>  /* Success/error/cleanup handling. */
>> @@ -221,6 +230,17 @@ void libxl__stream_read_start(libxl__egc *egc,
>>  stream->running = true;
>>  stream->phase   = SRS_PHASE_NORMAL;
>>  
>> +dc->ao   = stream->ao;
>> +dc->copywhat = "restore v2 stream";
>> +dc->writefd  = -1;
>> +
>> +if (stream->back_channel) {
>> +assert(!stream->legacy);
>> +
>> +dc->readfd = stream->fd;
> 
> Why is this needed? dc->readfd is set to stream->fd no matter it is back
> channel or not. This can be moved outside this if {}.

Yes, will fix it in the next version.

> 
>> +return;
>> +}
>> +
>>  if (stream->legacy) {
>>  /* Convert the legacy stream. */
>>  libxl__conversion_helper_state *chs = &stream->chs;
>> @@ -243,10 +263,7 @@ void libxl__stream_read_start(libxl__egc *egc,
>>  }
>>  /* stream->fd is now a v2 stream. */
>>  
>> -dc->ao   = stream->ao;
>> -dc->copywhat = "restore v2 stream";
>>  dc->readfd   = stream->fd;
>> -dc->writefd  = -1;
>>  
>>  /* Start reading the stream header. */
>>  rc = setup_read(stream, "stream header",
>> @@ -762,7 +779,9 @@ static void stream_done(libxl__egc *egc,
>>  LIBXL_STAILQ_FOREACH_SAFE(rec, &stream->record_queue, entry, trec)
>>  free_record(rec);
>>  
>> -check_all_finished(egc, stream, rc);
>> +if (!stream->back_channel)
>> +/* back channel stream doesn't have restore helper */
>> +check_all_finished(egc, stream, rc);
> 
> Even if it doesn't have restore helper, check_all_finished also checks
> if the stream and the conversion helper are till in use.  The
> explanation in the comment doesn't seem to justify this change.

In stream_done(), stream->running is set to false, so 
libxl__stream_read_in_use()
returns false.

Back channel stream doesn't support legacy stream, so there is no conversion 
helper.

I will update the comments in the next version.

Thanks
Wen Congyang

> 
> Wei.
> 
>>  }
>>  
>>  void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
>> -- 
>> 2.5.0
>>
>>
>>
>>
>> ___
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 16/31] secondary vm suspend/resume/checkpoint code

[Wen Congyang]

On 02/25/2016 11:56 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:20AM +0800, Wen Congyang wrote:
>> Secondary vm is running in colo mode. So we will do
>> the following things again and again:
>> 1. Resume secondary vm
>>a. Send CHECKPOINT_SVM_READY to master.
>>b. If it is not the first resume, call 
>> libxl__checkpoint_devices_preresume().
>>c. If it is the first resume(resume right after live migration),
>>   - call libxl__xc_domain_restore_done() to build the secondary vm.
>>   - enable secondary vm's logdirty.
>>   - call libxl__domain_resume() to resume secondary vm.
>>   - call libxl__checkpoint_devices_setup() to setup checkpoint devices.
>>d. Send CHECKPOINT_SVM_RESUMED to master.
>> 2. Wait a new checkpoint
>>a. Call libxl__checkpoint_devices_commit().
>>b. Read CHECKPOINT_NEW from master.
>> 3. Suspend secondary vm
>>a. Suspend secondary vm.
>>b. Call libxl__checkpoint_devices_postsuspend().
>>c. Send CHECKPOINT_SVM_SUSPENDED to master.
>>
>> Signed-off-by: Wen Congyang 
>> Signed-off-by: Yang Hongyang 
>> ---
>>  tools/libxc/xc_sr_common.h   |2 +
>>  tools/libxc/xc_sr_save.c |3 +-
>>  tools/libxl/Makefile |1 +
>>  tools/libxl/libxl_colo.h |   24 +
>>  tools/libxl/libxl_colo_restore.c | 1038 
>> ++
>>  tools/libxl/libxl_create.c   |   37 ++
>>  tools/libxl/libxl_internal.h |   19 +
>>  tools/libxl/libxl_save_callout.c |7 +-
>>  tools/libxl/libxl_stream_read.c  |   12 +
>>  tools/libxl/libxl_types.idl  |1 +
> 
> There is a bunch of TODOs in libxl_colo.c but I don't think you're in a
> better position to judge whether they should be blocker or not.
> 
>>  10 files changed, 1142 insertions(+), 2 deletions(-)
>>  create mode 100644 tools/libxl/libxl_colo.h
>>  create mode 100644 tools/libxl/libxl_colo_restore.c
>>
>> diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
>> index 5d9f497..2bfed64 100644
>> --- a/tools/libxc/xc_sr_common.h
>> +++ b/tools/libxc/xc_sr_common.h
>> @@ -184,10 +184,12 @@ struct xc_sr_context
>>   * migration stream
>>   * 0: Plain VM
>>   * 1: Remus
>> + * 2: COLO
>>   */
>>  enum {
>>  MIG_STREAM_NONE, /* plain stream */
>>  MIG_STREAM_REMUS,
>> +MIG_STREAM_COLO,
>>  } migration_stream;
>>  
>>  union /* Common save or restore data. */
>> diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
>> index fe210cc..7393355 100644
>> --- a/tools/libxc/xc_sr_save.c
>> +++ b/tools/libxc/xc_sr_save.c
>> @@ -846,7 +846,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, 
>> uint32_t dom,
>>  
>>  /* If altering migration_stream update this assert too. */
>>  assert(checkpointed_stream == MIG_STREAM_NONE ||
>> -   checkpointed_stream == MIG_STREAM_REMUS);
>> +   checkpointed_stream == MIG_STREAM_REMUS ||
>> +   checkpointed_stream == MIG_STREAM_COLO);
>>  
>>  /*
>>   * TODO: Find some time to better tweak the live migration algorithm.
> 
> [...]
> 
>> +
>> +#include "libxl_osdeps.h" /* must come before any other headers */
>> +
>> +#include "libxl_internal.h"
>> +#include "libxl_colo.h"
>> +#include "libxl_sr_stream_format.h"
>> +
>> +enum {
>> +LIBXL_COLO_SETUPED,
>> +LIBXL_COLO_SUSPENDED,
>> +LIBXL_COLO_RESUMED,
>> +};
>> +
>> +typedef struct libxl__colo_restore_checkpoint_state 
>> libxl__colo_restore_checkpoint_state;
>> +struct libxl__colo_restore_checkpoint_state {
>> +libxl__domain_suspend_state dsps;
>> +libxl__logdirty_switch lds;
>> +libxl__colo_restore_state *crs;
>> +libxl__stream_write_state sws;
>> +int status;
>> +bool preresume;
>> +/* used for teardown */
>> +int teardown_devices;
>> +int saved_rc;
>> +char *state_file;
>> +
>> +void (*callback)(libxl__egc *,
>> + libxl__colo_restore_checkpoint_state *,
>> + int);
>> +};
>> +
> 
> Shouldn't the enum and struct belong to libxl_colo.h ?

It only be used by restore side. I think it is OK to move them to libxl_colo.h.

> 
>> +
>> +static void libxl__colo_restore_domain_resume_callback(void *data);
>> +static void libxl__colo_restore_d

Re: [Xen-devel] [PATCH v10 17/31] primary vm suspend/resume/checkpoint code

[Wen Congyang]

On 02/25/2016 11:57 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:21AM +0800, Wen Congyang wrote:
> [...]
>> +#include "libxl_osdeps.h" /* must come before any other headers */
>> +
>> +#include "libxl_internal.h"
>> +#include "libxl_colo.h"
>> +
>> +static const libxl__checkpoint_device_instance_ops *colo_ops[] = {
>> +NULL,
>> +};
>> +
>> +/* = helper functions = */
>> +static int init_device_subkind(libxl__checkpoint_devices_state *cds)
>> +{
>> +/* init device subkind-specific state in the libxl ctx */
>> +int rc;
>> +STATE_AO_GC(cds->ao);
>> +
>> +rc = 0;
>> +return rc;
>> +}
>> +
>> +static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
>> +{
>> +/* cleanup device subkind-specific state in the libxl ctx */
>> +STATE_AO_GC(cds->ao);
>> +}
>> +
>> +/* = colo: setup save environment = */
>> +static void colo_save_setup_done(libxl__egc *egc,
>> + libxl__checkpoint_devices_state *cds,
>> + int rc);
>> +static void colo_save_setup_failed(libxl__egc *egc,
>> +   libxl__checkpoint_devices_state *cds,
>> +   int rc);
>> +static void libxl__colo_save_domain_suspend_callback(void *data);
>> +static void libxl__colo_save_domain_checkpoint_callback(void *data);
>> +static void libxl__colo_save_domain_resume_callback(void *data);
>> +static void libxl__colo_save_domain_wait_checkpoint_callback(void *data);
>> +
> 
> 
> An ordered list of callbacks is appreciated. This doesn't seem to cover
> all callbacks and I'm not sure if this is the order they are supposed to
> fire either.

OK, I will check it and fix it in the next version.

Thanks
Wen Congyang

> 
> Wei.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 18/31] libxc/restore: support COLO restore

[Wen Congyang]

On 02/25/2016 11:57 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:22AM +0800, Wen Congyang wrote:
> [...]
>> - * With Remus, we buffer the records sent by the primary at checkpoint,
>> + * With Remus/COLO, we buffer the records sent by the primary at checkpoint,
>>   * in case the primary will fail, we can recover from the last
>>   * checkpoint state.
>>   * This should be enough for most of the cases because primary only send
>> diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
>> index aef9bca..2ae8154 100644
>> --- a/tools/libxc/xc_sr_restore.c
>> +++ b/tools/libxc/xc_sr_restore.c
>> @@ -460,6 +460,49 @@ static int handle_checkpoint(struct xc_sr_context *ctx)
>>  else
>>  ctx->restore.buffer_all_records = true;
>>  
>> +if ( ctx->restore.checkpointed == MIG_STREAM_COLO )
>> +{
>> +#define HANDLE_CALLBACK_RETURN_VALUE(ret)   \
>> +do {\
>> +if ( ret == 1 ) \
>> +rc = 0; /* Success */   \
>> +else\
>> +{   \
>> +if ( ret == 2 ) \
>> +rc = BROKEN_CHANNEL;\
>> +else\
>> +rc = -1; /* Some unspecified error */   \
>> +goto err;   \
>> +}   \
>> +} while (0)
>> +
>> +/* COLO */
>> +
>> +/* We need to resume guest */
>> +rc = ctx->restore.ops.stream_complete(ctx);
>> +if ( rc )
>> +goto err;
>> +
>> +/* TODO: call restore_results */
>> +
>> +/* Resume secondary vm */
>> +ret = 
>> ctx->restore.callbacks->postcopy(ctx->restore.callbacks->data);
>> +HANDLE_CALLBACK_RETURN_VALUE(ret);
>> +
>> +/* Wait for a new checkpoint */
>> +ret = ctx->restore.callbacks->wait_checkpoint(
>> +
>> ctx->restore.callbacks->data);
>> +HANDLE_CALLBACK_RETURN_VALUE(ret);
>> +
>> +/* suspend secondary vm */
>> +ret = ctx->restore.callbacks->suspend(ctx->restore.callbacks->data);
>> +HANDLE_CALLBACK_RETURN_VALUE(ret);
>> +
>> +#undef HANDLE_CALLBACK_RETURN_VALUE
>> +
>> +/* TODO: send dirty pfn list to primary */
> 
> You replace the TODOs with actual code in the next two patches.
> 
> You can rearrange them a bit so that you don't need to add TODOs at all.

Yes, will fix it in the next version.

Thanks
Wen Congyang

> 
> 
> Wei.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 21/31] libxc/save: support COLO save

[Wen Congyang]

On 02/25/2016 11:58 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:25AM +0800, Wen Congyang wrote:
> [...]
>>  /*
>>   * Suspend the domain and send dirty memory.
>>   * This is the last iteration of the live migration and the
>> @@ -558,6 +610,16 @@ static int suspend_and_send_dirty(struct xc_sr_context 
>> *ctx)
>>  
>>  bitmap_or(dirty_bitmap, ctx->save.deferred_pages, ctx->save.p2m_size);
>>  
>> +if ( !ctx->save.live && ctx->save.checkpointed == MIG_STREAM_COLO )
>> +{
>> +rc = merge_secondary_dirty_bitmap(ctx);
>> +if ( rc )
>> +{
>> +PERROR("Failed to get secondary vm's dirty pages");
>> +goto out;
>> +}
>> +}
>> +
>>  rc = send_dirty_pages(ctx, stats.dirty_count + 
>> ctx->save.nr_deferred_pages);
>>  if ( rc )
>>  goto out;
>> @@ -791,13 +853,42 @@ static int save(struct xc_sr_context *ctx, uint16_t 
>> guest_type)
>>  if ( rc )
>>  goto err;
>>  
>> -rc = ctx->save.callbacks->postcopy(ctx->save.callbacks->data);
>> -if ( rc <= 0 )
>> -goto err;
>> +if ( ctx->save.checkpointed == MIG_STREAM_COLO )
>> +{
>> +rc = 
>> ctx->save.callbacks->checkpoint(ctx->save.callbacks->data);
>> +if ( !rc )
>> +{
>> +rc = -1;
>> +goto err;
>> +}
>> +}
>>  
>> -rc = ctx->save.callbacks->checkpoint(ctx->save.callbacks->data);
>> -if ( rc <= 0 )
>> +rc = ctx->save.callbacks->postcopy(ctx->save.callbacks->data);
>> +if ( !rc )
> 
> This original code for checking postcopy return value is if ( rc <= 0 ).

OK, I will check it.

Thanks
Wen Congyang

> 
> Wei.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 16/31] secondary vm suspend/resume/checkpoint code

[Wen Congyang]

On 02/25/2016 11:56 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:20AM +0800, Wen Congyang wrote:
>> Secondary vm is running in colo mode. So we will do
>> the following things again and again:
>> 1. Resume secondary vm
>>a. Send CHECKPOINT_SVM_READY to master.
>>b. If it is not the first resume, call 
>> libxl__checkpoint_devices_preresume().
>>c. If it is the first resume(resume right after live migration),
>>   - call libxl__xc_domain_restore_done() to build the secondary vm.
>>   - enable secondary vm's logdirty.
>>   - call libxl__domain_resume() to resume secondary vm.
>>   - call libxl__checkpoint_devices_setup() to setup checkpoint devices.
>>d. Send CHECKPOINT_SVM_RESUMED to master.
>> 2. Wait a new checkpoint
>>a. Call libxl__checkpoint_devices_commit().
>>b. Read CHECKPOINT_NEW from master.
>> 3. Suspend secondary vm
>>a. Suspend secondary vm.
>>b. Call libxl__checkpoint_devices_postsuspend().
>>c. Send CHECKPOINT_SVM_SUSPENDED to master.
>>
>> Signed-off-by: Wen Congyang 
>> Signed-off-by: Yang Hongyang 
>> ---
>>  tools/libxc/xc_sr_common.h   |2 +
>>  tools/libxc/xc_sr_save.c |3 +-
>>  tools/libxl/Makefile |1 +
>>  tools/libxl/libxl_colo.h |   24 +
>>  tools/libxl/libxl_colo_restore.c | 1038 
>> ++
>>  tools/libxl/libxl_create.c   |   37 ++
>>  tools/libxl/libxl_internal.h |   19 +
>>  tools/libxl/libxl_save_callout.c |7 +-
>>  tools/libxl/libxl_stream_read.c  |   12 +
>>  tools/libxl/libxl_types.idl  |1 +
> 
> There is a bunch of TODOs in libxl_colo.c but I don't think you're in a
> better position to judge whether they should be blocker or not.
> 
>>  10 files changed, 1142 insertions(+), 2 deletions(-)
>>  create mode 100644 tools/libxl/libxl_colo.h
>>  create mode 100644 tools/libxl/libxl_colo_restore.c
>>
>> diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
>> index 5d9f497..2bfed64 100644
>> --- a/tools/libxc/xc_sr_common.h
>> +++ b/tools/libxc/xc_sr_common.h
>> @@ -184,10 +184,12 @@ struct xc_sr_context
>>   * migration stream
>>   * 0: Plain VM
>>   * 1: Remus
>> + * 2: COLO
>>   */
>>  enum {
>>  MIG_STREAM_NONE, /* plain stream */
>>  MIG_STREAM_REMUS,
>> +MIG_STREAM_COLO,
>>  } migration_stream;
>>  
>>  union /* Common save or restore data. */
>> diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
>> index fe210cc..7393355 100644
>> --- a/tools/libxc/xc_sr_save.c
>> +++ b/tools/libxc/xc_sr_save.c
>> @@ -846,7 +846,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, 
>> uint32_t dom,
>>  
>>  /* If altering migration_stream update this assert too. */
>>  assert(checkpointed_stream == MIG_STREAM_NONE ||
>> -   checkpointed_stream == MIG_STREAM_REMUS);
>> +   checkpointed_stream == MIG_STREAM_REMUS ||
>> +   checkpointed_stream == MIG_STREAM_COLO);
>>  
>>  /*
>>   * TODO: Find some time to better tweak the live migration algorithm.
> 
> [...]
> 
>> +
>> +#include "libxl_osdeps.h" /* must come before any other headers */
>> +
>> +#include "libxl_internal.h"
>> +#include "libxl_colo.h"
>> +#include "libxl_sr_stream_format.h"
>> +
>> +enum {
>> +LIBXL_COLO_SETUPED,
>> +LIBXL_COLO_SUSPENDED,
>> +LIBXL_COLO_RESUMED,
>> +};
>> +
>> +typedef struct libxl__colo_restore_checkpoint_state 
>> libxl__colo_restore_checkpoint_state;
>> +struct libxl__colo_restore_checkpoint_state {
>> +libxl__domain_suspend_state dsps;
>> +libxl__logdirty_switch lds;
>> +libxl__colo_restore_state *crs;
>> +libxl__stream_write_state sws;
>> +int status;
>> +bool preresume;
>> +/* used for teardown */
>> +int teardown_devices;
>> +int saved_rc;
>> +char *state_file;
>> +
>> +void (*callback)(libxl__egc *,
>> + libxl__colo_restore_checkpoint_state *,
>> + int);
>> +};
>> +
> 
> Shouldn't the enum and struct belong to libxl_colo.h ?

If we inlucde libxl_colo.h in libxl_internal.h, we cannot move this into 
colo.h, because
this structure needs libxl__domain_suspend_state, libxl__logdirty_switch, ...
We cannot just declare it, because this structure needs

Re: [Xen-devel] [PATCH v10 10/31] tools/libxl: add back channel support to write stream

[Wen Congyang]

On 03/02/2016 11:02 PM, Wei Liu wrote:
> On Fri, Feb 26, 2016 at 10:11:27AM +0800, Wen Congyang wrote:
>> On 02/25/2016 11:54 PM, Wei Liu wrote:
>>> On Mon, Feb 22, 2016 at 10:52:14AM +0800, Wen Congyang wrote:
>>>> Add back channel support to write stream. If the write stream is
>>>> a back channel stream, this means the write stream is used by
>>>> Secondary to send some records back.
>>>>
>>>> Signed-off-by: Yang Hongyang 
>>>> Signed-off-by: Wen Congyang 
>>>> ---
>>>>  tools/libxl/libxl_dom_save.c |  1 +
>>>>  tools/libxl/libxl_internal.h |  1 +
>>>>  tools/libxl/libxl_stream_write.c | 26 --
>>>>  3 files changed, 22 insertions(+), 6 deletions(-)
>>>>
>>>> diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
>>>> index 72b61c7..18946c5 100644
>>>> --- a/tools/libxl/libxl_dom_save.c
>>>> +++ b/tools/libxl/libxl_dom_save.c
>>>> @@ -404,6 +404,7 @@ void libxl__domain_save(libxl__egc *egc, 
>>>> libxl__domain_save_state *dss)
>>>>  dss->sws.ao  = dss->ao;
>>>>  dss->sws.dss = dss;
>>>>  dss->sws.fd  = dss->fd;
>>>> +dss->sws.back_channel = false;
>>>>  dss->sws.completion_callback = stream_done;
>>>>  
>>>>  libxl__stream_write_start(egc, &dss->sws);
>>>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>>>> index 3d3e8e8..e02e554 100644
>>>> --- a/tools/libxl/libxl_internal.h
>>>> +++ b/tools/libxl/libxl_internal.h
>>>> @@ -3044,6 +3044,7 @@ struct libxl__stream_write_state {
>>>>  libxl__ao *ao;
>>>>  libxl__domain_save_state *dss;
>>>>  int fd;
>>>> +bool back_channel;
>>>>  void (*completion_callback)(libxl__egc *egc,
>>>>  libxl__stream_write_state *sws,
>>>>  int rc);
>>>> diff --git a/tools/libxl/libxl_stream_write.c 
>>>> b/tools/libxl/libxl_stream_write.c
>>>> index f6ea55d..5379126 100644
>>>> --- a/tools/libxl/libxl_stream_write.c
>>>> +++ b/tools/libxl/libxl_stream_write.c
>>>> @@ -49,6 +49,13 @@
>>>>   *  - if (hvm)
>>>>   *  - Emulator context record
>>>>   *  - Checkpoint end record
>>>> + *
>>>> + * For back channel stream:
>>>> + * - libxl__stream_write_start()
>>>> + *- Set up the stream to running state
>>>> + *
>>>> + * - Add a new API to write the record. When the record is written
>>>> + *   out, call stream->checkpoint_callback() to return.
>>>
>>> What does this mean? Which new API?
>>
>> The next patch introduces this API. The commits is very old.
>>
>> I think I can merge these two patches into one patch.
>>
> 
> Please reference the actual function / API.
> 
>>>
>>>>   */
>>>>  
>>>>  /* Success/error/cleanup handling. */
>>>> @@ -225,6 +232,15 @@ void libxl__stream_write_start(libxl__egc *egc,
>>>>  
>>>>  stream->running = true;
>>>>  
>>>> +dc->ao= ao;
>>>> +dc->readfd= -1;
>>>> +dc->copywhat  = "save v2 stream";
>>>> +dc->writefd   = stream->fd;
>>>> +dc->maxsz = -1;
>>>> +
>>>> +if (stream->back_channel)
>>>> +return;
>>>> +
>>>
>>> There seems to be very subtle change of behaviour.
>>>
>>> Before this patch, the dc->* are not set until the emulator check is
>>> done. With this path, it is possible in the normal case some of the
>>> fields are initialised in the error path.
>>>
>>> I think this is OK given the callbacks in the upper layer and in
>>> the writer don't rely on those fields to clean up. Just one thing to
>>> note.
>>>
>>> That said, I suggest you move all initialisation of dc->* in one place.
>>
>> OK, I will do it.
>>
>>>
>>>>  if (dss->type == LIBXL_DOMAIN_TYPE_HVM) {
>>>>  stream->device_model_version =
>>>>  libxl__device_model_version_running(gc, dss->domid);
>>>> @@ -249,12 +265,7 @@ void libxl__stream_write_start(libxl__egc *egc,
>

Re: [Xen-devel] [PATCH v10 22/31] implement the cmdline for COLO

[Wen Congyang]

On 03/02/2016 11:03 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:26AM +0800, Wen Congyang wrote:
> [...]
>> +if (libxl_defbool_val(info->colo)) {
>> +if (libxl_defbool_val(info->compression)) {
> 
> This can be simplified as
> 
>if (libxl_defbool_val(xxx) && libxl_defbool_val(yyy))

OK. will fix it in the next version.

> 
>> +LOG(ERROR, "cannot use memory checkpoint compression in COLO 
>> mode");
>> +rc = ERROR_FAIL;
>> +goto out;
>> +}
>> +}
>> +
>>  if (!libxl_defbool_val(info->allow_unsafe) &&
>>  (libxl_defbool_val(info->blackhole) ||
>>   !libxl_defbool_val(info->netbuf) ||
>> @@ -876,7 +892,10 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
>> libxl_domain_remus_info *info,
>>  dss->live = 1;
>>  dss->debug = 0;
>>  dss->remus = info;
>> -dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
>> +if (libxl_defbool_val(info->colo))
>> +dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_COLO;
>> +else
>> +dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
>>  
>>  assert(info);
>>  
>> diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
>> index df7268b..0dc7220 100644
>> --- a/tools/libxl/xl_cmdimpl.c
>> +++ b/tools/libxl/xl_cmdimpl.c
>> @@ -4440,6 +4440,8 @@ static void migrate_receive(int debug, int daemonize, 
>> int monitor,
>>  char rc_buf;
>>  char *migration_domname;
>>  struct domain_create dom_info;
>> +const char *ha = checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO ?
>> + "COLO" : "Remus";
>>  
>>  signal(SIGPIPE, SIG_IGN);
>>  /* if we get SIGPIPE we'd rather just have it as an error */
>> @@ -4460,6 +4462,9 @@ static void migrate_receive(int debug, int daemonize, 
>> int monitor,
>>  dom_info.send_back_fd = send_fd;
>>  dom_info.migration_domname_r = &migration_domname;
>>  dom_info.checkpointed_stream = checkpointed;
>> +if (checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO)
>> +/* COLO uses stdout to send control message to master */
>> +dom_info.quiet = 1;
>>  
> 
> It seems that dom_info->quiet affects stderr, not stdout. See the only
> place that checks this in xl_cmdimpl.c.
> 
>>  rc = create_domain(&dom_info);
>>  if (rc < 0) {
>> @@ -4472,11 +4477,12 @@ static void migrate_receive(int debug, int 
>> daemonize, int monitor,
>>  
>>  switch (checkpointed) {
>>  case LIBXL_CHECKPOINTED_STREAM_REMUS:
>> +case LIBXL_CHECKPOINTED_STREAM_COLO:
>>  /* If we are here, it means that the sender (primary) has crashed.
>>   * TODO: Split-Brain Check.
>>   */
>> -fprintf(stderr, "migration target: Remus Failover for domain %u\n",
>> -domid);
>> +fprintf(stderr, "migration target: %s Failover for domain %u\n",
>> +ha, domid);
>>  
>>  /*
>>   * If domain renaming fails, lets just continue (as we need the 
>> domain
>> @@ -4492,16 +4498,20 @@ static void migrate_receive(int debug, int 
>> daemonize, int monitor,
>>  rc = libxl_domain_rename(ctx, domid, migration_domname,
>>   common_domname);
>>  if (rc)
>> -fprintf(stderr, "migration target (Remus): "
>> +fprintf(stderr, "migration target (%s): "
>>  "Failed to rename domain from %s to %s:%d\n",
>> -migration_domname, common_domname, rc);
>> +ha, migration_domname, common_domname, rc);
>>  }
>>  
>> +if (checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO)
>> +/* The guest is running after failover in COLO mode */
>> +exit(rc ? -ERROR_FAIL: 0);
>> +
>>  rc = libxl_domain_unpause(ctx, domid);
>>  if (rc)
>> -fprintf(stderr, "migration target (Remus): "
>> +fprintf(stderr, "migration target (%s): "
>>  "Failed to unpause domain %s (id: %u):%d\n",
>> -common_domname, domid, rc);
>> +ha, common_domname, domid, rc);
>>  
>>  exit(rc ? -ERROR_FAIL: 0);

Re: [Xen-devel] [PATCH v10 24/31] Support colo mode for qemu disk

[Wen Congyang]

On 03/02/2016 11:04 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:28AM +0800, Wen Congyang wrote:
>> Usage: disk = 
>> ['...,colo,colo-host=xxx,colo-port=xxx,colo-export=xxx,active-disk=xxx,hidden-disk=xxx...']
>> For QEMU block replication details:
>> http://wiki.qemu.org/Features/BlockReplication
>>
>> Signed-off-by: Wen Congyang 
>> Signed-off-by: Yang Hongyang 
>> ---
>>  docs/man/xl.pod.1   |   2 +-
>>  docs/misc/xl-disk-configuration.txt |  50 ++
>>  tools/libxl/libxl.c |  62 +++-
>>  tools/libxl/libxl_create.c  |  25 -
>>  tools/libxl/libxl_device.c  |  54 +++
>>  tools/libxl/libxl_dm.c  | 184 
>> ++--
>>  tools/libxl/libxl_types.idl |   7 ++
>>  tools/libxl/libxlu_disk_l.l |   7 ++
>>  8 files changed, 382 insertions(+), 9 deletions(-)
>>
>> diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
>> index 1c6dd87..4f1901d 100644
>> --- a/docs/man/xl.pod.1
>> +++ b/docs/man/xl.pod.1
>> @@ -454,7 +454,7 @@ N.B: Remus support in xl is still in experimental 
>> (proof-of-concept) phase.
>>   Disk replication support is limited to DRBD disks.
>>  
>>   COLO support in xl is still in experimental (proof-of-concept) phase.
>> - There is no support for network or disk at the moment.
>> + There is no support for network at the moment.
> 
> You need some document here for the syntax, otherwise users have no clue
> how to configure disk replicate support. I also won't be able to
> meaningfully review this patch without a reference.

OK. will fix it in the next version.

> 
>>  
>>  B
>>  
>> diff --git a/docs/misc/xl-disk-configuration.txt 
>> b/docs/misc/xl-disk-configuration.txt
>> index 29f6ddb..6f23c2d 100644
>> --- a/docs/misc/xl-disk-configuration.txt
>> +++ b/docs/misc/xl-disk-configuration.txt
>> @@ -234,6 +234,56 @@ were intentionally created non-sparse to avoid 
>> fragmentation of the
>>  file.
>>  
>>  
> 
> Some nitpicking about the format below.
> 
>> +===
>> +COLO PARAMETERS
>> +===
>> +
>> +
>> +colo
>> +
>> +
>> +Enable COLO HA for disk. For better understanding block replication on
>> +QEMU, please refer to:
>> +http://wiki.qemu.org/Features/BlockReplication
>> +
>> +
>> +colo-host
>> +-
> 
> Blank line here please.
> 
>> +Description:   Secondary host's address
>> +Mandatory: Yes when COLO enabled
>> +
>> +
>> +colo-port
>> +-
> 
> Ditto.
> 
>> +Description:   Secondary port
>> +   We will run a nbd server on secondary host,
>> +   and the nbd server will listen this port.
>> +Mandatory: Yes when COLO enabled
>> +
>> +
>> +colo-export
>> +-
> 
> Here as well. And some more "-"s to match "colo-export".
> 
>> +Description:   We will run a nbd server on secondary host,
>> +   exportname is the nbd server's disk export name.
>> +Mandatory: Yes when COLO enabled
>> +
>> +
>> +active-disk
>> +---
>> +
>> +Description:   This is used by secondary. Secondary guest's write
>> +   will be buffered in this disk.
>> +Mandatory: Yes when COLO enabled
>> +
>> +
>> +hidden-disk
>> +---
>> +
>> +Description:   This is used by secondary. It buffers the original
>> +   content that is modified by the primary VM.
>> +Mandatory: Yes when COLO enabled
>> +
>> +
> 
> The rest of the patch is mainly for manipulating QEMU parameters. I've
> skipped it for now.

If you want to know about how qemu block repication works, you can see:
http://wiki.qemu.org/Features/BlockReplication

> 
>>  
>>  DEPRECATED PARAMETERS, PREFIXES AND SYNTAXES
>>  
>> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
>> index 12df81a..f691628 100644
>> --- a/tools/libxl/libxl.c
>> +++ b/tools/libxl/libxl.c
>> @@ -2309,6 +2309,8 @@ int libxl__device_disk_setdefault(libxl__gc *gc, 
>> libxl_device_disk *disk)
>>  int rc;
>>  
>>  libxl_defbool_setdefault(&disk->discard_enable, !

Re: [Xen-devel] [PATCH v10 31/31] cmdline switches and config vars to control colo-proxy

[Wen Congyang]

On 03/02/2016 11:05 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:35AM +0800, Wen Congyang wrote:
>> Add cmdline switches to 'xl migrate-receive' command to specify
>> a domain-specific hotplug script to setup COLO proxy.
>>
>> Add a new config var 'colo.default.agentscript' to xl.conf, that
>> allows the user to override the default global script used to
>> setup COLO proxy.
>>
>> Signed-off-by: Yang Hongyang 
>> Signed-off-by: Wen Congyang 
>> ---
>>  docs/man/xl.conf.pod.5  |  6 ++
>>  docs/man/xl.pod.1   |  1 -
>>  tools/libxl/libxl.c |  6 ++
>>  tools/libxl/libxl_create.c  | 14 --
>>  tools/libxl/libxl_types.idl |  1 +
>>  tools/libxl/xl.c|  3 +++
>>  tools/libxl/xl.h|  1 +
>>  tools/libxl/xl_cmdimpl.c| 47 
>> ++---
>>  8 files changed, 65 insertions(+), 14 deletions(-)
>>
>> diff --git a/docs/man/xl.conf.pod.5 b/docs/man/xl.conf.pod.5
>> index 8ae19bb..8f7fd28 100644
>> --- a/docs/man/xl.conf.pod.5
>> +++ b/docs/man/xl.conf.pod.5
>> @@ -111,6 +111,12 @@ Configures the default script used by Remus to setup 
>> network buffering.
>>  
>>  Default: C
>>  
>> +=item B
>> +
>> +Configures the default script used by COLO to setup colo-proxy.
>> +
>> +Default: C
>> +
>>  =item B
>>  
>>  Configures the default output format used by xl when printing "machine
>> diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
>> index 4f1901d..edeafcf 100644
>> --- a/docs/man/xl.pod.1
>> +++ b/docs/man/xl.pod.1
>> @@ -454,7 +454,6 @@ N.B: Remus support in xl is still in experimental 
>> (proof-of-concept) phase.
>>   Disk replication support is limited to DRBD disks.
>>  
>>   COLO support in xl is still in experimental (proof-of-concept) phase.
>> - There is no support for network at the moment.
> 
> 
> Same here, missing documentation on how to use the new parameters (if
> any). Please provide adequate documentation otherwise we can't
> meaningfully review the rest of this patch.

OK, will fix it in the next version.

Thanks
Wen Congyang

> 
> Wei.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Patching error while setting up COLO

[Wen Congyang]

On 03/03/2016 05:39 PM, Yu-An(Victor) Chen wrote:
> Hi Changlong,
> 
> Thanks for the reply,
> 
> Again when I am trying to do the following:
> 
> 5. build qemu-colo
> 1) cd ~/qemu-colo/; *git checkout colo-xen-v2*
> *
> *
> I got this error message *"error: pathspec 'colo-xen-v2' did not match any 
> file(s) known to git."* Even if I do git fetch, I still get the same error.
> 
> the qemu-colo I cloned from is provided by you 
> https://github.com/wencongyang/qemu-colo.git

https://github.com/wencongyang/qemu-xen, not qemu-colo

> 
> Thank you!
> 
> Victor
> 
> 
> 
> On Thu, Feb 25, 2016 at 9:07 PM, Changlong Xie  > wrote:
> 
> On 02/26/2016 12:55 PM, Yu-An(Victor) Chen wrote:
> 
> Hi Changlong,
> 
> Are you suggesting I should hold off on setting up COLO for now?
> 
> 
> No, just following my steps.
> 
> Thanks
> -Xie
> 
> 
> Thanks!
> 
> Victor
> 
> On Thu, Feb 25, 2016 at 8:19 PM, Changlong Xie 
> mailto:xiecl.f...@cn.fujitsu.com>>
> wrote:
> 
> On 02/26/2016 11:38 AM, Yu-An(Victor) Chen wrote:
> 
> Hi Changlong,
> 
> Thanks for the reply!
> 
> So I am trying to follow your new instructions, but when I am 
> trying to do
> this:
> 
>cd ~/colo-proxy/; git checkout 405527cbfa9f
> 
> I got the following error:
> 
> "error: pathspec '405527cbfa9f' did not match any file(s) 
> known to git."
> 
> I assume it is just a typo? Thank you!
> 
> 
> Hi victor
> 
> Please git clone
> 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Pating_colo-2Dproxy_tree_changlox&d=CwICaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=IitX1U91-NhsQt0q4MJOLQ&m=pCAkg_8tEQmGEZZoUlyePZjK7z-6aEmp-n6UrQRLWo4&s=Ww-EAIszC-zQuVcDc4XpigwVbMG_4t2SpTg2PV6HTjM&e=
> *Notice* that, currently we implement colo proxy as a kernel 
> module what
> is a temporary measure. But further more we'll intergrate it in 
> qemu and
> drop this one, so both qemu-colo and xen-colo will share the same 
> proxy.
> Please don't test this colo proxy now, there maybe some bugs, but 
> it's
> acceptable.
> 
> Thanks
>  -Xie
> 
> 
> Victor
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] Patching error while setting up COLO

[Wen Congyang]

On 03/04/2016 10:01 AM, Yu-An(Victor) Chen wrote:
> Hi,
> 
> So I git clone https://github.com/wencongyang/qemu-xen.git
> 
> but i only see branch "con-xen-v2" instead of " colo-xen-v2" so I assume I 
> use just use con-xen-v2.
> 
> But then the following step:
> 
> in both ~/qemu-colo and ~/qemu-xen
> 
> ./configure --enable-xen --target-list=x86_64-softmmu 
> --extra-cflags="-I$path_to_xen_source/tools/include 
> -I$path_to_xen_source/tools/libxc -I$path_to_xen_source/tools/xenstore" 
> --extra-ldflags="-L$path_to_xen_source/tools/libxc 
> -L$path_to_xen_source/tools/xenstore"


This command line is out of dated. The following is my building scripts:
#! /bin/bash

path_to_xen_source=/work/src/xen
#./configure --enable-xen --target-list=i386-softmmu \
#--extra-cflags="-I$path_to_xen_source/tools/include 
-I$path_to_xen_source/tools/libxc/include 
-I$path_to_xen_source/tools/xenstore/include" \
#--extra-ldflags="-L$path_to_xen_source/tools/libxc 
-L$path_to_xen_source/tools/xenstore"

extra_cflags=""
extra_cflags+=" -DXC_WANT_COMPAT_EVTCHN_API=1"
extra_cflags+=" -DXC_WANT_COMPAT_GNTTAB_API=1"
extra_cflags+=" -DXC_WANT_COMPAT_MAP_FOREIGN_API=1"
extra_cflags+=" -I$path_to_xen_source/tools/include"
extra_cflags+=" -I$path_to_xen_source/tools/libs/toollog/include"
extra_cflags+=" -I$path_to_xen_source/tools/libs/evtchn/include"
extra_cflags+=" -I$path_to_xen_source/tools/libs/gnttab/include"
extra_cflags+=" -I$path_to_xen_source/tools/libs/foreignmemory/include"
extra_cflags+=" -I$path_to_xen_source/tools/libxc/include"
extra_cflags+=" -I$path_to_xen_source/tools/xenstore/include"
extra_cflags+=" -I$path_to_xen_source/tools/xenstore/compat/include"
extra_cflags+=" "

extra_ldflags=""
extra_ldflags+=" -L$path_to_xen_source/tools/libxc"
extra_ldflags+=" -L$path_to_xen_source/tools/xenstore"
extra_ldflags+=" -L$path_to_xen_source/tools/libs/evtchn"
extra_ldflags+=" -L$path_to_xen_source/tools/libs/gnttab"
extra_ldflags+=" -L$path_to_xen_source/tools/libs/foreignmemory"
extra_ldflags+=" -Wl,-rpath-link=$path_to_xen_source/tools/libs/toollog"
extra_ldflags+=" -Wl,-rpath-link=$path_to_xen_source/tools/libs/evtchn"
extra_ldflags+=" -Wl,-rpath-link=$path_to_xen_source/tools/libs/gnttab"
extra_ldflags+=" -Wl,-rpath-link=$path_to_xen_source/tools/libs/call"
extra_ldflags+=" -Wl,-rpath-link=$path_to_xen_source/tools/libs/foreignmemory"
extra_ldflags+=" "

./configure --enable-xen --target-list=i386-softmmu \
--extra-cflags="$extra_cflags" \
--extra-ldflags="$extra_ldflags"

if [[ $? -ne 0 ]]; then
exit 1
fi

#make -j8 && make clean
make -j8

You can find the newest building way in tools/Makefile(xen's codes):
subdir-all-qemu-xen-dir: qemu-xen-dir-find  
if test -d $(QEMU_UPSTREAM_LOC) ; then \
source=$(QEMU_UPSTREAM_LOC); \  
else \  
source=.; \ 
fi; \   
cd qemu-xen-dir; \  
if $$source/scripts/tracetool.py --check-backend --backend stderr ; 
then \
enable_trace_backend='--enable-trace-backend=stderr'; \ 
else \  
enable_trace_backend='' ; \ 
fi ; \  
$$source/configure --enable-xen --target-list=i386-softmmu \        
$(QEMU_XEN_ENABLE_DEBUG) \  
$$enable_trace_backend \
--prefix=$(LIBEXEC) \   
--libdir=$(LIBEXEC_LIB) \   
--includedir=$(LIBEXEC_INC) \


Thanks
Wen Congyang

> 
> 
> I got the following error message:
> 
> "ERROR: User requested feature xen
>configure was not able to find it.
>Install xen devel"
> 
> I found out the the error came from just simply doing this:
> 
> ./configure --enable-xen  
> 
> I am thinking the reason is because I did this step wrong:
> 
> "path_to_xen_source=~/xen"
> 

Re: [Xen-devel] Patching error while setting up COLO

[Wen Congyang]

On 03/05/2016 09:51 AM, Yu-An(Victor) Chen wrote:
> Hi Congyang,
> 
> Thanks for your reply,
> 
> even with your script, and I modify the "path_to_xen_source" to point where 
> my xen directory is. I still got this error.
> 
> ERROR: User requested feature xen
>configure was not able to find it.
>Install xen devel
> 
> What do you think what I am missing? Thank you!

Do you build xen before?

Thanks
Wen Congyang

> 
> Victor
> 
> 
> 
> On Thu, Mar 3, 2016 at 6:15 PM, Wen Congyang  <mailto:we...@cn.fujitsu.com>> wrote:
> 
> On 03/04/2016 10:01 AM, Yu-An(Victor) Chen wrote:
> > Hi,
> >
> > So I git clone 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_wencongyang_qemu-2Dxen.git&d=CwICaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=IitX1U91-NhsQt0q4MJOLQ&m=4j1T2HKL4uKodf62b4Tz1XtOvX81uAqCqfOcD90CRAY&s=s0fo5ej8_vZ1PmOkDCuyIroS5Zi_KpDSHI8jqodSmrg&e=
> >
> > but i only see branch "con-xen-v2" instead of " colo-xen-v2" so I 
> assume I use just use con-xen-v2.
> >
> > But then the following step:
> >
> > in both ~/qemu-colo and ~/qemu-xen
> >
> > ./configure --enable-xen --target-list=x86_64-softmmu 
> --extra-cflags="-I$path_to_xen_source/tools/include 
> -I$path_to_xen_source/tools/libxc -I$path_to_xen_source/tools/xenstore" 
> --extra-ldflags="-L$path_to_xen_source/tools/libxc 
> -L$path_to_xen_source/tools/xenstore"
> 
> 
> This command line is out of dated. The following is my building scripts:
> #! /bin/bash
> 
> path_to_xen_source=/work/src/xen
> #./configure --enable-xen --target-list=i386-softmmu \
> #--extra-cflags="-I$path_to_xen_source/tools/include 
> -I$path_to_xen_source/tools/libxc/include 
> -I$path_to_xen_source/tools/xenstore/include" \
> #--extra-ldflags="-L$path_to_xen_source/tools/libxc 
> -L$path_to_xen_source/tools/xenstore"
> 
> extra_cflags=""
> extra_cflags+=" -DXC_WANT_COMPAT_EVTCHN_API=1"
> extra_cflags+=" -DXC_WANT_COMPAT_GNTTAB_API=1"
> extra_cflags+=" -DXC_WANT_COMPAT_MAP_FOREIGN_API=1"
> extra_cflags+=" -I$path_to_xen_source/tools/include"
> extra_cflags+=" -I$path_to_xen_source/tools/libs/toollog/include"
> extra_cflags+=" -I$path_to_xen_source/tools/libs/evtchn/include"
> extra_cflags+=" -I$path_to_xen_source/tools/libs/gnttab/include"
> extra_cflags+=" -I$path_to_xen_source/tools/libs/foreignmemory/include"
> extra_cflags+=" -I$path_to_xen_source/tools/libxc/include"
> extra_cflags+=" -I$path_to_xen_source/tools/xenstore/include"
> extra_cflags+=" -I$path_to_xen_source/tools/xenstore/compat/include"
> extra_cflags+=" "
> 
> extra_ldflags=""
> extra_ldflags+=" -L$path_to_xen_source/tools/libxc"
> extra_ldflags+=" -L$path_to_xen_source/tools/xenstore"
> extra_ldflags+=" -L$path_to_xen_source/tools/libs/evtchn"
> extra_ldflags+=" -L$path_to_xen_source/tools/libs/gnttab"
> extra_ldflags+=" -L$path_to_xen_source/tools/libs/foreignmemory"
> extra_ldflags+=" -Wl,-rpath-link=$path_to_xen_source/tools/libs/toollog"
> extra_ldflags+=" -Wl,-rpath-link=$path_to_xen_source/tools/libs/evtchn"
> extra_ldflags+=" -Wl,-rpath-link=$path_to_xen_source/tools/libs/gnttab"
> extra_ldflags+=" -Wl,-rpath-link=$path_to_xen_source/tools/libs/call"
> extra_ldflags+=" 
> -Wl,-rpath-link=$path_to_xen_source/tools/libs/foreignmemory"
> extra_ldflags+=" "
> 
> ./configure --enable-xen --target-list=i386-softmmu \
> --extra-cflags="$extra_cflags" \
> --extra-ldflags="$extra_ldflags"
> 
> if [[ $? -ne 0 ]]; then
> exit 1
> fi
> 
> #make -j8 && make clean
> make -j8
> 
> You can find the newest building way in tools/Makefile(xen's codes):
> subdir-all-qemu-xen-dir: qemu-xen-dir-find
> if test -d $(QEMU_UPSTREAM_LOC) ; then \
> source=$(QEMU_UPSTREAM_LOC); \
> else \
> source=.; \
> fi; \
> cd qemu-xen-dir; \
> if $$source/scripts/tracetool.py --check-backend --backend stderr 
> ; then \
> enable_trace_backend='--enable-trace-backend=stderr'; \
> else \
> 

Re: [Xen-devel] [PATCH v11 20/27] Support colo mode for qemu disk

[Wen Congyang]

On 03/05/2016 01:44 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 20/27] Support colo mode for qemu disk"):
>> From: Wen Congyang 
>>
>> Usage: disk = 
>> ['...,colo,colo-host=xxx,colo-port=xxx,colo-export=xxx,active-disk=xxx,hidden-disk=xxx...']
>> For QEMU block replication details:
>> http://wiki.qemu.org/Features/BlockReplication
> 
> So now I am slightly confused by the design, I think.
> 
> When you replicate a VM with COLO using xl, its memory state is
> transferred over ssh.  But its disk replication is done unencrypted
> and unauthenticated ?

Yes, it is a problem. I will think how to improve it.

> 
> And the disk replication is, out of band, and needs to be configured
> separately ?  This is rather awkward, although maybe not a
> showstopper.  (Maybe we can have a plan to fix it in the future...)

colo-host,colo-port should be the global configuration. And colo-export,
active-disk,hidden-disk must be configured separately, because each
disk should have a different configuration.

> 
> And, how does the disk replication, which doesn't depend on there
> being xl running, relate to the vm state replication, which does ?  I
> think at the very least I'd like to see some information about the
> principles of operation - either explained, or referred to, in the
> user manual.

OK. The disk replication doesn't depend on xl. We only can operate it
via qemu monitor command:
1. stop the vm
2. do the checkpoint
3. start the vm
1/3 is suspend/resume the guest. We only need to do 2 when both vm are
in the consistent state.

> 
> Is it possible to use COLO with an existing full-service disk
> replication service such as DRBD ?

DRBD doesn's support the case like COLO. Because both primary guest
and secondary guest need to write to the disk.

> 
>> +(a) An example for COLO replication's configuration: disk 
>> =['...,colo,colo-host
>> +=xxx,colo-port=xxx,colo-export=xxx,active-disk=xxx,hidden-disk=xxx...']
>> +
>> +=item B  :Secondary host's ip address.
>> +
>> +=item B  :Secondary host's port, we will run a nbd server on
>> +secondary host, and the nbd server will listen this port.
>> +
>> +=item B:Nbd server's disk export name of secondary host.
>> +
>> +=item B:Secondary's guest write will be buffered in this 
>> disk,
>> +and it's used by secondary.
>> +
>> +=item B:Primary's modified contents will be buffered in 
>> this
>> +disk, and it's used by secondary.
> 
> What would a typical configuration look like ?  I don't understand the
> relationship between active-disk and hidden-disk, etc.

QEMU has a feature: backing file
For example: A's backing file is B
1. If we read from A, but the sector is not allocated in A. We wil return a zero
   sector to the guest. If A has a backing file, we will read the sector from B
   instead of returning a zero sector.
2. The backing file doesn't affect the write operation.

QEMU has another feature: backup block job
Backup job has two file: one is source and another is the target. It has some 
running
mode. For block replication, we use the mode "sync=none". In this mode, we will 
read
the data from the source disk before we modify it, and write it to the target 
disk.
We keep a bitmap to remeber which sector is backuped from the source disk to the
target disk. If the target disk is an empty disk, and empty disk's backing file 
is
the source disk, we can read from the target disk to get the source disk's 
originnal data.


How does block replication work:
A. primary qemu:
1. use the block driver quorum: it will read from all children and write to all 
children.
   child 0: real disk
   child 1: nbd client
   reading from child 1 will fail, but we use the fifo mode. In this mode, we 
read from
   child 0 will success and we don't read from child 0
   write to child 1: because child 1 is nbd client, it will forward the write 
request to
   nbd server

B. secondary qemu:
We have 3 disks: active disk(called it A), hidden disk(called it H), and 
secondary disk
(real disk, called it S).
A's backing file is H, and H's backing file is S.
We also start a backup job: the source disk is S, and the target disk is H.
we run nbd server in secondary qemu. And the nbd server will write to S.

Before resuming both primary vm and secondary vm: the state is:
1. primary disk and secondary disk are in the consistent state(contain the same 
data)
2. active disk and hidden disk are the empty disk
When the guest is running:
1. NBD server receives the primary write operation and writes the data to S
2. Before we write data to S, the backup job will read the original data and 
backup it
   to H
3. The secondary

Re: [Xen-devel] [PATCH v11 20/27] Support colo mode for qemu disk

[Wen Congyang]

On 03/05/2016 04:30 AM, Konrad Rzeszutek Wilk wrote:
> On Fri, Mar 04, 2016 at 05:52:09PM +, Ian Jackson wrote:
>> Changlong Xie writes ("[PATCH v11 20/27] Support colo mode for qemu disk"):
>>> +Enable COLO HA for disk. For better understanding block replication on
>>> +QEMU, please refer to:
>>> +http://wiki.qemu.org/Features/BlockReplication
>>
>> Sorry, I missed this link on my first pass.  I still think that at the
>> very least this needs something more user-facing (ie, how should one
>> set this up).
>>
>> But, I'm kind of worried that qemu is the wrong place to be doing
>> this.
>>
>> How can this be made to work with PV guests ?
> 
> QEMU can also serve PV guests (qdisk).
> 
> I think your question is more of - what about making this work with
> PV block backend?

I don't know how to work with PV block backend. It is one reason that
why we only support pure HVM now.
For PV block backend, there is also other problem. For exampe resuming
it in the secondary side is very slow, because we need to disconnect and
reconnect.

Thanks
Wen Congyang

>>
>> What if an HVM guest has PV-on-HVM drivers ?  In this case there might
>> be two relevant qemus, one for the qdisk Xen PV block backend, and one
>> for the emulated IDE.
> 
> In both cases QEMU would use the same underlaying API to actually write/read
> out the blocks. That API would then use NBD, etc to replicate writes.
> 
> Maybe a little ASCII art?
> 
>   qdisk  ide
> \/
>\  /
>block API
> |
>QCOW2
> |
>NBD
> 
> Or such?
> 
>>
>> I don't understand how discrepant writes are detected.  Surely they
>> might occur and should trigger a resynch ?
>>
>> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 10/27] tools/libxl: add back channel support to write stream

[Wen Congyang]

On 03/05/2016 01:00 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 10/27] tools/libxl: add back channel 
> support to write stream"):
>> From: Wen Congyang 
>>
>> Add back channel support to write stream. If the write stream is
>> a back channel stream, this means the write stream is used by
>> Secondary to send some records back.
> 
> The general idea seems fine to me but I want an opinion from Andrew.
> 
> If I'm not mistaken there is no call site for this yet.  In which case
> this should be mentioned in the commit message.
> 
>> +/*- checkpoint state -*/
>> +void libxl__stream_write_checkpoint_state(libxl__egc *egc,
>> +  libxl__stream_write_state *stream,
>> +  libxl_sr_checkpoint_state *srcs)
> 
> Firstly, missing blank line.
> 
> Secondly, reading all this leads me to think that maybe the
> `checkpoint_state' record should be called something different.  Is it
> only ever going to be used for COLO ?  Maybe it should be

Yes, it is only used for COLO now.

> `COLOHA_STATE' or something (and all the functions etc. renamed
> consequently) ?
> 
> What do you think ?

COLO is FT, not HA. What aboyt COLOFT_STATE?

Thanks
Wen Congyang

> 
> Thanks,
> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 12/27] tools/libx{l, c}: introduce wait_checkpoint callback

[Wen Congyang]

On 03/05/2016 04:23 AM, Konrad Rzeszutek Wilk wrote:
> On Fri, Mar 04, 2016 at 05:03:16PM +, Ian Jackson wrote:
>> Changlong Xie writes ("[PATCH v11 12/27] tools/libx{l,c}: introduce 
>> wait_checkpoint callback"):
>>> From: Wen Congyang 
>>>
>>> Under COLO, we are doing checkpoint on demand, if this
>>> callback returns 1, we will take another checkpoint.
>>> 0 indicates unexpected error.
>>
>> This doesn't seem to have a corresponding implementation.  I think the
>> implementation ought to be in the same patch.
>>
>> If 0 is always an `unexpected error', perhaps the return value should
>> be an error code or something ?  I'm not sure.
> 
> I struggled with this API.
> 
> I like the idea of that negative value would imply 'unexpected error'.
> And 1 for 'OK, take another checkpoint'. Not sure if zero would be a valid
> return value..

IIRC, save/restore callback always use 0 for unexpected error, 1 for OK.
negative value for pipe is broken.

Thanks
Wen Congyang

> 
> 
>>
>> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 14/27] secondary vm suspend/resume/checkpoint code

[Wen Congyang]

On 03/05/2016 01:11 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 14/27] secondary vm 
> suspend/resume/checkpoint code"):
>> From: Wen Congyang 
>>
>> Secondary vm is running in colo mode. So we will do
>> the following things again and again:
> 
> I don't propose to review this in detail.  Skimreading it, it looks
> plausible.  I don't think a detailed review is needed.
> 
> I will review the changes to the core code.
> 
>> diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
>> index 5d9f497..2bfed64 100644
>> --- a/tools/libxc/xc_sr_common.h
>> +++ b/tools/libxc/xc_sr_common.h
>> @@ -184,10 +184,12 @@ struct xc_sr_context
>>   * migration stream
>>   * 0: Plain VM
>>   * 1: Remus
>> + * 2: COLO
>>   */
>>  enum {
>>  MIG_STREAM_NONE, /* plain stream */
>>  MIG_STREAM_REMUS,
>> +MIG_STREAM_COLO,
> 
> I think this shows that the duplicated list (in the comment, above the
> enum) is a mistake.  I would prefer it to be removed.

Do you mean remove the comments?

> 
>> +/* = colo: common functions = */

Add a blank line here? Will fix it in the next version.

>> +static void colo_enable_logdirty(libxl__colo_restore_state *crs, libxl__egc 
>> *egc)
> 
> Here's another missing blank line.  This seems to be a general theme:
> can you change this everywhere ?  Thanks.
> 
>> @@ -994,6 +1011,8 @@ static void domcreate_bootloader_done(libxl__egc *egc,
>>  const int restore_fd = dcs->restore_fd;
>>  libxl__domain_build_state *const state = &dcs->build_state;
>>  const int checkpointed_stream = dcs->restore_params.checkpointed_stream;
>> +libxl__colo_restore_state *const crs = &dcs->crs;
>> +libxl_domain_build_info *const info = &d_config->b_info;
>>  
>>  if (rc) {
>>  domcreate_rebuild_done(egc, dcs, rc);
>> @@ -1022,6 +1041,13 @@ static void domcreate_bootloader_done(libxl__egc *egc,
>>  
>>  /* Restore */
>>  
>> +/* COLO only supports HVM now */
>> +if (info->type != LIBXL_DOMAIN_TYPE_HVM &&
>> +checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_COLO) {
>> +rc = ERROR_FAIL;
>> +goto out;
> 
> Please log something here, or it may be very mysterious.

OK. Will add some comments to explain why only support pure HVM now.

> 
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index 6307b71..48b4858 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -87,6 +87,8 @@
>> @@ -3468,7 +3464,6 @@ libxl__stream_read_inuse(const 
>> libxl__stream_read_state *stream)
>>  return stream->running;
>>  }
>>  
>> -
>>  struct libxl__domain_create_state {
>>  /* filled in by user */
>>  libxl__ao *ao;
>> @@ -3484,6 +3479,8 @@ struct libxl__domain_create_state {
> 
> Unintentional whitespace change.

Sorry for the mistake. Will fix it in the next version.

Thanks
Wen Congyang

> 
> 
> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 15/27] primary vm suspend/resume/checkpoint code

[Wen Congyang]

On 03/05/2016 01:14 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 15/27] primary vm suspend/resume/checkpoint 
> code"):
>> From: Wen Congyang 
> 
> I would look at this on the same basis as the previous patch.
> 
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index 48b4858..5160939 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
> 
>> +struct libxl__stream_read_state {
>> +/* filled by the user */
>> +libxl__ao *ao;
>> +libxl__domain_create_state *dcs;
>> +int fd;
>> +bool legacy;
> 
> Can you please split out this code motion into a separate patch ?
> As it is it is very difficult to review.

OK, will fix it in the next version.

Thanks
Wen Congyang

> 
> Thanks,
> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 17/27] libxc/save: support COLO save

[Wen Congyang]

On 03/05/2016 01:18 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 17/27] libxc/save: support COLO save"):
>> From: Wen Congyang 
>>
>> After suspend primary vm, get dirty bitmap on secondary vm,
>> and send pages both dirty on primary/secondary to secondary.
> 
> This patch again seems like a plausible kind of thing.  Again, I'd
> like to hear from Andrew.
> 
>> +static int merge_secondary_dirty_bitmap(struct xc_sr_context *ctx)
>> +{
> 
> This function might want the word `colo' in its name somewhere.

OK, will fix it in the next version.

Thanks
Wen Congyang

> 
> Thanks,
> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 18/27] implement the cmdline for COLO

[Wen Congyang]

On 03/05/2016 01:22 AM, Ian Jackson wrote:
>> --- a/docs/man/xl.pod.1
>> +++ b/docs/man/xl.pod.1
> ...
>> + COLO support in xl is still in experimental (proof-of-concept) phase.
>> + There is no support for network or disk at the moment.
> 
> I think you need to spell out the lack of storage and network handling
> means that the guest will corrupt its disk and confuse its network
> peers.

OK, will fix it in the next version.

> 
>> @@ -875,7 +890,10 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
>> libxl_domain_remus_info *info,
>>  dss->live = 1;
>>  dss->debug = 0;
>>  dss->remus = info;
>> -dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
>> +if (libxl_defbool_val(info->colo))
>> +dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_COLO;
>> +else
>> +dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
> 
> If you prefer, a ? : expression would do here as well.
> 
>  +dss->checkpointed_stream =
>libxl_defbool_val() ? LIBXL_CHECKPOINTED_STREAM_COLO :...

If so, this line will contains more than 80 characters. So I will not
change it.

Thanks
Wen Congyang

> 
> (only completed with sensible formatting).  Up to you - it's fine as
> it is, too.
> 
> Most of this patch looks good to me.
> 
> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH] libxc: move migration_stream's definition to xenguest.h

[Wen Congyang]

xc_save_domain()'s parameter use this type, so it should
be public.

Signed-off-by: Wen Congyang 
---
 tools/libxc/include/xenguest.h | 10 ++
 tools/libxc/xc_sr_common.h | 10 --
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index affc42b..888536e 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -238,4 +238,9 @@ xen_pfn_t *xc_map_m2p(xc_interface *xch,
   unsigned long max_mfn,
   int prot,
   unsigned long *mfn0);
+
+typedef enum {
+MIG_STREAM_NONE, /* plain stream */
+MIG_STREAM_REMUS,
+} migration_stream;
 #endif /* XENGUEST_H */
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 66f595f..e7568b5 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -180,16 +180,6 @@ struct xc_sr_context
 
 xc_dominfo_t dominfo;
 
-/*
- * migration stream
- * 0: Plain VM
- * 1: Remus
- */
-enum {
-MIG_STREAM_NONE, /* plain stream */
-MIG_STREAM_REMUS,
-} migration_stream;
-
 union /* Common save or restore data. */
 {
 struct /* Save data. */
-- 
2.5.0



___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH] libxc: move migration_stream's definition to xenguest.h

[Wen Congyang]

On 03/08/2016 06:38 PM, Andrew Cooper wrote:
> On 08/03/16 05:32, Wen Congyang wrote:
>> xc_save_domain()'s parameter use this type, so it should
>> be public.
> 
> xc_domain_save() currently uses an int, which also needs fixing.

OK. Will fix it in the next version.

> 
>>
>> Signed-off-by: Wen Congyang 
> 
> Does this even compile?  You have removed a variable without any
> replacement.

Yes, I compile it. The variable is not used.

> 
>> ---
>>  tools/libxc/include/xenguest.h | 10 ++
>>  tools/libxc/xc_sr_common.h | 10 --
>>  2 files changed, 10 insertions(+), 10 deletions(-)
>>
>> diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
>> index affc42b..888536e 100644
>> --- a/tools/libxc/include/xenguest.h
>> +++ b/tools/libxc/include/xenguest.h
>> @@ -238,4 +238,9 @@ xen_pfn_t *xc_map_m2p(xc_interface *xch,
>>unsigned long max_mfn,
>>int prot,
>>unsigned long *mfn0);
>> +
>> +typedef enum {
>> +MIG_STREAM_NONE, /* plain stream */
>> +MIG_STREAM_REMUS,
>> +} migration_stream;
> 
> This typedef should be beside xc_domain_save() as that is where it is
> intended to be used.  It also needs xc_ prefixes as it is part of the
> public interface, and a typedef wants a _t suffix.

OK. Will fix it in the next version.

Thanks
Wen Congyang

> 
> ~Andrew
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v2 2/2] tools: change checkpointed_stream's type from int to xc_migration_stream_t

[Wen Congyang]

Signed-off-by: Wen Congyang 
---
 tools/libxc/include/xenguest.h  |  4 ++--
 tools/libxc/xc_nomigrate.c  |  4 ++--
 tools/libxc/xc_sr_restore.c |  2 +-
 tools/libxc/xc_sr_save.c|  2 +-
 tools/libxl/libxl_save_helper.c | 38 +++---
 5 files changed, 25 insertions(+), 25 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index 973e5a6..029082d 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -93,7 +93,7 @@ typedef enum {
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t 
max_iters,
uint32_t max_factor, uint32_t flags /* XCFLAGS_xxx */,
struct save_callbacks* callbacks, int hvm,
-   int checkpointed_stream);
+   xc_migration_stream_t checkpointed_stream);
 
 /* callbacks provided by xc_domain_restore */
 struct restore_callbacks {
@@ -131,7 +131,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
   domid_t store_domid, unsigned int console_evtchn,
   unsigned long *console_mfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
-  int checkpointed_stream,
+  xc_migration_stream_t checkpointed_stream,
   struct restore_callbacks *callbacks);
 
 /**
diff --git a/tools/libxc/xc_nomigrate.c b/tools/libxc/xc_nomigrate.c
index c9124df..77db59c 100644
--- a/tools/libxc/xc_nomigrate.c
+++ b/tools/libxc/xc_nomigrate.c
@@ -23,7 +23,7 @@
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t 
max_iters,
uint32_t max_factor, uint32_t flags,
struct save_callbacks* callbacks, int hvm,
-   int checkpointed_stream)
+   xc_migration_stream_t checkpointed_stream)
 {
 errno = ENOSYS;
 return -1;
@@ -34,7 +34,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t 
dom,
   domid_t store_domid, unsigned int console_evtchn,
   unsigned long *console_mfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
-  int checkpointed_stream,
+  xc_migration_stream_t checkpointed_stream,
   struct restore_callbacks *callbacks)
 {
 errno = ENOSYS;
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index d4d33fd..b2a3558 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -725,7 +725,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
   domid_t store_domid, unsigned int console_evtchn,
   unsigned long *console_gfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
-  int checkpointed_stream,
+  xc_migration_stream_t checkpointed_stream,
   struct restore_callbacks *callbacks)
 {
 struct xc_sr_context ctx =
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index e258b7c..fc51f60 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -830,7 +830,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
uint32_t max_iters, uint32_t max_factor, uint32_t flags,
struct save_callbacks* callbacks, int hvm,
-   int checkpointed_stream)
+   xc_migration_stream_t checkpointed_stream)
 {
 struct xc_sr_context ctx =
 {
diff --git a/tools/libxl/libxl_save_helper.c b/tools/libxl/libxl_save_helper.c
index 6bdcf13..5503311 100644
--- a/tools/libxl/libxl_save_helper.c
+++ b/tools/libxl/libxl_save_helper.c
@@ -246,14 +246,14 @@ int main(int argc, char **argv)
 
 if (!strcmp(mode,"--save-domain")) {
 
-io_fd =atoi(NEXTARG);
-uint32_t dom = strtoul(NEXTARG,0,10);
-uint32_t max_iters =   strtoul(NEXTARG,0,10);
-uint32_t max_factor =  strtoul(NEXTARG,0,10);
-uint32_t flags =   strtoul(NEXTARG,0,10);
-int hvm =  atoi(NEXTARG);
-unsigned cbflags = strtoul(NEXTARG,0,10);
-int checkpointed_stream =  strtoul(NEXTARG,0,10);
+io_fd = atoi(NEXTARG);
+uint32_t dom =  strtoul(NEXTARG,0,10);
+uint32_t max_iters =strtoul(NEXTARG,0,10);
+uint32_t max_factor =   strtoul(NEXTARG,0,10);
+uint32_t flags =strtoul(NEXTARG,0,10);
+int hvm =   atoi(NEXTARG);
+unsign

[Xen-devel] [PATCH v2 1/2] libxc: move migration_stream's definition to xenguest.h

[Wen Congyang]

xc_domain_save() and xc_domain_restore's parameter will use this type,
so it should be public.

Signed-off-by: Wen Congyang 
---
 tools/libxc/include/xenguest.h |  5 +
 tools/libxc/xc_sr_common.h | 10 --
 2 files changed, 5 insertions(+), 10 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index affc42b..973e5a6 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -75,6 +75,11 @@ struct save_callbacks {
 void* data;
 };
 
+typedef enum {
+MIG_STREAM_NONE, /* plain stream */
+MIG_STREAM_REMUS,
+} xc_migration_stream_t;
+
 /**
  * This function will save a running domain.
  *
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 66f595f..e7568b5 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -180,16 +180,6 @@ struct xc_sr_context
 
 xc_dominfo_t dominfo;
 
-/*
- * migration stream
- * 0: Plain VM
- * 1: Remus
- */
-enum {
-MIG_STREAM_NONE, /* plain stream */
-MIG_STREAM_REMUS,
-} migration_stream;
-
 union /* Common save or restore data. */
 {
 struct /* Save data. */
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v3 3/3] libxl: rename checkpointed_stream to stream_type

[Wen Congyang]

Signed-off-by: Wen Congyang 
---
 tools/libxl/libxl.c  |  4 ++--
 tools/libxl/libxl.h  |  8 
 tools/libxl/libxl_create.c   |  4 ++--
 tools/libxl/libxl_dom_save.c |  6 +++---
 tools/libxl/libxl_internal.h |  2 +-
 tools/libxl/libxl_save_callout.c |  4 ++--
 tools/libxl/libxl_stream_read.c  |  4 ++--
 tools/libxl/libxl_stream_write.c |  2 +-
 tools/libxl/libxl_types.idl  |  2 +-
 tools/libxl/xl_cmdimpl.c | 16 
 10 files changed, 30 insertions(+), 22 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 4cdc169..7c2c9fc 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -876,7 +876,7 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
libxl_domain_remus_info *info,
 dss->live = 1;
 dss->debug = 0;
 dss->remus = info;
-dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
+dss->stream_type = LIBXL_CHECKPOINTED_STREAM_REMUS;
 
 assert(info);
 
@@ -937,7 +937,7 @@ int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, 
int fd, int flags,
 dss->type = type;
 dss->live = flags & LIBXL_SUSPEND_LIVE;
 dss->debug = flags & LIBXL_SUSPEND_DEBUG;
-dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_NONE;
+dss->stream_type = LIBXL_CHECKPOINTED_STREAM_NONE;
 
 rc = libxl__fd_flags_modify_save(gc, dss->fd,
  ~(O_NONBLOCK|O_NDELAY), 0,
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index f9e3ef5..86b1d06 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -863,6 +863,14 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, 
libxl_mac *src);
 #define LIBXL_HAVE_SRM_V1 1
 
 /*
+ * LIBXL_HAVE_STREAM_TYPE
+ *
+ * If this is define, then the libxl_domain_create_restore() interfaces;s
+ * parameter checkpointed_stream is renamed to stream_type
+ */
+#define LIBXL_HAVE_STREAM_TYPE 1
+
+/*
  * libxl_domain_build_info has the u.hvm.gfx_passthru_kind field and
  * the libxl_gfx_passthru_kind enumeration is defined.
 */
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index f1028bc..b28eb89 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -993,7 +993,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 libxl_domain_config *const d_config = dcs->guest_config;
 const int restore_fd = dcs->restore_fd;
 libxl__domain_build_state *const state = &dcs->build_state;
-const int checkpointed_stream = dcs->restore_params.checkpointed_stream;
+const int stream_type = dcs->restore_params.stream_type;
 
 if (rc) {
 domcreate_rebuild_done(egc, dcs, rc);
@@ -1033,7 +1033,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 dcs->srs.completion_callback = domcreate_stream_done;
 
 if (restore_fd >= 0) {
-switch (checkpointed_stream) {
+switch (stream_type) {
 case LIBXL_CHECKPOINTED_STREAM_REMUS:
 libxl__remus_restore_setup(egc, dcs);
 /* fall through */
diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index f3288b9..ff92ef0 100644
--- a/tools/libxl/libxl_dom_save.c
+++ b/tools/libxl/libxl_dom_save.c
@@ -338,7 +338,7 @@ void libxl__domain_save(libxl__egc *egc, 
libxl__domain_save_state *dss)
 unsigned int nr_vnodes = 0, nr_vmemranges = 0, nr_vcpus = 0;
 libxl__domain_suspend_state *dsps = &dss->dsps;
 
-if (dss->checkpointed_stream != LIBXL_CHECKPOINTED_STREAM_NONE && !r_info) 
{
+if (dss->stream_type != LIBXL_CHECKPOINTED_STREAM_NONE && !r_info) {
 LOG(ERROR, "Migration stream is checkpointed, but there's no "
"checkpoint info!");
 rc = ERROR_INVAL;
@@ -383,12 +383,12 @@ void libxl__domain_save(libxl__egc *egc, 
libxl__domain_save_state *dss)
 goto out;
 }
 
-if (dss->checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_REMUS) {
+if (dss->stream_type == LIBXL_CHECKPOINTED_STREAM_REMUS) {
 if (libxl_defbool_val(r_info->compression))
 dss->xcflags |= XCFLAGS_CHECKPOINT_COMPRESS;
 }
 
-if (dss->checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_NONE)
+if (dss->stream_type == LIBXL_CHECKPOINTED_STREAM_NONE)
 callbacks->suspend = libxl__domain_suspend_callback;
 
 callbacks->switch_qemu_logdirty = 
libxl__domain_suspend_common_switch_qemu_logdirty;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 005fe53..0aada0d 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3128,7 +3128,7 @@ struct libxl__domain_save_state {
 libxl_domain_type type;
 int live;
 int debug;
-int checkpointed_stream;
+libxl_checkpointed_stream stream_type;
 const libxl_domain_remus_info *remus;
 /* private */
 int rc;
diff --git a/tools/libxl/libxl_save_callout.

[Xen-devel] [PATCH v3 2/3] tools: change checkpointed_stream's type from int to xc_migration_stream_t

[Wen Congyang]

checkpointed_stream is also renamed to stream_type

Signed-off-by: Wen Congyang 
---
v2->v3: Rename checkpointed_stream to stream_type
 tools/libxc/include/xenguest.h  |  8 
 tools/libxc/xc_nomigrate.c  |  4 ++--
 tools/libxc/xc_sr_restore.c | 10 +-
 tools/libxc/xc_sr_save.c|  8 
 tools/libxl/libxl_save_helper.c | 42 -
 5 files changed, 36 insertions(+), 36 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index cf521c3..4f0b06e 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -86,14 +86,14 @@ typedef enum {
  * @parm xch a handle to an open hypervisor interface
  * @parm fd the file descriptor to save a domain to
  * @parm dom the id of the domain
- * @param checkpointed_stream XC_MIG_STREAM_NONE if the far end of the stream
+ * @param stream_type XC_MIG_STREAM_NONE if the far end of the stream
  *doesn't use checkpointing
  * @return 0 on success, -1 on failure
  */
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t 
max_iters,
uint32_t max_factor, uint32_t flags /* XCFLAGS_xxx */,
struct save_callbacks* callbacks, int hvm,
-   int checkpointed_stream);
+   xc_migration_stream_t stream_type);
 
 /* callbacks provided by xc_domain_restore */
 struct restore_callbacks {
@@ -121,7 +121,7 @@ struct restore_callbacks {
  * @parm hvm non-zero if this is a HVM restore
  * @parm pae non-zero if this HVM domain has PAE support enabled
  * @parm superpages non-zero to allocate guest memory with superpages
- * @parm checkpointed_stream non-zero if the far end of the stream is using 
checkpointing
+ * @parm stream_type non-zero if the far end of the stream is using 
checkpointing
  * @parm callbacks non-NULL to receive a callback to restore toolstack
  *   specific data
  * @return 0 on success, -1 on failure
@@ -131,7 +131,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
   domid_t store_domid, unsigned int console_evtchn,
   unsigned long *console_mfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
-  int checkpointed_stream,
+  xc_migration_stream_t stream_type,
   struct restore_callbacks *callbacks);
 
 /**
diff --git a/tools/libxc/xc_nomigrate.c b/tools/libxc/xc_nomigrate.c
index c9124df..08e1f8c 100644
--- a/tools/libxc/xc_nomigrate.c
+++ b/tools/libxc/xc_nomigrate.c
@@ -23,7 +23,7 @@
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t 
max_iters,
uint32_t max_factor, uint32_t flags,
struct save_callbacks* callbacks, int hvm,
-   int checkpointed_stream)
+   xc_migration_stream_t stream_type)
 {
 errno = ENOSYS;
 return -1;
@@ -34,7 +34,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t 
dom,
   domid_t store_domid, unsigned int console_evtchn,
   unsigned long *console_mfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
-  int checkpointed_stream,
+  xc_migration_stream_t stream_type,
   struct restore_callbacks *callbacks)
 {
 errno = ENOSYS;
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index d4d33fd..819401d 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -725,7 +725,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
   domid_t store_domid, unsigned int console_evtchn,
   unsigned long *console_gfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
-  int checkpointed_stream,
+  xc_migration_stream_t stream_type,
   struct restore_callbacks *callbacks)
 {
 struct xc_sr_context ctx =
@@ -739,16 +739,16 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
 ctx.restore.console_domid = console_domid;
 ctx.restore.xenstore_evtchn = store_evtchn;
 ctx.restore.xenstore_domid = store_domid;
-ctx.restore.checkpointed = checkpointed_stream;
+ctx.restore.checkpointed = stream_type;
 ctx.restore.callbacks = callbacks;
 
 /* Sanity checks for callbacks. */
-if ( checkpointed_stream )
+if ( stream_type )
 assert(callbacks->checkpoint);
 
 DPRINTF("fd %d, dom %u, hvm %u, pae %u, superpages %d"
-", checkpointed_stream %d", io_fd, dom, hvm, pae,
-superpages, checkpointed_stream);
+", stream_type %d", io_fd, dom, hvm, pae,
+superpages, stream_type);

[Xen-devel] [PATCH v3 1/3] libxc: move migration_stream's definition to xenguest.h

[Wen Congyang]

xc_domain_save() and xc_domain_restore's parameter will use this type,
so it should be public.

Signed-off-by: Wen Congyang 
---
v2->v3: Rename MIG_STREAM_* to XC_MIG_STREAM_*
 tools/libxc/include/xenguest.h |  7 ++-
 tools/libxc/xc_sr_common.h | 10 --
 tools/libxc/xc_sr_save.c   | 12 ++--
 3 files changed, 12 insertions(+), 17 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index affc42b..cf521c3 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -75,13 +75,18 @@ struct save_callbacks {
 void* data;
 };
 
+typedef enum {
+XC_MIG_STREAM_NONE, /* plain stream */
+XC_MIG_STREAM_REMUS,
+} xc_migration_stream_t;
+
 /**
  * This function will save a running domain.
  *
  * @parm xch a handle to an open hypervisor interface
  * @parm fd the file descriptor to save a domain to
  * @parm dom the id of the domain
- * @param checkpointed_stream MIG_STREAM_NONE if the far end of the stream
+ * @param checkpointed_stream XC_MIG_STREAM_NONE if the far end of the stream
  *doesn't use checkpointing
  * @return 0 on success, -1 on failure
  */
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 66f595f..e7568b5 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -180,16 +180,6 @@ struct xc_sr_context
 
 xc_dominfo_t dominfo;
 
-/*
- * migration stream
- * 0: Plain VM
- * 1: Remus
- */
-enum {
-MIG_STREAM_NONE, /* plain stream */
-MIG_STREAM_REMUS,
-} migration_stream;
-
 union /* Common save or restore data. */
 {
 struct /* Save data. */
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index e258b7c..ab59673 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -629,7 +629,7 @@ static int send_domain_memory_live(struct xc_sr_context 
*ctx)
 if ( rc )
 goto out;
 
-if ( ctx->save.debug && ctx->save.checkpointed != MIG_STREAM_NONE )
+if ( ctx->save.debug && ctx->save.checkpointed != XC_MIG_STREAM_NONE )
 {
 rc = verify_frames(ctx);
 if ( rc )
@@ -758,7 +758,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 
 if ( ctx->save.live )
 rc = send_domain_memory_live(ctx);
-else if ( ctx->save.checkpointed != MIG_STREAM_NONE )
+else if ( ctx->save.checkpointed != XC_MIG_STREAM_NONE )
 rc = send_domain_memory_checkpointed(ctx);
 else
 rc = send_domain_memory_nonlive(ctx);
@@ -778,7 +778,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 if ( rc )
 goto err;
 
-if ( ctx->save.checkpointed != MIG_STREAM_NONE )
+if ( ctx->save.checkpointed != XC_MIG_STREAM_NONE )
 {
 /*
  * We have now completed the initial live portion of the checkpoint
@@ -799,7 +799,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 if ( rc <= 0 )
 goto err;
 }
-} while ( ctx->save.checkpointed != MIG_STREAM_NONE );
+} while ( ctx->save.checkpointed != XC_MIG_STREAM_NONE );
 
 xc_report_progress_single(xch, "End of stream");
 
@@ -845,8 +845,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t 
dom,
 ctx.save.checkpointed = checkpointed_stream;
 
 /* If altering migration_stream update this assert too. */
-assert(checkpointed_stream == MIG_STREAM_NONE ||
-   checkpointed_stream == MIG_STREAM_REMUS);
+assert(checkpointed_stream == XC_MIG_STREAM_NONE ||
+   checkpointed_stream == XC_MIG_STREAM_REMUS);
 
 /*
  * TODO: Find some time to better tweak the live migration algorithm.
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 07/27] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams

[Wen Congyang]

On 03/05/2016 12:51 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 07/27] docs/libxl: Introduce 
> CHECKPOINT_CONTEXT to support migration v2 colo streams"):
>> From: Wen Congyang 
> 
> I think we will want to see an ack from Andy Cooper on this, in due
> course.
> 
>> It is the negotiation record for COLO.
>> Primary->Secondary:
>> control_id  0x: Secondary VM is out of sync, start a new 
>> checkpoint
>> Secondary->Primary:
>> 0x0001: Secondary VM is suspended
>> 0x0002: Secondary VM is ready
>> 0x0003: Secondary VM is resumed
> 
> I don't think it is necessary to repeat the enum assignment here in
> the commit message.

OK, will fix it in the next version.

> 
> 
>> +CHECKPOINT\_STATE
>> +--
> 
> This documentation patch ought to explicitly mention COLO, and have
> cross-references to the various documents (eg, the README added in the
> previous patch).
> 
>> +A checkpoint state record contains the control information for checkpoint.
>> +
>> + 0 1 2 3 4 5 6 7 octet
>> ++++
>> +| control_id | padding|
>> ++++
>> +
>> +
>> +FieldDescription
>> + ---
>> +control_id   0x: Secondary VM is out of sync, start a new 
>> checkpoint
>> + (Primary -> Secondary)
>> +
>> + 0x0001: Secondary VM is suspended (Secondary -> 
>> Primary)
>> +
>> + 0x0002: Secondary VM is ready (Secondary -> Primary)
>> +
>> + 0x0003: Secondary VM is resumed (Secondary -> Primary)
> 
> I think this should be accompanied by an explanation of what order
> these messages are sent in, and what both ends may or may not do
> during that time.

OK, will fix it in the next version.

> 
> 
>> @@ -212,6 +214,11 @@ class VerifyLibxl(VerifyBase):
>>  if len(content) != 0:
>>  raise RecordError("Checkpoint end record with non-zero length")
>>  
>> +def verify_record_checkpoint_state(self, content):
>> +""" Checkpoint state """
>> +if len(content) == 0:
>> +raise RecordError("Checkpoint state record with zero length")
>> +
> 
> I'm not verify familiar with this area of the code, but I think that
> this should probably check that the control_id is as expected.  Can it
> know what the right sequencing is ?

To Andrew Cooper:
What is the purpost of this script? If it is not used for live system, I think
the stream should not contain checkpoint state record.

Thanks
Wen Congyang

> 
> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 10/27] tools/libxl: add back channel support to write stream

[Wen Congyang]

On 03/05/2016 01:00 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 10/27] tools/libxl: add back channel 
> support to write stream"):
>> From: Wen Congyang 
>>
>> Add back channel support to write stream. If the write stream is
>> a back channel stream, this means the write stream is used by
>> Secondary to send some records back.
> 
> The general idea seems fine to me but I want an opinion from Andrew.
> 
> If I'm not mistaken there is no call site for this yet.  In which case
> this should be mentioned in the commit message.
> 
>> +/*- checkpoint state -*/
>> +void libxl__stream_write_checkpoint_state(libxl__egc *egc,
>> +  libxl__stream_write_state *stream,
>> +  libxl_sr_checkpoint_state *srcs)
> 
> Firstly, missing blank line.
> 
> Secondly, reading all this leads me to think that maybe the
> `checkpoint_state' record should be called something different.  Is it
> only ever going to be used for COLO ?  Maybe it should be
> `COLOHA_STATE' or something (and all the functions etc. renamed
> consequently) ?

Another question: what should be renamed? The function name or the type name
or both?

Thanks
Wen Congyang

> 
> What do you think ?
> 
> Thanks,
> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v10 26/31] COLO proxy: implement setup/teardown of COLO proxy module

[Wen Congyang]

gt;> +tv.tv_usec = timeout_us % 100;
>> +setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
>> +}
>> +
>> +iov.iov_base = tmp;
>> +iov.iov_len = size;
>> +next:
>> +ret = recvmsg(cps->sock_fd, &mh, 0);
>> +if (ret <= 0) {
>> +if (errno != EAGAIN && errno != EWOULDBLOCK)
> 
> -EINTR ?

IIRC, WAGAIN and EWOULDBLOCK may have different value in some system.
EINTR is not handled here.

> 
>> +LOGE(ERROR, "can't recv msg from kernel by netlink");
>> +goto err;
>> +}
>> +
>> +len += ret;
>> +if (mh.msg_flags & MSG_TRUNC) {
>> +size += 16384;
>> +tmp = libxl__realloc(NOGC, tmp, size);
> 
> You really should check 'tmp'.
> 
> If this loop continues on for some time the 'size' may be
> in milions and this realloc will fail.

OK, will fix it in the next version.

> 
>> +iov.iov_base = tmp + len;
>> +iov.iov_len = size - len;
>> +goto next;
> 
>> +}
>> +
>> +*buff = tmp;
>> +ret = len;
>> +goto out;
>> +
>> +err:
>> +free(tmp);
>> +*buff = NULL;
>> +
>> +out:
>> +if (timeout_us) {
>> +tv.tv_sec = 0;
>> +tv.tv_usec = 0;
>> +setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
>> +}
>> +return ret;
>> +}
>> +
>> +/* = colo-proxy: setup and teardown == */
>> +
>> +int colo_proxy_setup(libxl__colo_proxy_state *cps)
>> +{
>> +int skfd = 0;
>> +struct sockaddr_nl sa;
>> +struct nlmsghdr *h;
>> +int i = 1;
>> +int ret = ERROR_FAIL;
>> +uint8_t *buff = NULL;
>> +int64_t size;
>> +
>> +STATE_AO_GC(cps->ao);
>> +
>> +skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
>> +if (skfd < 0) {
>> +LOG(ERROR, "can not create a netlink socket: %s", strerror(errno));
>> +goto out;
>> +}
>> +cps->sock_fd = skfd;
>> +memset(&sa, 0, sizeof(sa));
>> +sa.nl_family = AF_NETLINK;
>> +sa.nl_groups = 0;
>> +retry:
>> +sa.nl_pid = i++;
>> +
>> +if (i > 10) {
>> +LOG(ERROR, "netlink bind error");
>> +goto out;
>> +}
>> +
>> +ret = bind(skfd, (struct sockaddr *)&sa, sizeof(sa));
>> +if (ret < 0 && errno == EADDRINUSE) {
>> +LOG(ERROR, "colo index %d has already in used", sa.nl_pid);
>> +goto retry;
>> +} else if (ret < 0) {
>> +LOG(ERROR, "netlink bind error");
>> +goto out;
>> +}
>> +
>> +cps->index = sa.nl_pid;
>> +ret = colo_proxy_send(cps, NULL, 0, COLO_PROXY_INIT);
>> +if (ret < 0) {
>> +goto out;
>> +}
> 
> Ditto. You can remove it.

OK, will check all codes.

Thanks
Wen Congyang

> 
>> +/* receive ack */
>> +size = colo_proxy_recv(cps, &buff, 50);
>> +if (size < 0) {
>> +LOG(ERROR, "Can't recv msg from kernel by netlink: %s",
>> +strerror(errno));
>> +goto out;
>> +}
>> +
>> +if (size) {
>> +h = (struct nlmsghdr *)buff;
>> +if (h->nlmsg_type == NLMSG_ERROR) {
>> +/* ack's type is NLMSG_ERROR */
>> +struct nlmsgerr *err = (struct nlmsgerr *)NLMSG_DATA(h);
>> +
>> +if (size - sizeof(*h) < sizeof(*err)) {
>> +LOG(ERROR, "NLMSG_LENGTH is too short");
>> +goto out;
>> +}
>> +
>> +if (err->error) {
>> +LOG(ERROR, "NLMSG_ERROR contains error %d", err->error);
>> +goto out;
>> +}
>> +}
>> +}
>> +
>> +ret = 0;
>> +
>> +out:
>> +free(buff);
>> +if (ret) {
>> +close(cps->sock_fd);
>> +cps->sock_fd = -1;
>> +}
>> +return ret;
>> +}
>> +
>> +void colo_proxy_teardown(libxl__colo_proxy_state *cps)
>> +{
>> +if (cps->sock_fd >= 0) {
>> +close(cps->sock_fd);
>> +cps->sock_fd = -1;
>> +}
>> +}
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index 3af5fdd..3b44b09 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -3112,6 +3112,15 @@ libxl__stream_read_inuse(const 
>> libxl__stream_read_state *stream)
>>  }
>>  
>>  /*- colo related state structure -*/
>> +typedef struct libxl__colo_proxy_state libxl__colo_proxy_state;
>> +struct libxl__colo_proxy_state {
>> +/* set by caller of colo_proxy_setup */
>> +libxl__ao *ao;
>> +
>> +int sock_fd;
>> +int index;
>> +};
>> +
>>  typedef struct libxl__colo_save_state libxl__colo_save_state;
>>  struct libxl__colo_save_state {
>>  int send_fd;
>> @@ -3126,6 +3135,9 @@ struct libxl__colo_save_state {
>>  /* private, used by qdisk block replication */
>>  bool qdisk_used;
>>  bool qdisk_setuped;
>> +
>> +/* private, used by colo-proxy */
>> +libxl__colo_proxy_state cps;
>>  };
>>  
>>  /*- Domain suspend (save) state structure -*/
>> @@ -3535,6 +3547,9 @@ struct libxl__colo_restore_state {
>>  bool qdisk_setuped;
>>  const char *host;
>>  const char *port;
>> +
>> +/* private, used by colo-proxy */
>> +libxl__colo_proxy_state cps;
>>  };
>>  
>>  struct libxl__domain_create_state {
>> -- 
>> 2.5.0
>>
>>
>>
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v4 3/3] libxl: rename checkpointed_stream to stream_type

[Wen Congyang]

Signed-off-by: Wen Congyang 
---
v3->v4: Remove the new macro, and updte the macro LIBXL_HAVE_CHECKPOINTED_STREAM
 tools/libxl/libxl.c  |  4 ++--
 tools/libxl/libxl.h  |  4 +++-
 tools/libxl/libxl_create.c   |  4 ++--
 tools/libxl/libxl_dom_save.c |  6 +++---
 tools/libxl/libxl_internal.h |  2 +-
 tools/libxl/libxl_save_callout.c |  4 ++--
 tools/libxl/libxl_stream_read.c  |  4 ++--
 tools/libxl/libxl_stream_write.c |  2 +-
 tools/libxl/libxl_types.idl  |  2 +-
 tools/libxl/xl_cmdimpl.c | 16 
 10 files changed, 25 insertions(+), 23 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 93e228d..7579dd2 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -876,7 +876,7 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
libxl_domain_remus_info *info,
 dss->live = 1;
 dss->debug = 0;
 dss->remus = info;
-dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
+dss->stream_type = LIBXL_CHECKPOINTED_STREAM_REMUS;
 
 assert(info);
 
@@ -937,7 +937,7 @@ int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, 
int fd, int flags,
 dss->type = type;
 dss->live = flags & LIBXL_SUSPEND_LIVE;
 dss->debug = flags & LIBXL_SUSPEND_DEBUG;
-dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_NONE;
+dss->stream_type = LIBXL_CHECKPOINTED_STREAM_NONE;
 
 rc = libxl__fd_flags_modify_save(gc, dss->fd,
  ~(O_NONBLOCK|O_NDELAY), 0,
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index f9e3ef5..c55094b 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -879,7 +879,9 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, 
libxl_mac *src);
 /*
  * LIBXL_HAVE_CHECKPOINTED_STREAM
  *
- * If this is defined, then libxl_checkpointed_stream exists.
+ * If this is defined, then libxl_checkpointed_stream exists, and the
+ * libxl_domain_create_restore() interface's parameter checkpointed_stream
+ * is renamed to stream_type
  */
 #define LIBXL_HAVE_CHECKPOINTED_STREAM 1
 
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index f1028bc..b28eb89 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -993,7 +993,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 libxl_domain_config *const d_config = dcs->guest_config;
 const int restore_fd = dcs->restore_fd;
 libxl__domain_build_state *const state = &dcs->build_state;
-const int checkpointed_stream = dcs->restore_params.checkpointed_stream;
+const int stream_type = dcs->restore_params.stream_type;
 
 if (rc) {
 domcreate_rebuild_done(egc, dcs, rc);
@@ -1033,7 +1033,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 dcs->srs.completion_callback = domcreate_stream_done;
 
 if (restore_fd >= 0) {
-switch (checkpointed_stream) {
+switch (stream_type) {
 case LIBXL_CHECKPOINTED_STREAM_REMUS:
 libxl__remus_restore_setup(egc, dcs);
 /* fall through */
diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index f3288b9..ff92ef0 100644
--- a/tools/libxl/libxl_dom_save.c
+++ b/tools/libxl/libxl_dom_save.c
@@ -338,7 +338,7 @@ void libxl__domain_save(libxl__egc *egc, 
libxl__domain_save_state *dss)
 unsigned int nr_vnodes = 0, nr_vmemranges = 0, nr_vcpus = 0;
 libxl__domain_suspend_state *dsps = &dss->dsps;
 
-if (dss->checkpointed_stream != LIBXL_CHECKPOINTED_STREAM_NONE && !r_info) 
{
+if (dss->stream_type != LIBXL_CHECKPOINTED_STREAM_NONE && !r_info) {
 LOG(ERROR, "Migration stream is checkpointed, but there's no "
"checkpoint info!");
 rc = ERROR_INVAL;
@@ -383,12 +383,12 @@ void libxl__domain_save(libxl__egc *egc, 
libxl__domain_save_state *dss)
 goto out;
 }
 
-if (dss->checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_REMUS) {
+if (dss->stream_type == LIBXL_CHECKPOINTED_STREAM_REMUS) {
 if (libxl_defbool_val(r_info->compression))
 dss->xcflags |= XCFLAGS_CHECKPOINT_COMPRESS;
 }
 
-if (dss->checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_NONE)
+if (dss->stream_type == LIBXL_CHECKPOINTED_STREAM_NONE)
 callbacks->suspend = libxl__domain_suspend_callback;
 
 callbacks->switch_qemu_logdirty = 
libxl__domain_suspend_common_switch_qemu_logdirty;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 005fe53..0aada0d 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3128,7 +3128,7 @@ struct libxl__domain_save_state {
 libxl_domain_type type;
 int live;
 int debug;
-int checkpointed_stream;
+libxl_checkpointed_stream stream_type;
 const libxl_domain_remus_info *remus;
 /* private */
 int rc;
dif

[Xen-devel] [PATCH v4 1/3] libxc: move migration_stream's definition to xenguest.h

[Wen Congyang]

xc_domain_save() and xc_domain_restore's parameter will use this type,
so it should be public.

Signed-off-by: Wen Congyang 
Acked-by: Wei Liu 
---
v2->v3: Rename MIG_STREAM_* to XC_MIG_STREAM_*
 tools/libxc/include/xenguest.h |  7 ++-
 tools/libxc/xc_sr_common.h | 10 --
 tools/libxc/xc_sr_save.c   | 12 ++--
 3 files changed, 12 insertions(+), 17 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index affc42b..cf521c3 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -75,13 +75,18 @@ struct save_callbacks {
 void* data;
 };
 
+typedef enum {
+XC_MIG_STREAM_NONE, /* plain stream */
+XC_MIG_STREAM_REMUS,
+} xc_migration_stream_t;
+
 /**
  * This function will save a running domain.
  *
  * @parm xch a handle to an open hypervisor interface
  * @parm fd the file descriptor to save a domain to
  * @parm dom the id of the domain
- * @param checkpointed_stream MIG_STREAM_NONE if the far end of the stream
+ * @param checkpointed_stream XC_MIG_STREAM_NONE if the far end of the stream
  *doesn't use checkpointing
  * @return 0 on success, -1 on failure
  */
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 66f595f..e7568b5 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -180,16 +180,6 @@ struct xc_sr_context
 
 xc_dominfo_t dominfo;
 
-/*
- * migration stream
- * 0: Plain VM
- * 1: Remus
- */
-enum {
-MIG_STREAM_NONE, /* plain stream */
-MIG_STREAM_REMUS,
-} migration_stream;
-
 union /* Common save or restore data. */
 {
 struct /* Save data. */
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index e258b7c..ab59673 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -629,7 +629,7 @@ static int send_domain_memory_live(struct xc_sr_context 
*ctx)
 if ( rc )
 goto out;
 
-if ( ctx->save.debug && ctx->save.checkpointed != MIG_STREAM_NONE )
+if ( ctx->save.debug && ctx->save.checkpointed != XC_MIG_STREAM_NONE )
 {
 rc = verify_frames(ctx);
 if ( rc )
@@ -758,7 +758,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 
 if ( ctx->save.live )
 rc = send_domain_memory_live(ctx);
-else if ( ctx->save.checkpointed != MIG_STREAM_NONE )
+else if ( ctx->save.checkpointed != XC_MIG_STREAM_NONE )
 rc = send_domain_memory_checkpointed(ctx);
 else
 rc = send_domain_memory_nonlive(ctx);
@@ -778,7 +778,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 if ( rc )
 goto err;
 
-if ( ctx->save.checkpointed != MIG_STREAM_NONE )
+if ( ctx->save.checkpointed != XC_MIG_STREAM_NONE )
 {
 /*
  * We have now completed the initial live portion of the checkpoint
@@ -799,7 +799,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 if ( rc <= 0 )
 goto err;
 }
-} while ( ctx->save.checkpointed != MIG_STREAM_NONE );
+} while ( ctx->save.checkpointed != XC_MIG_STREAM_NONE );
 
 xc_report_progress_single(xch, "End of stream");
 
@@ -845,8 +845,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t 
dom,
 ctx.save.checkpointed = checkpointed_stream;
 
 /* If altering migration_stream update this assert too. */
-assert(checkpointed_stream == MIG_STREAM_NONE ||
-   checkpointed_stream == MIG_STREAM_REMUS);
+assert(checkpointed_stream == XC_MIG_STREAM_NONE ||
+   checkpointed_stream == XC_MIG_STREAM_REMUS);
 
 /*
  * TODO: Find some time to better tweak the live migration algorithm.
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v4 2/3] tools: change checkpointed_stream's type from int to xc_migration_stream_t

[Wen Congyang]

checkpointed_stream is also renamed to stream_type

Signed-off-by: Wen Congyang 
Acked-by: Wei Liu 
---
v2->v3: Rename checkpointed_stream to stream_type
 tools/libxc/include/xenguest.h  |  8 
 tools/libxc/xc_nomigrate.c  |  4 ++--
 tools/libxc/xc_sr_restore.c | 10 +-
 tools/libxc/xc_sr_save.c|  8 
 tools/libxl/libxl_save_helper.c | 42 -
 5 files changed, 36 insertions(+), 36 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index cf521c3..4f0b06e 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -86,14 +86,14 @@ typedef enum {
  * @parm xch a handle to an open hypervisor interface
  * @parm fd the file descriptor to save a domain to
  * @parm dom the id of the domain
- * @param checkpointed_stream XC_MIG_STREAM_NONE if the far end of the stream
+ * @param stream_type XC_MIG_STREAM_NONE if the far end of the stream
  *doesn't use checkpointing
  * @return 0 on success, -1 on failure
  */
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t 
max_iters,
uint32_t max_factor, uint32_t flags /* XCFLAGS_xxx */,
struct save_callbacks* callbacks, int hvm,
-   int checkpointed_stream);
+   xc_migration_stream_t stream_type);
 
 /* callbacks provided by xc_domain_restore */
 struct restore_callbacks {
@@ -121,7 +121,7 @@ struct restore_callbacks {
  * @parm hvm non-zero if this is a HVM restore
  * @parm pae non-zero if this HVM domain has PAE support enabled
  * @parm superpages non-zero to allocate guest memory with superpages
- * @parm checkpointed_stream non-zero if the far end of the stream is using 
checkpointing
+ * @parm stream_type non-zero if the far end of the stream is using 
checkpointing
  * @parm callbacks non-NULL to receive a callback to restore toolstack
  *   specific data
  * @return 0 on success, -1 on failure
@@ -131,7 +131,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
   domid_t store_domid, unsigned int console_evtchn,
   unsigned long *console_mfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
-  int checkpointed_stream,
+  xc_migration_stream_t stream_type,
   struct restore_callbacks *callbacks);
 
 /**
diff --git a/tools/libxc/xc_nomigrate.c b/tools/libxc/xc_nomigrate.c
index c9124df..08e1f8c 100644
--- a/tools/libxc/xc_nomigrate.c
+++ b/tools/libxc/xc_nomigrate.c
@@ -23,7 +23,7 @@
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t 
max_iters,
uint32_t max_factor, uint32_t flags,
struct save_callbacks* callbacks, int hvm,
-   int checkpointed_stream)
+   xc_migration_stream_t stream_type)
 {
 errno = ENOSYS;
 return -1;
@@ -34,7 +34,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t 
dom,
   domid_t store_domid, unsigned int console_evtchn,
   unsigned long *console_mfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
-  int checkpointed_stream,
+  xc_migration_stream_t stream_type,
   struct restore_callbacks *callbacks)
 {
 errno = ENOSYS;
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index d4d33fd..819401d 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -725,7 +725,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
   domid_t store_domid, unsigned int console_evtchn,
   unsigned long *console_gfn, domid_t console_domid,
   unsigned int hvm, unsigned int pae, int superpages,
-  int checkpointed_stream,
+  xc_migration_stream_t stream_type,
   struct restore_callbacks *callbacks)
 {
 struct xc_sr_context ctx =
@@ -739,16 +739,16 @@ int xc_domain_restore(xc_interface *xch, int io_fd, 
uint32_t dom,
 ctx.restore.console_domid = console_domid;
 ctx.restore.xenstore_evtchn = store_evtchn;
 ctx.restore.xenstore_domid = store_domid;
-ctx.restore.checkpointed = checkpointed_stream;
+ctx.restore.checkpointed = stream_type;
 ctx.restore.callbacks = callbacks;
 
 /* Sanity checks for callbacks. */
-if ( checkpointed_stream )
+if ( stream_type )
 assert(callbacks->checkpoint);
 
 DPRINTF("fd %d, dom %u, hvm %u, pae %u, superpages %d"
-", checkpointed_stream %d", io_fd, dom, hvm, pae,
-superpages, checkpointed_stream);
+", stream_type %d", io_fd, dom, hvm, pae,
+  

Re: [Xen-devel] [PATCH v4 3/3] libxl: rename checkpointed_stream to stream_type

[Wen Congyang]

On 03/15/2016 08:54 PM, Wei Liu wrote:
> On Tue, Mar 15, 2016 at 09:58:37AM +0800, Wen Congyang wrote:
>> Signed-off-by: Wen Congyang 
>> ---
>> v3->v4: Remove the new macro, and updte the macro 
>> LIBXL_HAVE_CHECKPOINTED_STREAM
>>  tools/libxl/libxl.c  |  4 ++--
>>  tools/libxl/libxl.h  |  4 +++-
>>  tools/libxl/libxl_create.c   |  4 ++--
>>  tools/libxl/libxl_dom_save.c |  6 +++---
>>  tools/libxl/libxl_internal.h |  2 +-
>>  tools/libxl/libxl_save_callout.c |  4 ++--
>>  tools/libxl/libxl_stream_read.c  |  4 ++--
>>  tools/libxl/libxl_stream_write.c |  2 +-
>>  tools/libxl/libxl_types.idl  |  2 +-
>>  tools/libxl/xl_cmdimpl.c | 16 
>>  10 files changed, 25 insertions(+), 23 deletions(-)
>>
>> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
>> index 93e228d..7579dd2 100644
>> --- a/tools/libxl/libxl.c
>> +++ b/tools/libxl/libxl.c
>> @@ -876,7 +876,7 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
>> libxl_domain_remus_info *info,
>>  dss->live = 1;
>>  dss->debug = 0;
>>  dss->remus = info;
>> -dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
>> +dss->stream_type = LIBXL_CHECKPOINTED_STREAM_REMUS;
>>  
>>  assert(info);
>>  
>> @@ -937,7 +937,7 @@ int libxl_domain_suspend(libxl_ctx *ctx, uint32_t domid, 
>> int fd, int flags,
>>  dss->type = type;
>>  dss->live = flags & LIBXL_SUSPEND_LIVE;
>>  dss->debug = flags & LIBXL_SUSPEND_DEBUG;
>> -dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_NONE;
>> +dss->stream_type = LIBXL_CHECKPOINTED_STREAM_NONE;
>>  
>>  rc = libxl__fd_flags_modify_save(gc, dss->fd,
>>   ~(O_NONBLOCK|O_NDELAY), 0,
>> diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
>> index f9e3ef5..c55094b 100644
>> --- a/tools/libxl/libxl.h
>> +++ b/tools/libxl/libxl.h
>> @@ -879,7 +879,9 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, 
>> libxl_mac *src);
>>  /*
>>   * LIBXL_HAVE_CHECKPOINTED_STREAM
>>   *
>> - * If this is defined, then libxl_checkpointed_stream exists.
>> + * If this is defined, then libxl_checkpointed_stream exists, and the
>> + * libxl_domain_create_restore() interface's parameter checkpointed_stream
>> + * is renamed to stream_type
>>   */
>>  #define LIBXL_HAVE_CHECKPOINTED_STREAM 1
> 
> I just realised this patch have not provided compatibility shim for
> older version of LIBXL_API.
> 
> It now breaks backward compatibility because it changes existing API.
> It's my fault for not observing this earlier, sorry.
> 
> On the other hand, I don't think this patch is strictly necessary.
> Andrew's original complain was about libxc not exporting XC_MIG_* types.
> I think libxl can function just fine as is with the old name.
> 
> So if you really wish to change the name of that field, I'm afraid
> compatibility shim for all functions that accept
> libxl_domain_restore_params is required.
> 
> I suggest we drop this patch for now and focus on core COLO
> functionality, what do you think?

It is OK to drop this patch.

Thanks
Wen Congyang

> 
> Wei.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v11 20/27] Support colo mode for qemu disk

[Wen Congyang]

On 03/18/2016 01:18 AM, Ian Jackson wrote:
> Wen Congyang writes ("Re: [PATCH v11 20/27] Support colo mode for qemu disk"):
>> How does block replication work:
> 
> Thanks for this explanation, which is reallt helpful.
> 
> I would like to repeat back to you what I think I have understood:
> 
> Between resynchs, you allow each VM to run in parallel and to generate
> possibly-divergent disk writes.
> 
> So on host B you retain both the A and B disk writes.  They are
> stored as differences (qcow2) for performance reasons.
> 
> If A fails and it becomes necessary to resume the VM only on B, you
> use B's version of the VM (both disk and memory).
> 
> If B fails then A can use the A version (disk and memory).
> 
> If the two are still up, but they diverge in network traffic, you
> resynch the memory from A to B, and drop B's disk and replace it with
> a copy of A's.
> 
> Have I understood correctly ?

Yes.

> 
> 
> If so, what software, where, arranges for the management of the
> different qcow2 `layers' ?  Ie, what creates the layers; what resynchs
> them, etc. ?

active disk and hidden disk are seperate disk. The management application
can create an empty qcow disk before running COLO. These two disks are
empty disk, and have the same size with the secondary disk.

Thanks
Wen Congyang

> 
> The reason I started asking all these questions is because of these
> parameters in your disk config:
> 
>  +("colo_enable", libxl_defbool),
>  +("colo_restore_enable", libxl_defbool),
>  +("colo_host", string),
>  +("colo_port", string),
>  +("colo_export", string),
>  +("active_disk", string),
>  +("hidden_disk", string)
> 
> For COLO to work properly it is necessary that the `active_disk' and
> `hidden_disk' to relate in a specific way to the main disk: they must
> be related snapshots (qcow2, currently).
> 
> Would it be possible for these disk names to have formulaic,
> predicatable, names, so that they wouldn't need to be specified
> separately ?
> 
> Is there any value in being able to specify them separately ?
> 
> Ian.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH 12/35] libxl: events: Make timeout and async exec setup take an ao, not a gc

[Wen Congyang]

On 06/26/2015 01:44 AM, Ian Jackson wrote:
> Change the timeout setup functions to take a libxl__ao, not a
> libxl__gc.  This is going to be needed for ao abort, because timeouts
> are going to be a main hook for ao abort requests - so the timeouts
> need to be associated with an ao.
> 
> This means that timeouts can only occur as part of a long-running
> libxl function (but this is of course correct, as libxl shouldn't have
> any global timeouts, and indeed all the call sites have an ao).
> 
> Also remove the gc parameter from libxl__async_exec_start.  It can
> just use the gc from the ao supplied in the aes.
> 
> All the callers follow the obvious patterns and therefore supply the
> ao's gc to libxl__async_exec_start and the timeout setup functions.
> There is therefore no functional change in this patch.

Sorry, I just reviewed this patch, and don't do a building test.
libxl__async_exec_start() is also used in libxl_netbuffer.c.

Thanks
Wen Congyang

> 
> Signed-off-by: Ian Jackson 
> CC: Yang Hongyang 
> CC: Wen Congyang 
> CC: Lai Jiangshan 
> Acked-by: Wen Congyang 
> Acked-by: Ian Campbell 
> 
> ---
> v2: This patch split off from "Permit timeouts to signal cancellation".
> Rebased; consequently, deal with libxl__async_exec_start.
> CC'd authors of the libxl__async_exec_* functions.
> ---
>  tools/libxl/libxl_aoutils.c |8 +---
>  tools/libxl/libxl_device.c  |4 ++--
>  tools/libxl/libxl_dom.c |8 
>  tools/libxl/libxl_event.c   |6 --
>  tools/libxl/libxl_internal.h|6 +++---
>  tools/libxl/libxl_remus_disk_drbd.c |2 +-
>  tools/libxl/libxl_test_timedereg.c  |9 +
>  7 files changed, 24 insertions(+), 19 deletions(-)
> 
> diff --git a/tools/libxl/libxl_aoutils.c b/tools/libxl/libxl_aoutils.c
> index ef679dd..1502ffe 100644
> --- a/tools/libxl/libxl_aoutils.c
> +++ b/tools/libxl/libxl_aoutils.c
> @@ -46,7 +46,7 @@ int libxl__xswait_start(libxl__gc *gc, libxl__xswait_state 
> *xswa)
>  {
>  int rc;
>  
> -rc = libxl__ev_time_register_rel(gc, &xswa->time_ev,
> +rc = libxl__ev_time_register_rel(xswa->ao, &xswa->time_ev,
>   xswait_timeout_callback, 
> xswa->timeout_ms);
>  if (rc) goto err;
>  
> @@ -547,16 +547,18 @@ void libxl__async_exec_init(libxl__async_exec_state 
> *aes)
>  libxl__ev_child_init(&aes->child);
>  }
>  
> -int libxl__async_exec_start(libxl__gc *gc, libxl__async_exec_state *aes)
> +int libxl__async_exec_start(libxl__async_exec_state *aes)
>  {
>  pid_t pid;
>  
>  /* Convenience aliases */
> +libxl__ao *ao = aes->ao;
> +AO_GC;
>  libxl__ev_child *const child = &aes->child;
>  char ** const args = aes->args;
>  
>  /* Set execution timeout */
> -if (libxl__ev_time_register_rel(gc, &aes->time,
> +if (libxl__ev_time_register_rel(ao, &aes->time,
>  async_exec_timeout,
>  aes->timeout_ms)) {
>  LOG(ERROR, "unable to register timeout for executing: %s", 
> aes->what);
> diff --git a/tools/libxl/libxl_device.c b/tools/libxl/libxl_device.c
> index 951d125..b6276f6 100644
> --- a/tools/libxl/libxl_device.c
> +++ b/tools/libxl/libxl_device.c
> @@ -808,7 +808,7 @@ void libxl__initiate_device_remove(libxl__egc *egc,
>   * TODO: 4.2 Bodge due to QEMU, see comment on top of
>   * libxl__initiate_device_remove in libxl_internal.h
>   */
> -rc = libxl__ev_time_register_rel(gc, &aodev->timeout,
> +rc = libxl__ev_time_register_rel(ao, &aodev->timeout,
>   device_qemu_timeout,
>   LIBXL_QEMU_BODGE_TIMEOUT * 
> 1000);
>  if (rc) {
> @@ -1034,7 +1034,7 @@ static void device_hotplug(libxl__egc *egc, 
> libxl__ao_device *aodev)
>  aes->stdfds[1] = 2;
>  aes->stdfds[2] = -1;
>  
> -rc = libxl__async_exec_start(gc, aes);
> +rc = libxl__async_exec_start(aes);
>  if (rc)
>  goto out;
>  
> diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
> index ccbcb6e..40a2d79 100644
> --- a/tools/libxl/libxl_dom.c
> +++ b/tools/libxl/libxl_dom.c
> @@ -1200,7 +1200,7 @@ static void 
> domain_suspend_switch_qemu_xen_traditional_logdirty
>  switch_logdirty_xswatch, lds->ret_path);
>  if (rc) goto out;
>  
> -rc = libxl__ev_time_register_rel(gc, &lds->timeout,
> 

[Xen-devel] fix building error

[Wen Congyang]

Signed-off-by: Wen Congyang 
---
 tools/libxl/libxl_netbuffer.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/libxl/libxl_netbuffer.c b/tools/libxl/libxl_netbuffer.c
index ff2d6c7..107e867 100644
--- a/tools/libxl/libxl_netbuffer.c
+++ b/tools/libxl/libxl_netbuffer.c
@@ -310,7 +310,7 @@ static void nic_setup(libxl__egc *egc, libxl__remus_device 
*dev)
 }
 
 setup_async_exec(dev, "setup");
-rc = libxl__async_exec_start(gc, &dev->aodev.aes);
+rc = libxl__async_exec_start(&dev->aodev.aes);
 if (rc)
 goto out;
 
@@ -402,7 +402,7 @@ static void nic_teardown(libxl__egc *egc, 
libxl__remus_device *dev)
 
 setup_async_exec(dev, "teardown");
 
-rc = libxl__async_exec_start(gc, &dev->aodev.aes);
+rc = libxl__async_exec_start(&dev->aodev.aes);
 if (rc)
 goto out;
 
-- 
2.4.3

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] fix building error

[Wen Congyang]

On 06/30/2015 04:26 PM, Ian Campbell wrote:
> On Tue, 2015-06-30 at 14:10 +0800, Wen Congyang wrote:
> 
> It would be usual do describe here which commit caused the breakage
> (i.e. removed the parameter). Also please make the subject something
> more descriptive such as "remove now unnecessary gc from
> libxl__async_exec_start calls" e.g.
> 
> Subject: remove now unnecessary gc from libxl__async_exec_start calls
> 
> These were removed in commit XXXXXX.

OK, I will repost it.

Thanks
Wen Congyang

> 
>> Signed-off-by: Wen Congyang 
>> ---
>>  tools/libxl/libxl_netbuffer.c | 4 ++--
>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/tools/libxl/libxl_netbuffer.c b/tools/libxl/libxl_netbuffer.c
>> index ff2d6c7..107e867 100644
>> --- a/tools/libxl/libxl_netbuffer.c
>> +++ b/tools/libxl/libxl_netbuffer.c
>> @@ -310,7 +310,7 @@ static void nic_setup(libxl__egc *egc, 
>> libxl__remus_device *dev)
>>  }
>>  
>>  setup_async_exec(dev, "setup");
>> -rc = libxl__async_exec_start(gc, &dev->aodev.aes);
>> +rc = libxl__async_exec_start(&dev->aodev.aes);
>>  if (rc)
>>  goto out;
>>  
>> @@ -402,7 +402,7 @@ static void nic_teardown(libxl__egc *egc, 
>> libxl__remus_device *dev)
>>  
>>  setup_async_exec(dev, "teardown");
>>  
>> -rc = libxl__async_exec_start(gc, &dev->aodev.aes);
>> +rc = libxl__async_exec_start(&dev->aodev.aes);
>>  if (rc)
>>  goto out;
>>  
> 
> 
> .
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v2] remove now unnecessary gc from libxl__async_exec_start calls

[Wen Congyang]

These were removed in commit f5f8400f.

Signed-off-by: Wen Congyang 
---
 tools/libxl/libxl_netbuffer.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tools/libxl/libxl_netbuffer.c b/tools/libxl/libxl_netbuffer.c
index ff2d6c7..107e867 100644
--- a/tools/libxl/libxl_netbuffer.c
+++ b/tools/libxl/libxl_netbuffer.c
@@ -310,7 +310,7 @@ static void nic_setup(libxl__egc *egc, libxl__remus_device 
*dev)
 }
 
 setup_async_exec(dev, "setup");
-rc = libxl__async_exec_start(gc, &dev->aodev.aes);
+rc = libxl__async_exec_start(&dev->aodev.aes);
 if (rc)
 goto out;
 
@@ -402,7 +402,7 @@ static void nic_teardown(libxl__egc *egc, 
libxl__remus_device *dev)
 
 setup_async_exec(dev, "teardown");
 
-rc = libxl__async_exec_start(gc, &dev->aodev.aes);
+rc = libxl__async_exec_start(&dev->aodev.aes);
 if (rc)
 goto out;
 
-- 
2.4.3

___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v3 COLOPre 08/26] tools/libxc: support to resume uncooperative HVM guests

[Wen Congyang]

On 06/30/2015 12:27 AM, Ian Campbell wrote:
> On Thu, 2015-06-25 at 14:25 +0800, Yang Hongyang wrote:
>> From: Wen Congyang 
>>
>> 1. suspend
>> a. PVHVM and PV: we use the same way to suspend the guest(send the suspend
> 
> space between "guest" and the open parenthesis please.
> 
>>request to the guest). If the guest doesn't support evtchn, the xenstore
>>variant will be used, suspending the guest via XenBus control node.
>> b. pure HVM: we call xc_domain_shutdown(..., SHUTDOWN_suspend) to suspend
>>the guest
>>
>> 2. Resume:
>> a. fast path
>>In this case, we don't change the guest's state.
>>PV: modify the return code to 1, and than call the domctl:
>>XEN_DOMCTL_resumedomain
>>PVHVM: same with PV
>>HVM: do nothing in modify_returncode, and than call the domctl:
>> XEN_DOMCTL_resumedomain
>> b. slow
>>In this case, we have changed the guest's state.
> 
> "have" or "will"? AIUI the latter would be more accurate.
> 
>>PV: update start info, and reset all secondary CPU states. Than call the
>>domctl: XEN_DOMCTL_resumedomain
>>PVHVM and HVM can not be resumed.
> 
> I'm confused -- isn't the purpose of this patch to make PVHM support
> resume?

Without this patch, HVM(both PVHVM and pure HVM) can not be resumed in slow
path if its state is changed.

> 
>> For PVHVM, in my test, only call the domctl: XEN_DOMCTL_resumedomain
>> can work. I am not sure if we should update start info and reset all
>> secondary CPU states.
>>
>> For pure HVM guest, in my test, only call the domctl:
>> XEN_DOMCTL_resumedomain can work.
>>
>> So we can call libxl__domain_resume(..., 1) if we don't change the guest
>> state, otherwise call libxl__domain_resume(..., 0).
> 
> Hrm, so it sounds here like the correctness of this new functionality
> requires the caller to have not messed with the domain's state? What
> sort of changes are to the guest state are we talking about here?

Any state can be changed: memory, device state. If the caller messes
with the domain's state, the guest may be crashed.

> 
> Isn't that a new requirement for this call? If so then it should be
> documented somewhere, specifically what sorts of changes are and are not
> allowed and the types of guests which are affected.

The state should be the one saved by xen.

> 
>>
>> Signed-off-by: Wen Congyang 
>> Signed-off-by: Yang Hongyang 
>> ---
>>  tools/libxc/xc_resume.c | 22 ++
>>  1 file changed, 18 insertions(+), 4 deletions(-)
>>
>> diff --git a/tools/libxc/xc_resume.c b/tools/libxc/xc_resume.c
>> index e67bebd..bd82334 100644
>> --- a/tools/libxc/xc_resume.c
>> +++ b/tools/libxc/xc_resume.c
>> @@ -109,6 +109,23 @@ static int xc_domain_resume_cooperative(xc_interface 
>> *xch, uint32_t domid)
>>  return do_domctl(xch, &domctl);
>>  }
>>  
>> +static int xc_domain_resume_hvm(xc_interface *xch, uint32_t domid)
>> +{
>> +DECLARE_DOMCTL;
>> +
>> +/*
>> + * If it is PVHVM, the hypercall return code is 0, because this
>> + * is not a fast path resume, we do not modify_returncode as in
>> + * xc_domain_resume_cooperative.
>> + * (resuming it in a new domain context)
>> + *
>> + * If it is a HVM, the hypercall is a NOP.
>> + */
>> +domctl.cmd = XEN_DOMCTL_resumedomain;
>> +domctl.domain = domid;
>> +return do_domctl(xch, &domctl);
> 
> There are already several open coded instances of this
> XEN_DOMCTL_resumedomain, and I think putting this particular one into a
> helper is actually more confusing than just inlining this at the caller.
> 
> In particular when reading this function my first question was "how do
> we know this is not a fast path resume", the answer being that the only
> caller is the slow path resume case, but that's not evident from the
> context (what if someone adds a second call?)

Only the caller know it. See xc_domain_resume()'s third parameter.

Thanks
Wen Congyang

> 
> So I think at least the comment ought to go at the callsite, at which
> point this function doesn't add much.
> 
> (Ideally all the open coded do_domctl would go into a single
> do_domainresume or something, but I don't think you need to do that
> unless you really want to).
> 
> 
> 
> .
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v3 COLOPre 21/26] tools/libxl: refactor write stream to support back channel

[Wen Congyang]

On 06/30/2015 06:28 PM, Ian Campbell wrote:
> On Thu, 2015-06-25 at 14:25 +0800, Yang Hongyang wrote:
>> From: Wen Congyang 
> 
> Refactor how?
> 
> (Tip: In general no change can be sufficiently described in just the
> subject line unless it is absolutely the most trivial possible one).
> 
> It looks like mostly checkpoint_callback is renamed to
> write_records_callback and the type of its argument and that of
> completion_callback changed. But why?

checkpoint callback just write a record and return to the caller.
For back channel, we also write a record from secondary to primary.
So I rename it to write_records_callback.

In secondary side, there is no libxl__domain_save_state, so I change
the argument: use stream instead of dss. The caller can use container_of()
to get dss.

The back channel only needs this feature: use the read/write stream to read
and write record.

Thanks
Wen Congyang

> 
> 
> 
> .
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v3 COLOPre 21/26] tools/libxl: refactor write stream to support back channel

[Wen Congyang]

On 07/01/2015 06:45 PM, Ian Campbell wrote:
> On Wed, 2015-07-01 at 13:33 +0800, Wen Congyang wrote:
>> On 06/30/2015 06:28 PM, Ian Campbell wrote:
>>> On Thu, 2015-06-25 at 14:25 +0800, Yang Hongyang wrote:
>>>> From: Wen Congyang 
>>>
>>> Refactor how?
>>>
>>> (Tip: In general no change can be sufficiently described in just the
>>> subject line unless it is absolutely the most trivial possible one).
>>>
>>> It looks like mostly checkpoint_callback is renamed to
>>> write_records_callback and the type of its argument and that of
>>> completion_callback changed. But why?
>>
>> checkpoint callback just write a record and return to the caller.
>> For back channel, we also write a record from secondary to primary.
>> So I rename it to write_records_callback.
>>
>> In secondary side, there is no libxl__domain_save_state, so I change
>> the argument: use stream instead of dss. The caller can use container_of()
>> to get dss.
>>
>> The back channel only needs this feature: use the read/write stream to read
>> and write record.
> 
> Please use this to improve the commit message.

OK, I will do it.

Thanks
Wen Congyang

> 
> 
> .
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v1 COLO Pre 04/12] tools/libxl: introduce a new API libxl__domain_restore() to load qemu state

[Wen Congyang]

On 06/02/2015 05:26 PM, Yang Hongyang wrote:
> Secondary vm is running in colo mode. So we will do
> the following things again and again:
> 1. suspend both primay vm and secondary vm
> 2. sync the state
> 3. resume both primary vm and secondary vm
> We will send qemu's state each time in step2, and
> slave's qemu should read it each time before resuming
> secondary vm. Introduce a new API libxl__domain_restore()
> to do it. This API should be called before resuming
> secondary vm.
> 
> Signed-off-by: Yang Hongyang 
> Signed-off-by: Wen Congyang 
> ---
>  tools/libxl/Makefile|  3 +-
>  tools/libxl/libxl_dom_restore.c | 76 
> +
>  tools/libxl/libxl_internal.h|  4 +++
>  tools/libxl/libxl_qmp.c | 10 ++
>  4 files changed, 92 insertions(+), 1 deletion(-)
>  create mode 100644 tools/libxl/libxl_dom_restore.c
> 
> diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
> index a87ee04..39ce836 100644
> --- a/tools/libxl/Makefile
> +++ b/tools/libxl/Makefile
> @@ -96,7 +96,8 @@ LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o 
> libxl_pci.o \
>   libxl_json.o libxl_aoutils.o libxl_numa.o libxl_vnuma.o 
> \
>   libxl_save_callout.o _libxl_save_msgs_callout.o \
>   libxl_qmp.o libxl_event.o libxl_fork.o 
> libxl_dom_suspend.o \
> - libxl_toolstack.o libxl_dom_save.o $(LIBXL_OBJS-y)
> + libxl_toolstack.o libxl_dom_save.o libxl_dom_restore.o \
> + $(LIBXL_OBJS-y)
>  LIBXL_OBJS += libxl_genid.o
>  LIBXL_OBJS += _libxl_types.o libxl_flask.o _libxl_types_internal.o
>  
> diff --git a/tools/libxl/libxl_dom_restore.c b/tools/libxl/libxl_dom_restore.c
> new file mode 100644
> index 000..df15ece
> --- /dev/null
> +++ b/tools/libxl/libxl_dom_restore.c
> @@ -0,0 +1,76 @@
> +/*
> + * Copyright (C) 2015 FUJITSU LIMITED
> + * Author Yang Hongyang 
> + *Wen congyang 
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU Lesser General Public License as published
> + * by the Free Software Foundation; version 2 and later. with the special
> + * exception on linking described in file LICENSE.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU Lesser General Public License for more details.
> + */
> +
> +#include "libxl_osdeps.h" /* must come before any other headers */
> +
> +#include "libxl_internal.h"
> +
> +/*- main code for restoring, in order of execution -*/
> +
> +int libxl__domain_restore(libxl__gc *gc, uint32_t domid)
> +{
> +int rc = 0;
> +
> +libxl_domain_type type = libxl__domain_type(gc, domid);
> +if (type != LIBXL_DOMAIN_TYPE_HVM) {
> +rc = ERROR_FAIL;
> +goto out;
> +}
> +
> +rc = libxl__domain_restore_device_model(gc, domid);
> +if (rc)
> +LOG(ERROR, "failed to restore device mode for domain %u:%d",
> +domid, rc);
> +out:
> +return rc;
> +}
> +
> +int libxl__domain_restore_device_model(libxl__gc *gc, uint32_t domid)
> +{
> +char *state_file;
> +int rc;
> +
> +switch (libxl__device_model_version_running(gc, domid)) {
> +case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL:
> +/* not supported now */
> +rc = ERROR_INVAL;
> +break;
> +case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN:
> +/*
> + * This function may be called too many times for the same gc,
> + * so we use NOGC, and free the memory before return to avoid
> + * OOM.
> + */
> +state_file = libxl__sprintf(NOGC,
> +XC_DEVICE_MODEL_RESTORE_FILE".%d",
> +domid);
> +rc = libxl__qmp_restore(gc, domid, state_file);
> +free(state_file);
> +break;
> +default:
> +rc = ERROR_INVAL;
> +}
> +
> +return rc;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 4bab0de..71728ff 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -1022,6 +1022,7 @@ _hidden int libxl__domain_rename(libxl__gc *gc, 
> uint32_t domid,
>  
>  _hidden int libxl__toolstack_rest

Re: [Xen-devel] [PATCH v1 COLO Pre 02/12] libxc/restore: zero ioreq page only one time

[Wen Congyang]

On 06/02/2015 06:16 PM, Andrew Cooper wrote:
> On 02/06/15 10:26, Yang Hongyang wrote:
>> ioreq page contains evtchn which will be set when we resume the
>> secondary vm the first time. The hypervisor will check if the
>> evtchn is corrupted, so we cannot zero the ioreq page more
>> than one time.
>>
>> The ioreq->state is always STATE_IOREQ_NONE after the vm is
>> suspended, so it is OK if we only zero it one time.
>>
>> Signed-off-by: Yang Hongyang 
>> Signed-off-by: Wen congyang 
>> CC: Andrew Cooper 
> 
> Is the qemu process for the secondary running at this point?  If so,
> this is very much unsafe.

No, we restore the secondary vm while it has been suspended.
The problem is that: the ioreq page containes evtchn which is
used by hypervisor to notify qemu. Before migration finished,
we can clear it because the evtchn is invalid, and qemu will
allocate it and save it in ioreq page later.

Thanks
Wen Congyang

> 
> ~Andrew
> 
>> ---
>>  tools/libxc/xc_sr_restore_x86_hvm.c | 3 ++-
>>  1 file changed, 2 insertions(+), 1 deletion(-)
>>
>> diff --git a/tools/libxc/xc_sr_restore_x86_hvm.c 
>> b/tools/libxc/xc_sr_restore_x86_hvm.c
>> index 6f5af0e..06177e0 100644
>> --- a/tools/libxc/xc_sr_restore_x86_hvm.c
>> +++ b/tools/libxc/xc_sr_restore_x86_hvm.c
>> @@ -78,7 +78,8 @@ static int handle_hvm_params(struct xc_sr_context *ctx,
>>  break;
>>  case HVM_PARAM_IOREQ_PFN:
>>  case HVM_PARAM_BUFIOREQ_PFN:
>> -xc_clear_domain_page(xch, ctx->domid, entry->value);
>> +if ( !ctx->restore.buffer_all_records )
>> +xc_clear_domain_page(xch, ctx->domid, entry->value);
>>  break;
>>  }
>>  
> 
> .
> 


___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v7 12/18] tools/libx{l, c}: add back channel to libxc

[Wen Congyang]

On 01/30/2016 12:38 AM, Konrad Rzeszutek Wilk wrote:
> On Fri, Jan 29, 2016 at 01:27:28PM +0800, Wen Congyang wrote:
>> In COLO mode, both VMs are running, and are considered in sync if the
>> visible network traffic is identical.  After some time, they fall out of
>> sync.
>>
>> At this point, the two VMs have definitely diverged.  Lets call the
>> primary dirty bitmap set A, while the secondary dirty bitmap set B.
>>
>> Sets A and B are different.
>>
>> Under normal migration, the page data for set A will be sent from the
>> primary to the secondary.
>>
>> However, the set difference B - A (the one in B but not in A, lets
>> call this C) is out-of-date on the secondary (with respect to the
>> primary) and will not be sent by the primary (to secondary), as it
>> was not memory dirtied by the primary. The secondary needs C page data
>> to reconstruct an exact copy of the primary at the checkpoint.
>>
>> The secondary cannot calculate C as it doesn't know A.  Instead, the
>> secondary must send B to the primary, at which point the primary
>> calculates the union of A and B (lets call this D) which is all the
>> pages dirtied by both the primary and the secondary, and sends all page
>> data covered by D.
>>
>> In the general case, D is a superset of both A and B.  Without the
>> backchannel dirty bitmap, a COLO checkpoint can't reconstruct a valid
>> copy of the primary.
>>
>> We transfer the dirty bitmap on libxc side, so we need to introduce back
>> channel to libxc.
>>
>> Note: it is different from the paper. We change the original design to
>> the current one, according to our following concerns:
>> 1. The original design needs extra memory on Secondary host. When there's
>>multiple backups on one host, the memory cost is high.
>> 2. The memory cache code will be another 1k+, it will make the review
>>more time consuming.
>>
>> Note: the back channel will be used in the patch
>>  libxc/restore: send dirty pfn list to primary when checkpoint under COLO
>> to send dirty pfn list from secondary to primary. The patch is posted in
>> another series.
>>
>> Signed-off-by: Yang Hongyang 
>> Signed-off-by: Andrew Cooper 
>> CC: Ian Campbell 
>> CC: Ian Jackson 
>> CC: Wei Liu 
> 
> It is a bit confusing to have 'back_fd' and then 'send_fd'. 
> 
> Could you change the 'send_fd' (in this patch) to be called 
> 'send_back_fd' so that the connection between:
>  tools/libxl: Add back channel to allow migration target send data back
> and this patch is clear?
> 
> Or perhaps also add it in the commit description that you are using
> the 'send_fd' provided by ' tools/libxl: Add back channel to allow migration 
> target send data back'

Before this series:
In libxl:
we have send_fd/recv_fd(libxl_domain_remus_start()), and only have 
restore_fd(libxl_domain_create_restore())
In libxc:
We have io_fd(xc_domain_save()/xc_domain_restore())
The fd in libxc is provided by libxl.

I think after this series, we can add the following fd:
1. add a send_back_fd in libxl_domain_create_restore()
2. add a recv_fd in xc_domain_save()
3. add a send_back_fd in xc_domain_restore()

What about this?

Thanks
Wen Congyang

> 
> Otherwise: Reviewed-by: Konrad Rzeszutek Wilk 
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v7 01/18] libxl/remus: init checkpoint_callback in Remus setup callback

[Wen Congyang]

On 02/04/2016 03:39 AM, Wei Liu wrote:
> On Fri, Jan 29, 2016 at 01:27:17PM +0800, Wen Congyang wrote:
>> init stream {read/write} state checkpoint_callback in Remus setup callback.
>> There's no functional change, it's just refactoring so that we can move
>> all remus code into one file.
>>
>> Signed-off-by: Yang Hongyang 
>> Signed-off-by: Wen Congyang 
>> CC: Andrew Cooper 
>> CC: Ian Campbell 
>> CC: Ian Jackson 
>> CC: Wei Liu 
>> Reviewed-by: Konrad Rzeszutek Wilk 
>> ---
>>  tools/libxl/libxl.c  |  2 ++
>>  tools/libxl/libxl_create.c   | 10 +-
>>  tools/libxl/libxl_dom.c  |  5 +
>>  tools/libxl/libxl_internal.h |  4 
>>  4 files changed, 16 insertions(+), 5 deletions(-)
>>
>> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
>> index 94b5656..5346a0c 100644
>> --- a/tools/libxl/libxl.c
>> +++ b/tools/libxl/libxl.c
>> @@ -917,6 +917,8 @@ static void libxl__remus_setup(libxl__egc *egc,
>>  rds->domid = dss->domid;
>>  rds->callback = remus_setup_done;
>>  
>> +dss->sws.checkpoint_callback = remus_checkpoint_stream_written;
>> +
>>  libxl__remus_devices_setup(egc, rds);
>>  return;
>>  
>> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
>> index e491d83..8b1efe5 100644
>> --- a/tools/libxl/libxl_create.c
>> +++ b/tools/libxl/libxl_create.c
>> @@ -718,6 +718,12 @@ static void remus_checkpoint_stream_done(
>>  libxl__xc_domain_saverestore_async_callback_done(egc, &stream->shs, rc);
>>  }
>>  
>> +static void libxl__remus_restore_setup(libxl__egc *egc,
>> +   libxl__domain_create_state *dcs)
>> +{
>> +dcs->srs.checkpoint_callback = remus_checkpoint_stream_done;
>> +}
>> +
>>  /*- main domain creation -*/
>>  
>>  /* We have a linear control flow; only one event callback is
>> @@ -1004,6 +1010,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
>>  libxl__domain_build_state *const state = &dcs->build_state;
>>  libxl__srm_restore_autogen_callbacks *const callbacks =
>>  &dcs->srs.shs.callbacks.restore.a;
>> +const int checkpointed_stream = dcs->restore_params.checkpointed_stream;
>>  
>>  if (rc) {
>>  domcreate_rebuild_done(egc, dcs, rc);
>> @@ -1042,9 +1049,10 @@ static void domcreate_bootloader_done(libxl__egc *egc,
> 
> A few lines above in this function, there is a line like:
> 
> /* Restore */
> callbacks->checkpoint = libxl__remus_domain_restore_checkpoint_callback;
> 
> Do you not need to move this into libxl__remus_restore_setup as well? As
> far as I can tell that's only useful for remus.
> 
>>  dcs->srs.fd = restore_fd;
>>  dcs->srs.legacy = (dcs->restore_params.stream_version == 1);
>>  dcs->srs.completion_callback = domcreate_stream_done;
>> -dcs->srs.checkpoint_callback = remus_checkpoint_stream_done;
>>  
>>  if (restore_fd >= 0) {
>> +if (checkpointed_stream)
>> +libxl__remus_restore_setup(egc, dcs);
>>  libxl__stream_read_start(egc, &dcs->srs);
>>  return;
>>  }
>> diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
>> index 2269998..9e28bc4 100644
>> --- a/tools/libxl/libxl_dom.c
>> +++ b/tools/libxl/libxl_dom.c
>> @@ -1569,8 +1569,6 @@ out:
>>  
>>  /*- remus asynchronous checkpoint callback -*/
>>  
>> -static void remus_checkpoint_stream_written(
>> -libxl__egc *egc, libxl__stream_write_state *sws, int rc);
>>  static void remus_devices_commit_cb(libxl__egc *egc,
>>  libxl__remus_devices_state *rds,
>>  int rc);
>> @@ -1588,7 +1586,7 @@ static void 
>> libxl__remus_domain_save_checkpoint_callback(void *data)
>>  libxl__stream_write_start_checkpoint(egc, &dss->sws);
>>  }
>>  
>> -static void remus_checkpoint_stream_written(
>> +void remus_checkpoint_stream_written(
>>  libxl__egc *egc, libxl__stream_write_state *sws, int rc)
>>  {
>>  libxl__domain_suspend_state *dss = CONTAINER_OF(sws, *dss, sws);
>> @@ -1761,7 +1759,6 @@ void libxl__domain_save(libxl__egc *egc, 
>> libxl__domain_suspend_state *dss)
>>  callbacks->suspend = libxl__remus_domain_suspend_callback;
>>  callbacks->postcopy = libxl__remus_domain_resume_callback;
>>  callbacks->checkpoint = 
>> libxl__remus_domain_save_checkpoint_callback;
> 
> Do you not want to move this to libxl__remus_setup?

I think so, and will fix these two in the next version.

Thanks
Wen Congyang

> 
> 
> Wei.
> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v7 07/18] migration/save: pass checkpointed_stream from libxl to libxc

[Wen Congyang]

On 02/04/2016 03:40 AM, Wei Liu wrote:
> On Fri, Jan 29, 2016 at 01:27:23PM +0800, Wen Congyang wrote:
>> Pass checkpointed_stream from libxl to libxc.
>> It won't affact legacy migration because legacy migration
>> won't use this param.
>>
>> Signed-off-by: Yang Hongyang 
>> Signed-off-by: Wen Congyang 
>> CC: Ian Campbell 
>> CC: Ian Jackson 
>> CC: Wei Liu 
>> CC: Andrew Cooper 
> 
> Acked-by: Wei Liu 
> 
> With one nit below.
> 
>>  
>> -if ( ctx->save.debug && !ctx->save.checkpointed )
>> +if ( ctx->save.debug &&
>> +         ctx->save.checkpointed != MIG_STREAM_NONE )
> 
> You can fold this line to previous one.

Will fix it in the next version.

Thanks
Wen Congyang

> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v7 08/18] tools/libxl: introduce libxl__domain_restore_device_model to load qemu state

[Wen Congyang]

On 02/04/2016 03:40 AM, Wei Liu wrote:
> On Fri, Jan 29, 2016 at 01:27:24PM +0800, Wen Congyang wrote:
>> In normal migration, the qemu state is passed to qemu as a parameter.
>> With COLO, secondary vm is running. So we will do the following steps
>> at every checkpoint:
>> 1. suspend both primary vm and secondary vm
>> 2. sync the state
>> 3. resume both primary vm and secondary vm
>> Primary will send qemu's state in step2, and secondary's qemu should
>> read it and restore the state before it is resumed. We can not pass
>> the state to qemu as a parameter because secondary QEMU already started
>> at this point, so we introduce libxl__domain_restore_device_model() to
>> do it. This API MUST be called before resuming secondary vm.
>>
>> Signed-off-by: Yang Hongyang 
>> Signed-off-by: Wen Congyang 
>> Cc: Anthony Perard 
>> ---
>>  tools/libxl/libxl_dom_save.c | 20 
>>  tools/libxl/libxl_internal.h |  4 
>>  tools/libxl/libxl_qmp.c  | 10 ++
>>  3 files changed, 34 insertions(+)
>>
>> diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
>> index cd2e7de..7383d2d 100644
>> --- a/tools/libxl/libxl_dom_save.c
>> +++ b/tools/libxl/libxl_dom_save.c
>> @@ -518,6 +518,26 @@ int 
>> libxl__restore_emulator_xenstore_data(libxl__domain_create_state *dcs,
>>  return rc;
>>  }
>>  
>> +int libxl__domain_restore_device_model(libxl__gc *gc, uint32_t domid,
>> +   const char *restore_file)
>> +{
>> +int rc;
>> +
>> +switch (libxl__device_model_version_running(gc, domid)) {
>> +case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL:
>> +/* Will never be supported. */
>> +rc = ERROR_INVAL;
>> +break;
> 
> I'm not entirely sure if this statement would be true. The function name
> is generic enough to indicate this case should be supported.
> 
> However, this function is not used anywhere in this series, so I don't
> know whether my comment makes sense.
> 
> One way of moving forward is to stick this patch to COLO series itself.
> Let's skip this in this prerequisite series.

OK, I will put it in the COLO series itself.
This API is used for COLO, and COLO requries the newest qemu with block 
replication.
The block replication is still in the way. The tranditional qemu doesn't support
block replication and it is hard to backport it.

> 
>> +case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN:
>> +rc = libxl__qmp_restore(gc, domid, restore_file);
>> +break;
>> +default:
>> +rc = ERROR_INVAL;
>> +}
>> +
>> +return rc;
>> +}
>> +
>>  /*
>>   * Local variables:
>>   * mode: C
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index fbd1acb..896c119 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -1117,6 +1117,8 @@ _hidden int libxl__domain_rename(libxl__gc *gc, 
>> uint32_t domid,
>>   const char *old_name, const char *new_name,
>>   xs_transaction_t trans);
>>  
>> +_hidden int libxl__domain_restore_device_model(libxl__gc *gc, uint32_t 
>> domid,
>> +   const char *restore_file);
>>  _hidden int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t 
>> domid);
>>  
>>  _hidden const char *libxl__userdata_path(libxl__gc *gc, uint32_t domid,
>> @@ -1760,6 +1762,8 @@ _hidden int libxl__qmp_stop(libxl__gc *gc, int domid);
>>  _hidden int libxl__qmp_resume(libxl__gc *gc, int domid);
>>  /* Save current QEMU state into fd. */
>>  _hidden int libxl__qmp_save(libxl__gc *gc, int domid, const char *filename);
>> +/* Load current QEMU state from fd. */
> 
> This comment is wrong, it loads QEMU state from file, not fd.

will fix it in the next version.

Thanks
Wen Congyang

> 
>> +_hidden int libxl__qmp_restore(libxl__gc *gc, int domid, const char 
>> *filename);
>>  /* Set dirty bitmap logging status */
>>  _hidden int libxl__qmp_set_global_dirty_log(libxl__gc *gc, int domid, bool 
>> enable);
>>  _hidden int libxl__qmp_insert_cdrom(libxl__gc *gc, int domid, const 
>> libxl_device_disk *disk);
>> diff --git a/tools/libxl/libxl_qmp.c b/tools/libxl/libxl_qmp.c
>> index 714038b..eec8a44 100644
>> --- a/tools/libxl/libxl_qmp.c
>> +++ b/tools/libxl/libxl_qmp.c
>> @@ -905,6 +905,16 @@ int libxl__qmp_save(libxl__gc *gc,

Re: [Xen-devel] [PATCH v7 12/18] tools/libx{l, c}: add back channel to libxc

[Wen Congyang]

On 02/04/2016 03:40 AM, Wei Liu wrote:
> On Fri, Jan 29, 2016 at 01:27:28PM +0800, Wen Congyang wrote:
>> In COLO mode, both VMs are running, and are considered in sync if the
>> visible network traffic is identical.  After some time, they fall out of
>> sync.
>>
>> At this point, the two VMs have definitely diverged.  Lets call the
>> primary dirty bitmap set A, while the secondary dirty bitmap set B.
>>
>> Sets A and B are different.
>>
>> Under normal migration, the page data for set A will be sent from the
>> primary to the secondary.
>>
>> However, the set difference B - A (the one in B but not in A, lets
>> call this C) is out-of-date on the secondary (with respect to the
>> primary) and will not be sent by the primary (to secondary), as it
>> was not memory dirtied by the primary. The secondary needs C page data
>> to reconstruct an exact copy of the primary at the checkpoint.
>>
>> The secondary cannot calculate C as it doesn't know A.  Instead, the
>> secondary must send B to the primary, at which point the primary
>> calculates the union of A and B (lets call this D) which is all the
>> pages dirtied by both the primary and the secondary, and sends all page
>> data covered by D.
>>
>> In the general case, D is a superset of both A and B.  Without the
>> backchannel dirty bitmap, a COLO checkpoint can't reconstruct a valid
>> copy of the primary.
>>
>> We transfer the dirty bitmap on libxc side, so we need to introduce back
>> channel to libxc.
>>
>> Note: it is different from the paper. We change the original design to
>> the current one, according to our following concerns:
>> 1. The original design needs extra memory on Secondary host. When there's
>>multiple backups on one host, the memory cost is high.
>> 2. The memory cache code will be another 1k+, it will make the review
>>more time consuming.
>>
>> Note: the back channel will be used in the patch
> 
> "will not be used" ?
> 
> I don't see any read / write to the newly introduced fd.

It is used in COLO series.

Some patches in this series just introduce an API. Thess APIs will be used
in COLO series. Do you mean that these patches should be put in COLO series?
If so, I will check all patches.

> 
>>  libxc/restore: send dirty pfn list to primary when checkpoint under COLO
>> to send dirty pfn list from secondary to primary. The patch is posted in
>> another series.
>>
>> Signed-off-by: Yang Hongyang 
>> Signed-off-by: Andrew Cooper 
>> CC: Ian Campbell 
>> CC: Ian Jackson 
>> CC: Wei Liu 
>> ---
> [...]
>>  
>>  /*- helper execution -*/
>> +static int dup_fd_helper(libxl__gc *gc, int fd, const char *what)
>> +{
>> +int dup_fd = fd;
>> +
>> +if (fd <= 2) {
>> +dup_fd = dup(fd);
>> +if (dup_fd < 0) {
>> +LOGE(ERROR,"dup %s", what);
>> +exit(-1);
>> +}
>> +}
>> +libxl_fd_set_cloexec(CTX, dup_fd, 0);
>> +
>> +return dup_fd;
>> +}
>>  
> 
> It would be better if introduction of this helper to be separated into a
> different patch.

OK, will fix it in the next version.

Thanks
Wen Congyang

> 
> 
> .
> 




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-devel] [PATCH v7 05/18] tools/libxc: support to resume uncooperative HVM guests

[Wen Congyang]

On 02/04/2016 03:40 AM, Wei Liu wrote:
> On Fri, Jan 29, 2016 at 01:27:21PM +0800, Wen Congyang wrote:
>> Before this patch:
>> 1. suspend
>> a. PVHVM and PV: we use the same way to suspend the guest (send the suspend
>>request to the guest). If the guest doesn't support evtchn, the xenstore
>>variant will be used, suspending the guest via XenBus control node.
>> b. pure HVM: we call xc_domain_shutdown(..., SHUTDOWN_suspend) to suspend
>>the guest
>>
>> 2. Resume:
>> a. fast path(fast=1)
>>Do not change the guest state. We call libxl__domain_resume(.., 1) which
>>calls xc_domain_resume(..., 1 /* fast=1*/) to resume the guest.
>>PV:   modify the return code to 1, and than call the domctl:
>>  XEN_DOMCTL_resumedomain
>>PVHVM:same with PV
>>pure HVM: do nothing in modify_returncode, and than call the domctl:
> 
> "then"
> 
>>  XEN_DOMCTL_resumedomain
>> b. slow
>>Used when the guest's state have been changed. Will call
>>libxl__domain_resume(..., 0) to resume the guest.
>>PV:   update start info, and reset all secondary CPU states. Than call
>>  the domctl: XEN_DOMCTL_resumedomain
>>PVHVM:can not be resumed. You will get the following error message:
>>  "Cannot resume uncooperative HVM guests"
>>purt HVM: same with PVHVM
> 
> "pure"
> 
>>
>> After this patch:
>> 1. suspend
>>unchanged
>>
>> 2. Resume
>> a. fast path:
>>unchanged
>> b. slow
>>PV:   unchanged
>>PVHVM:call XEN_DOMCTL_resumedomain to resume the guest. Because we
>>  don't modify the return code, the PV driver will disconnect
>>  and reconnect.
>>  The guest ends up doing the XENMAPSPACE_shared_info
>>  XENMEM_add_to_physmap hypercall and resetting all of its CPU
>>  states to point to the shared_info(well except the ones past 
>> 32).
>>  That is the Linux kernel does that - regardless whether the
>>  SCHEDOP_shutdown:SHUTDOWN_suspend returns 1 or not.
>>Pure HVM: call XEN_DOMCTL_resumedomain to resume the guest.
> 
> In summary, this patch only changes slow path resume. Further more, it
> only affects PVHVM and pure HVM variants.
> 
> With you patch, pure HVM is able to resume with effectively the same
> path via XEN_DOMCTL_resumedomain, albeit it is done in two functions
> (_cooperative and _any).
> 
> And according to the recently change in documentation, slow path is
> always safe.
> 
> I think the commit message can be simplified a bit. This is assuming
> using XEN_DOMCTL_resumedomain to resume (PV)HVM in slow path is safe.
> 
> ===
> 
> Use XEN_DOMCTL_resumedomain to resume (PV)HVM guest in slow path
> 
> Previously it was not possible to resume PVHVM or pure HVM guest in slow
> path because libxc didn't support that.
> 
> Using XEN_DOMCTL_resumedomain without modifying guest state  to resume a
> guest is considered to be always safe.  Introduce a function to do that
> for (PV)HVM guests in slow path resume.
> 
> This patch fixes a bug that denies (PV)HVM slow path resume.  This will
> enable COLO to work properly:  COLO requires HVM guest to start in the
> new context that has been set up by COLO, hence slow path resume is
> required.
> 
> ===
> 
> Does this sound right? Especially the wording about safety.

It sounds right.

Thanks
Wen Congyang

> 
> Ian and Ian, you seemed to have suggested Congyang to write the above
> commit message. What do you think about my updated one?
> 
>>
>> Under COLO, we will update the guest's state(modify memory, cpu's registers,
>> device status...). In this case, we cannot use the fast path to resume it.
>> Keep the return code 0, and use a slow path to resume the guest. While
>> resuming HVM using slow path is not supported currently, this patch is to
>> make the resume call to not fail.
>>
>> Signed-off-by: Wen Congyang 
>> Signed-off-by: Yang Hongyang 
>> ---
>>  tools/libxc/xc_resume.c | 25 +
>>  1 file changed, 21 insertions(+), 4 deletions(-)
>>
>> diff --git a/tools/libxc/xc_resume.c b/tools/libxc/xc_resume.c
>> index 87d4324..4a9b035 100644
>> --- a/tools/libxc/xc_resume.c
>> +++ b/tools/libxc/xc_resume.c
>> @@ -108,6 +108,26 @@ static int xc_domain_resume_cooperative(xc_interface 
>> *xch, uint32_t domid)
>>  return do_domctl(xch, &domctl);
>>  }
>>  

[Xen-devel] [PATCH v8 11/13] tools/libxl: store remus_ops in checkpoint device state

[Wen Congyang]

Checkpoint device is an abstract layer to do checkpoint.
COLO can also use it to do checkpoint. But there are
still some codes in checkpoint device which touch remus.

This patch and:
 tools/libxl: move remus state into a seperate structure
 tools/libxl: seperate device init/cleanup from checkpoint device layer
will seperate remus from checkpoint device layer.

We use remus ops directly in checkpoint device. Store it
in checkpoint device state so that we do not aware of
remus_ops in the checkpoint device layer.

It is pure refactoring and no functional changes.

Signed-off-by: Wen Congyang 
Signed-off-by: Yang Hongyang 
Acked-by:Ian Campbell 
Reviewed-by: Konrad Rzeszutek Wilk 
Acked-by: Wei Liu 
---
 tools/libxl/libxl_checkpoint_device.c | 10 +-
 tools/libxl/libxl_internal.h  |  2 ++
 tools/libxl/libxl_remus.c |  9 +
 3 files changed, 12 insertions(+), 9 deletions(-)

diff --git a/tools/libxl/libxl_checkpoint_device.c 
b/tools/libxl/libxl_checkpoint_device.c
index 226f159..bbc6dc4 100644
--- a/tools/libxl/libxl_checkpoint_device.c
+++ b/tools/libxl/libxl_checkpoint_device.c
@@ -17,14 +17,6 @@
 
 #include "libxl_internal.h"
 
-extern const libxl__checkpoint_device_instance_ops remus_device_nic;
-extern const libxl__checkpoint_device_instance_ops remus_device_drbd_disk;
-static const libxl__checkpoint_device_instance_ops *remus_ops[] = {
-&remus_device_nic,
-&remus_device_drbd_disk,
-NULL,
-};
-
 /*- helper functions -*/
 
 static int init_device_subkind(libxl__checkpoint_devices_state *cds)
@@ -172,7 +164,7 @@ static void device_setup_iterate(libxl__egc *egc, 
libxl__ao_device *aodev)
 goto out;
 
 do {
-dev->ops = remus_ops[++dev->ops_index];
+dev->ops = dev->cds->ops[++dev->ops_index];
 if (!dev->ops) {
 libxl_device_nic * nic = NULL;
 libxl_device_disk * disk = NULL;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index bde7a15..2847d13 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2895,6 +2895,8 @@ struct libxl__checkpoint_devices_state {
 uint32_t domid;
 libxl__checkpoint_callback *callback;
 int device_kind_flags;
+/* The ops must be pointer array, and the last ops must be NULL. */
+const libxl__checkpoint_device_instance_ops **ops;
 
 /*- private for abstract layer only -*/
 
diff --git a/tools/libxl/libxl_remus.c b/tools/libxl/libxl_remus.c
index d41a439..86f81c3 100644
--- a/tools/libxl/libxl_remus.c
+++ b/tools/libxl/libxl_remus.c
@@ -18,6 +18,14 @@
 
 #include "libxl_internal.h"
 
+extern const libxl__checkpoint_device_instance_ops remus_device_nic;
+extern const libxl__checkpoint_device_instance_ops remus_device_drbd_disk;
+static const libxl__checkpoint_device_instance_ops *remus_ops[] = {
+&remus_device_nic,
+&remus_device_drbd_disk,
+NULL,
+};
+
 /* Remus setup and teardown -*/
 
 static void remus_setup_done(libxl__egc *egc,
@@ -55,6 +63,7 @@ void libxl__remus_setup(libxl__egc *egc,
 cds->ao = ao;
 cds->domid = dss->domid;
 cds->callback = remus_setup_done;
+cds->ops = remus_ops;
 
 dss->sws.checkpoint_callback = remus_checkpoint_stream_written;
 
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v8 09/13] tools/libxl: rename remus device to checkpoint device

[Wen Congyang]

This patch is auto generated by the following commands:
 1. git mv tools/libxl/libxl_remus_device.c 
tools/libxl/libxl_checkpoint_device.c
 2. perl -pi -e 's/libxl_remus_device/libxl_checkpoint_device/g' 
tools/libxl/Makefile
 3. perl -pi -e 's/\blibxl__remus_devices/libxl__checkpoint_devices/g' 
tools/libxl/*.[ch]
 4. perl -pi -e 's/\blibxl__remus_device\b/libxl__checkpoint_device/g' 
tools/libxl/*.[ch]
 5. perl -pi -e 
's/\blibxl__remus_device_instance_ops\b/libxl__checkpoint_device_instance_ops/g'
 tools/libxl/*.[ch]
 6. perl -pi -e 's/\blibxl__remus_callback\b/libxl__checkpoint_callback/g' 
tools/libxl/*.[ch]
 7. perl -pi -e 's/\bremus_device_init\b/checkpoint_device_init/g' 
tools/libxl/*.[ch]
 8. perl -pi -e 's/\bremus_devices_setup\b/checkpoint_devices_setup/g' 
tools/libxl/*.[ch]
 9. perl -pi -e 's/\bdefine_remus_checkpoint_api\b/define_checkpoint_api/g' 
tools/libxl/*.[ch]
10. perl -pi -e 's/\brds\b/cds/g' tools/libxl/*.[ch]
11. perl -pi -e 's/REMUS_DEVICE/CHECKPOINT_DEVICE/g' tools/libxl/*.[ch] 
tools/libxl/*.idl
12. perl -pi -e 's/REMUS_DEVOPS/CHECKPOINT_DEVOPS/g' tools/libxl/*.[ch] 
tools/libxl/*.idl
13. perl -pi -e 's/\bremus\b/checkpoint/g' 
tools/libxl/libxl_checkpoint_device.[ch]
14. perl -pi -e 's/\bremus device/checkpoint device/g' 
tools/libxl/libxl_internal.h
15. perl -pi -e 's/\bRemus device/checkpoint device/g' 
tools/libxl/libxl_internal.h
16. perl -pi -e 's/\bremus abstract/checkpoint abstract/g' 
tools/libxl/libxl_internal.h
17. perl -pi -e 's/\bremus invocation/checkpoint invocation/g' 
tools/libxl/libxl_internal.h
18. perl -pi -e 's/\blibxl__remus_device_\(/libxl__checkpoint_device_(/g' 
tools/libxl/libxl_internal.h

The patch also fixes the following backword compatibility:
  The error code ERROR_REMUS_XXX was introduced in Xen 4.5, and
  changed to ERROR_CHECKPOINT_XXX after previous renaming.

Signed-off-by: Wen Congyang 
Signed-off-by: Yang Hongyang 
Reviewed-Lightly-by: Konrad Rzeszutek Wilk 
Acked-by: Wei Liu 
---
 tools/libxl/Makefile   |   2 +-
 tools/libxl/libxl.h|  12 ++
 ...xl_remus_device.c => libxl_checkpoint_device.c} | 198 ++---
 tools/libxl/libxl_internal.h   | 112 ++--
 tools/libxl/libxl_netbuffer.c  | 108 +--
 tools/libxl/libxl_nonetbuffer.c|  10 +-
 tools/libxl/libxl_remus.c  |  76 
 tools/libxl/libxl_remus_disk_drbd.c|  52 +++---
 tools/libxl/libxl_types.idl|   4 +-
 9 files changed, 293 insertions(+), 281 deletions(-)
 rename tools/libxl/{libxl_remus_device.c => libxl_checkpoint_device.c} (52%)

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 263ea0e..789a12e 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -64,7 +64,7 @@ else
 LIBXL_OBJS-y += libxl_no_convert_callout.o
 endif
 
-LIBXL_OBJS-y += libxl_remus.o libxl_remus_device.o libxl_remus_disk_drbd.o
+LIBXL_OBJS-y += libxl_remus.o libxl_checkpoint_device.o libxl_remus_disk_drbd.o
 
 LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
 LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 6225db1..f9e3ef5 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -883,6 +883,18 @@ void libxl_mac_copy(libxl_ctx *ctx, libxl_mac *dst, 
libxl_mac *src);
  */
 #define LIBXL_HAVE_CHECKPOINTED_STREAM 1
 
+/*
+ * ERROR_REMUS_XXX error code only exists from Xen 4.5, Xen 4.6 and it
+ * is changed to ERROR_CHECKPOINT_XXX in Xen 4.7
+ */
+#if defined(LIBXL_API_VERSION) && LIBXL_API_VERSION >= 0x040500 \
+   && LIBXL_API_VERSION < 0x040700
+#define ERROR_REMUS_DEVOPS_DOES_NOT_MATCH \
+ERROR_CHECKPOINT_DEVOPS_DOES_NOT_MATCH
+#define ERROR_REMUS_DEVICE_NOT_SUPPORTED \
+ERROR_CHECKPOINT_DEVICE_NOT_SUPPORTED
+#endif
+
 typedef char **libxl_string_list;
 void libxl_string_list_dispose(libxl_string_list *sl);
 int libxl_string_list_length(const libxl_string_list *sl);
diff --git a/tools/libxl/libxl_remus_device.c 
b/tools/libxl/libxl_checkpoint_device.c
similarity index 52%
rename from tools/libxl/libxl_remus_device.c
rename to tools/libxl/libxl_checkpoint_device.c
index a6cb7f6..109cd23 100644
--- a/tools/libxl/libxl_remus_device.c
+++ b/tools/libxl/libxl_checkpoint_device.c
@@ -17,9 +17,9 @@
 
 #include "libxl_internal.h"
 
-extern const libxl__remus_device_instance_ops remus_device_nic;
-extern const libxl__remus_device_instance_ops remus_device_drbd_disk;
-static const libxl__remus_device_instance_ops *remus_ops[] = {
+extern const libxl__checkpoint_device_instance_ops remus_device_nic;
+extern const libxl__checkpoint_de

[Xen-devel] [PATCH v8 05/13] tools/libxc: support to resume uncooperative HVM guests

[Wen Congyang]

Before this patch:
1. suspend
a. PVHVM and PV: we use the same way to suspend the guest (send the suspend
   request to the guest). If the guest doesn't support evtchn, the xenstore
   variant will be used, suspending the guest via XenBus control node.
b. pure HVM: we call xc_domain_shutdown(..., SHUTDOWN_suspend) to suspend
   the guest

2. Resume:
a. fast path(fast=1)
   Do not change the guest state. We call libxl__domain_resume(.., 1) which
   calls xc_domain_resume(..., 1 /* fast=1*/) to resume the guest.
   PV:   modify the return code to 1, and than call the domctl:
 XEN_DOMCTL_resumedomain
   PVHVM:same with PV
   pure HVM: do nothing in modify_returncode, and than call the domctl:
 XEN_DOMCTL_resumedomain
b. slow
   Used when the guest's state have been changed. Will call
   libxl__domain_resume(..., 0) to resume the guest.
   PV:   update start info, and reset all secondary CPU states. Than call
 the domctl: XEN_DOMCTL_resumedomain
   PVHVM:can not be resumed. You will get the following error message:
 "Cannot resume uncooperative HVM guests"
   pure HVM: same with PVHVM

After this patch:
1. suspend
   unchanged

2. Resume
a. fast path:
   unchanged
b. slow
   PV:   unchanged
   PVHVM:call XEN_DOMCTL_resumedomain to resume the guest. Because we
 don't modify the return code, the PV driver will disconnect
 and reconnect.
 The guest ends up doing the XENMAPSPACE_shared_info
 XENMEM_add_to_physmap hypercall and resetting all of its CPU
 states to point to the shared_info(well except the ones past 32).
 That is the Linux kernel does that - regardless whether the
 SCHEDOP_shutdown:SHUTDOWN_suspend returns 1 or not.
   Pure HVM: call XEN_DOMCTL_resumedomain to resume the guest.

Under COLO, we will update the guest's state(modify memory, cpu's registers,
device status...). In this case, we cannot use the fast path to resume it.
Keep the return code 0, and use a slow path to resume the guest. While
resuming HVM using slow path is not supported currently, this patch is to
make the resume call to not fail.

Signed-off-by: Wen Congyang 
Signed-off-by: Yang Hongyang 
Reviewed-by: Konrad Rzeszutek Wilk 
---
 tools/libxc/xc_resume.c | 25 +
 1 file changed, 21 insertions(+), 4 deletions(-)

diff --git a/tools/libxc/xc_resume.c b/tools/libxc/xc_resume.c
index e692b81..4eedf87 100644
--- a/tools/libxc/xc_resume.c
+++ b/tools/libxc/xc_resume.c
@@ -108,6 +108,26 @@ static int xc_domain_resume_cooperative(xc_interface *xch, 
uint32_t domid)
 return do_domctl(xch, &domctl);
 }
 
+static int xc_domain_resume_hvm(xc_interface *xch, uint32_t domid)
+{
+DECLARE_DOMCTL;
+
+/*
+ * The domctl XEN_DOMCTL_resumedomain unpause each vcpu. After
+ * the domctl, the guest will run.
+ *
+ * If it is PVHVM, the guest called the hypercall
+ *SCHEDOP_shutdown:SHUTDOWN_suspend
+ * to suspend itself. We don't modify the return code, so the PV driver
+ * will disconnect and reconnect.
+ *
+ * If it is a HVM, the guest will continue running.
+ */
+domctl.cmd = XEN_DOMCTL_resumedomain;
+domctl.domain = domid;
+return do_domctl(xch, &domctl);
+}
+
 static int xc_domain_resume_any(xc_interface *xch, uint32_t domid)
 {
 DECLARE_DOMCTL;
@@ -137,10 +157,7 @@ static int xc_domain_resume_any(xc_interface *xch, 
uint32_t domid)
  */
 #if defined(__i386__) || defined(__x86_64__)
 if ( info.hvm )
-{
-ERROR("Cannot resume uncooperative HVM guests");
-return rc;
-}
+return xc_domain_resume_hvm(xch, domid);
 
 if ( xc_domain_get_guest_width(xch, domid, &dinfo->guest_width) != 0 )
 {
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v8 01/13] libxl/remus: init checkpoint callback in Remus setup callback

[Wen Congyang]

Init stream {read/write} state checkpoint_callback, suspend/resume/checkpoint
callback in Remus setup callback.
There's no functional change, it's just refactoring so that we can move
all remus code into one file.

Signed-off-by: Yang Hongyang 
Signed-off-by: Wen Congyang 
CC: Andrew Cooper 
CC: Ian Campbell 
CC: Ian Jackson 
CC: Wei Liu 
Reviewed-by: Konrad Rzeszutek Wilk 
---
 tools/libxl/libxl.c  |  8 
 tools/libxl/libxl_create.c   | 18 ++
 tools/libxl/libxl_dom.c  | 18 +-
 tools/libxl/libxl_internal.h |  7 +++
 4 files changed, 34 insertions(+), 17 deletions(-)

diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 2d18b8d..38029cd 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -899,6 +899,8 @@ static void libxl__remus_setup(libxl__egc *egc,
 /* Convenience aliases */
 libxl__remus_devices_state *const rds = &dss->rds;
 const libxl_domain_remus_info *const info = dss->remus;
+libxl__srm_save_autogen_callbacks *const callbacks =
+&dss->sws.shs.callbacks.save.a;
 
 STATE_AO_GC(dss->ao);
 
@@ -917,6 +919,12 @@ static void libxl__remus_setup(libxl__egc *egc,
 rds->domid = dss->domid;
 rds->callback = remus_setup_done;
 
+dss->sws.checkpoint_callback = remus_checkpoint_stream_written;
+
+callbacks->suspend = libxl__remus_domain_suspend_callback;
+callbacks->postcopy = libxl__remus_domain_resume_callback;
+callbacks->checkpoint = libxl__remus_domain_save_checkpoint_callback;
+
 libxl__remus_devices_setup(egc, rds);
 return;
 
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index de5d27f..7293d0b 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -730,6 +730,17 @@ static void remus_checkpoint_stream_done(
 libxl__xc_domain_saverestore_async_callback_done(egc, &stream->shs, rc);
 }
 
+static void libxl__remus_restore_setup(libxl__egc *egc,
+   libxl__domain_create_state *dcs)
+{
+/* Convenience aliases */
+libxl__srm_restore_autogen_callbacks *const callbacks =
+&dcs->srs.shs.callbacks.restore.a;
+
+callbacks->checkpoint = libxl__remus_domain_restore_checkpoint_callback;
+dcs->srs.checkpoint_callback = remus_checkpoint_stream_done;
+}
+
 /*- main domain creation -*/
 
 /* We have a linear control flow; only one event callback is
@@ -1014,8 +1025,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 libxl_domain_config *const d_config = dcs->guest_config;
 const int restore_fd = dcs->restore_fd;
 libxl__domain_build_state *const state = &dcs->build_state;
-libxl__srm_restore_autogen_callbacks *const callbacks =
-&dcs->srs.shs.callbacks.restore.a;
+const int checkpointed_stream = dcs->restore_params.checkpointed_stream;
 
 if (rc) {
 domcreate_rebuild_done(egc, dcs, rc);
@@ -1043,7 +1053,6 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 }
 
 /* Restore */
-callbacks->checkpoint = libxl__remus_domain_restore_checkpoint_callback;
 
 rc = libxl__build_pre(gc, domid, d_config, state);
 if (rc)
@@ -1054,9 +1063,10 @@ static void domcreate_bootloader_done(libxl__egc *egc,
 dcs->srs.fd = restore_fd;
 dcs->srs.legacy = (dcs->restore_params.stream_version == 1);
 dcs->srs.completion_callback = domcreate_stream_done;
-dcs->srs.checkpoint_callback = remus_checkpoint_stream_done;
 
 if (restore_fd >= 0) {
+if (checkpointed_stream)
+libxl__remus_restore_setup(egc, dcs);
 libxl__stream_read_start(egc, &dcs->srs);
 return;
 }
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 2269998..7835d4d 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -1489,7 +1489,7 @@ static void remus_devices_preresume_cb(libxl__egc *egc,
libxl__remus_devices_state *rds,
int rc);
 
-static void libxl__remus_domain_suspend_callback(void *data)
+void libxl__remus_domain_suspend_callback(void *data)
 {
 libxl__save_helper_state *shs = data;
 libxl__egc *egc = shs->egc;
@@ -1532,7 +1532,7 @@ out:
 libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, !rc);
 }
 
-static void libxl__remus_domain_resume_callback(void *data)
+void libxl__remus_domain_resume_callback(void *data)
 {
 libxl__save_helper_state *shs = data;
 libxl__egc *egc = shs->egc;
@@ -1569,8 +1569,6 @@ out:
 
 /*- remus asynchronous checkpoint callback -*/
 
-static void remus_checkpoint_stream_written(
-libxl__egc *egc, libxl__stream_write_state *sws, int rc);
 static void remus_devices_commit_cb(libxl__egc *egc,
 libxl__remus_devices_state *rds,
   

[Xen-devel] [PATCH v8 07/13] migration/save: pass checkpointed_stream from libxl to libxc

[Wen Congyang]

Pass checkpointed_stream from libxl to libxc.
It won't affact legacy migration because legacy migration
won't use this param.

Signed-off-by: Yang Hongyang 
Signed-off-by: Wen Congyang 
CC: Ian Campbell 
CC: Ian Jackson 
CC: Wei Liu 
CC: Andrew Cooper 
Reviewed-by: Konrad Rzeszutek Wilk 
Acked-by: Wei Liu 
---
 tools/libxc/include/xenguest.h   |  6 --
 tools/libxc/xc_nomigrate.c   |  3 ++-
 tools/libxc/xc_sr_common.h   | 12 +++-
 tools/libxc/xc_sr_save.c | 17 +++--
 tools/libxl/libxl.c  |  2 ++
 tools/libxl/libxl_dom_save.c | 11 ---
 tools/libxl/libxl_internal.h |  1 +
 tools/libxl/libxl_save_callout.c |  2 +-
 tools/libxl/libxl_save_helper.c  |  3 ++-
 tools/libxl/libxl_stream_write.c |  2 +-
 tools/libxl/libxl_types.idl  |  1 +
 11 files changed, 44 insertions(+), 16 deletions(-)

diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index d48b3ff..affc42b 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -29,7 +29,6 @@
 #define XCFLAGS_HVM   (1 << 2)
 #define XCFLAGS_STDVGA(1 << 3)
 #define XCFLAGS_CHECKPOINT_COMPRESS(1 << 4)
-#define XCFLAGS_CHECKPOINTED(1 << 5)
 
 #define X86_64_B_SIZE   64 
 #define X86_32_B_SIZE   32
@@ -82,11 +81,14 @@ struct save_callbacks {
  * @parm xch a handle to an open hypervisor interface
  * @parm fd the file descriptor to save a domain to
  * @parm dom the id of the domain
+ * @param checkpointed_stream MIG_STREAM_NONE if the far end of the stream
+ *doesn't use checkpointing
  * @return 0 on success, -1 on failure
  */
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t 
max_iters,
uint32_t max_factor, uint32_t flags /* XCFLAGS_xxx */,
-   struct save_callbacks* callbacks, int hvm);
+   struct save_callbacks* callbacks, int hvm,
+   int checkpointed_stream);
 
 /* callbacks provided by xc_domain_restore */
 struct restore_callbacks {
diff --git a/tools/libxc/xc_nomigrate.c b/tools/libxc/xc_nomigrate.c
index 902429e..c9124df 100644
--- a/tools/libxc/xc_nomigrate.c
+++ b/tools/libxc/xc_nomigrate.c
@@ -22,7 +22,8 @@
 
 int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t 
max_iters,
uint32_t max_factor, uint32_t flags,
-   struct save_callbacks* callbacks, int hvm)
+   struct save_callbacks* callbacks, int hvm,
+   int checkpointed_stream)
 {
 errno = ENOSYS;
 return -1;
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 60b43e8..66f595f 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -180,6 +180,16 @@ struct xc_sr_context
 
 xc_dominfo_t dominfo;
 
+/*
+ * migration stream
+ * 0: Plain VM
+ * 1: Remus
+ */
+enum {
+MIG_STREAM_NONE, /* plain stream */
+MIG_STREAM_REMUS,
+} migration_stream;
+
 union /* Common save or restore data. */
 {
 struct /* Save data. */
@@ -191,7 +201,7 @@ struct xc_sr_context
 bool live;
 
 /* Plain VM, or checkpoints over time. */
-bool checkpointed;
+int checkpointed;
 
 /* Further debugging information in the stream. */
 bool debug;
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index ccb000e..e258b7c 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -629,7 +629,7 @@ static int send_domain_memory_live(struct xc_sr_context 
*ctx)
 if ( rc )
 goto out;
 
-if ( ctx->save.debug && !ctx->save.checkpointed )
+if ( ctx->save.debug && ctx->save.checkpointed != MIG_STREAM_NONE )
 {
 rc = verify_frames(ctx);
 if ( rc )
@@ -758,7 +758,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 
 if ( ctx->save.live )
 rc = send_domain_memory_live(ctx);
-else if ( ctx->save.checkpointed )
+else if ( ctx->save.checkpointed != MIG_STREAM_NONE )
 rc = send_domain_memory_checkpointed(ctx);
 else
 rc = send_domain_memory_nonlive(ctx);
@@ -778,7 +778,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 if ( rc )
 goto err;
 
-if ( ctx->save.checkpointed )
+if ( ctx->save.checkpointed != MIG_STREAM_NONE )
 {
 /*
  * We have now completed the initial live portion of the checkpoint
@@ -799,7 +799,7 @@ static int save(struct xc_sr_context *ctx, uint16_t 
guest_type)
 if ( rc <= 0 )
 goto err;
 }
-} while ( ctx->save.checkpointed );
+} while ( ctx->save.checkpointed != MIG_STREAM_NONE );
 
 xc_report_progress_single(xch, "End of stream");
 
@@ -829,7 +829,8 @@ sta

[Xen-devel] [PATCH v8 00/13] Prerequisite patches for COLO

[Wen Congyang]

This patchset is Prerequisite for COLO feature. Refer to:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping

Patch status:
1. Acked patches: patch 2-4, 6-13
2. Reviewd patches: all
3. New patches: none
Note:
1. Patch 1 and 7 is updated according to Wei Liu's comments
2. Patch 2-3 is updated because patch 1 is updated
3. Patch 8, 9, 11, 12 in v7 is moved to another series
4. Patch 13, 14 in v7 is fold into one patch(patch 9)
5. The commit message for patch 5 is not updated(wait the reply
   from Ian C, and Ian J)

You can get the codes from here:
https://github.com/wencongyang/xen/tree/colo_pre_v8
You can get the whole colo related patches from here:
https://github.com/wencongyang/xen/tree/colo_v10

v6->v7:
 - Addressed comments from Konrad Rzeszutek Wilk

v5->v6:
 - Fix some bugs found in the test

v4->v5:
 - Rebased to the latest xen
 - Addressed comments from last round

v3->v4:
 - Rebased to the latest migration v2 branch
 - Addressed comments from last round

v2->v3:
 - Merge '[PATCH v2 0/6] Misc cleanups for libxl' into this patchset
   for easy review
 - Addressed review comments
 - Add back channel to libxc
 - Introduce should_checkpoint callback
 - Introduce DIRTY_BITMAP record on libxc side
 - Introduce COLO_CONTEXT record on libxl side
 - Ported to Libxl migration v2

v1->v2:
 - Rebased to [PATCH v2 0/6] Misc cleanups for libxl
 - Add a bugfix for the error handling of process_record

Wen Congyang (13):
  libxl/remus: init checkpoint callback in Remus setup callback
  tools/libxl: move remus code into libxl_remus.c
  tools/libxl: move save/restore code into libxl_dom_save.c
  libxl/save: Refactor libxl__domain_suspend_state
  tools/libxc: support to resume uncooperative HVM guests
  tools/libxl: introduce enum type libxl_checkpointed_stream
  migration/save: pass checkpointed_stream from libxl to libxc
  tools/libxl: export logdirty_init
  tools/libxl: rename remus device to checkpoint device
  tools/libxl: adjust the indentation
  tools/libxl: store remus_ops in checkpoint device state
  tools/libxl: move remus state into a seperate structure
  tools/libxl: seperate device init/cleanup from checkpoint device layer

 tools/libxc/include/xenguest.h|   6 +-
 tools/libxc/xc_nomigrate.c|   3 +-
 tools/libxc/xc_resume.c   |  25 +-
 tools/libxc/xc_sr_common.h|  12 +-
 tools/libxc/xc_sr_save.c  |  17 +-
 tools/libxl/Makefile  |   4 +-
 tools/libxl/libxl.c   |  81 +---
 tools/libxl/libxl.h   |  19 +
 tools/libxl/libxl_checkpoint_device.c | 282 +
 tools/libxl/libxl_create.c|  44 +-
 tools/libxl/libxl_dom.c   | 740 --
 tools/libxl/libxl_dom_save.c  | 521 
 tools/libxl/libxl_dom_suspend.c   | 207 ++
 tools/libxl/libxl_internal.h  | 217 ++
 tools/libxl/libxl_netbuffer.c | 117 +++---
 tools/libxl/libxl_nonetbuffer.c   |  10 +-
 tools/libxl/libxl_remus.c | 424 +++
 tools/libxl/libxl_remus_device.c  | 327 ---
 tools/libxl/libxl_remus_disk_drbd.c   |  56 +--
 tools/libxl/libxl_save_callout.c  |   4 +-
 tools/libxl/libxl_save_helper.c   |   3 +-
 tools/libxl/libxl_stream_read.c   |   7 +-
 tools/libxl/libxl_stream_write.c  |  18 +-
 tools/libxl/libxl_types.idl   |  10 +-
 tools/libxl/xl_cmdimpl.c  |  18 +-
 25 files changed, 1709 insertions(+), 1463 deletions(-)
 create mode 100644 tools/libxl/libxl_checkpoint_device.c
 create mode 100644 tools/libxl/libxl_dom_save.c
 create mode 100644 tools/libxl/libxl_remus.c
 delete mode 100644 tools/libxl/libxl_remus_device.c

-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

[Xen-devel] [PATCH v8 03/13] tools/libxl: move save/restore code into libxl_dom_save.c

[Wen Congyang]

This is purely code motion.

Signed-off-by: Yang Hongyang 
Signed-off-by: Wen Congyang 
CC: Ian Jackson 
Acked-by: Ian Campbell 
Reviewed-by: Konrad Rzeszutek Wilk 
Acked-by: Wei Liu 
---
 tools/libxl/Makefile |   2 +-
 tools/libxl/libxl_dom.c  | 509 
 tools/libxl/libxl_dom_save.c | 538 +++
 3 files changed, 539 insertions(+), 510 deletions(-)
 create mode 100644 tools/libxl/libxl_dom_save.c

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 7d64ecc..263ea0e 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -105,7 +105,7 @@ LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o 
libxl_pci.o \
libxl_stream_read.o libxl_stream_write.o \
libxl_save_callout.o _libxl_save_msgs_callout.o \
libxl_qmp.o libxl_event.o libxl_fork.o \
-   libxl_dom_suspend.o $(LIBXL_OBJS-y)
+   libxl_dom_suspend.o libxl_dom_save.o $(LIBXL_OBJS-y)
 LIBXL_OBJS += libxl_genid.o
 LIBXL_OBJS += _libxl_types.o libxl_flask.o _libxl_types_internal.o
 
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index d74f1a4..664adad 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -24,7 +24,6 @@
 #include 
 #include 
 #include 
-#include 
 
 libxl_domain_type libxl__domain_type(libxl__gc *gc, uint32_t domid)
 {
@@ -1107,514 +1106,6 @@ int libxl__qemu_traditional_cmd(libxl__gc *gc, uint32_t 
domid,
 return libxl__xs_printf(gc, XBT_NULL, path, "%s", cmd);
 }
 
-/*
- * Inspect the buffer between start and end, and return a pointer to the
- * character following the NUL terminator of start, or NULL if start is not
- * terminated before end.
- */
-static const char *next_string(const char *start, const char *end)
-{
-if (start >= end) return NULL;
-
-size_t total_len = end - start;
-size_t len = strnlen(start, total_len);
-
-if (len == total_len)
-return NULL;
-else
-return start + len + 1;
-}
-
-int libxl__restore_emulator_xenstore_data(libxl__domain_create_state *dcs,
-  const char *ptr, uint32_t size)
-{
-STATE_AO_GC(dcs->ao);
-const char *next = ptr, *end = ptr + size, *key, *val;
-int rc;
-
-const uint32_t domid = dcs->guest_domid;
-const uint32_t dm_domid = libxl_get_stubdom_id(CTX, domid);
-const char *xs_root = libxl__device_model_xs_path(gc, dm_domid, domid, "");
-
-while (next < end) {
-key = next;
-next = next_string(next, end);
-
-/* Sanitise 'key'. */
-if (!next) {
-rc = ERROR_FAIL;
-LOG(ERROR, "Key in xenstore data not NUL terminated");
-goto out;
-}
-if (key[0] == '\0') {
-rc = ERROR_FAIL;
-LOG(ERROR, "empty key found in xenstore data");
-goto out;
-}
-if (key[0] == '/') {
-rc = ERROR_FAIL;
-LOG(ERROR, "Key in xenstore data not relative");
-goto out;
-}
-
-val = next;
-next = next_string(next, end);
-
-/* Sanitise 'val'. */
-if (!next) {
-rc = ERROR_FAIL;
-LOG(ERROR, "Val in xenstore data not NUL terminated");
-goto out;
-}
-
-libxl__xs_printf(gc, XBT_NULL,
- GCSPRINTF("%s/%s", xs_root, key),
- "%s", val);
-}
-
-rc = 0;
-
- out:
-return rc;
-}
-
-/* Domain suspend (save) */
-
-static void stream_done(libxl__egc *egc,
-libxl__stream_write_state *sws, int rc);
-static void domain_save_done(libxl__egc *egc,
- libxl__domain_suspend_state *dss, int rc);
-
-/*- complicated callback, called by xc_domain_save -*/
-
-/*
- * We implement the other end of protocol for controlling qemu-dm's
- * logdirty.  There is no documentation for this protocol, but our
- * counterparty's implementation is in
- * qemu-xen-traditional.git:xenstore.c in the function
- * xenstore_process_logdirty_event
- */
-
-static void switch_logdirty_timeout(libxl__egc *egc, libxl__ev_time *ev,
-const struct timeval *requested_abs,
-int rc);
-static void switch_logdirty_xswatch(libxl__egc *egc, libxl__ev_xswatch*,
-const char *watch_path, const char *event_path);
-static void switch_logdirty_done(libxl__egc *egc,
- libxl__domain_suspend_state *dss, int rc);
-
-static void logdirty_init(libxl__logdirty_switch *lds)
-{
-lds->cmd_path = 0;
-libxl__ev_xswatch_init(&lds->watch);
-libxl__ev_tim

[Xen-devel] [PATCH v8 02/13] tools/libxl: move remus code into libxl_remus.c

[Wen Congyang]

After previous refactoring, we are now able to move all remus code
into a separate file libxl_remus.c.

Export following functions for internal use:
- setup/teardown Remus:
  * libxl__remus_setup
  * libxl__remus_teardown
  * libxl__remus_restore_setup

Signed-off-by: Yang Hongyang 
Signed-off-by: Wen Congyang 
CC: Ian Campbell 
CC: Ian Jackson 
Acked-by:Ian Campbell 
Reviewed-by: Konrad Rzeszutek Wilk 
Acked-by: Wei Liu 
---
 tools/libxl/Makefile |   2 +-
 tools/libxl/libxl.c  |  75 -
 tools/libxl/libxl_create.c   |  32 
 tools/libxl/libxl_dom.c  | 223 --
 tools/libxl/libxl_internal.h |  14 +-
 tools/libxl/libxl_remus.c| 362 +++
 6 files changed, 371 insertions(+), 337 deletions(-)
 create mode 100644 tools/libxl/libxl_remus.c

diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 620720e..7d64ecc 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -64,7 +64,7 @@ else
 LIBXL_OBJS-y += libxl_no_convert_callout.o
 endif
 
-LIBXL_OBJS-y += libxl_remus_device.o libxl_remus_disk_drbd.o
+LIBXL_OBJS-y += libxl_remus.o libxl_remus_device.o libxl_remus_disk_drbd.o
 
 LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
 LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 38029cd..d6ce7da 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -831,12 +831,6 @@ out:
 return ptr;
 }
 
-static void libxl__remus_setup(libxl__egc *egc,
-   libxl__domain_suspend_state *dss);
-static void remus_setup_done(libxl__egc *egc,
- libxl__remus_devices_state *rds, int rc);
-static void remus_setup_failed(libxl__egc *egc,
-   libxl__remus_devices_state *rds, int rc);
 static void remus_failover_cb(libxl__egc *egc,
   libxl__domain_suspend_state *dss, int rc);
 
@@ -893,75 +887,6 @@ int libxl_domain_remus_start(libxl_ctx *ctx, 
libxl_domain_remus_info *info,
 return AO_CREATE_FAIL(rc);
 }
 
-static void libxl__remus_setup(libxl__egc *egc,
-   libxl__domain_suspend_state *dss)
-{
-/* Convenience aliases */
-libxl__remus_devices_state *const rds = &dss->rds;
-const libxl_domain_remus_info *const info = dss->remus;
-libxl__srm_save_autogen_callbacks *const callbacks =
-&dss->sws.shs.callbacks.save.a;
-
-STATE_AO_GC(dss->ao);
-
-if (libxl_defbool_val(info->netbuf)) {
-if (!libxl__netbuffer_enabled(gc)) {
-LOG(ERROR, "Remus: No support for network buffering");
-goto out;
-}
-rds->device_kind_flags |= (1 << LIBXL__DEVICE_KIND_VIF);
-}
-
-if (libxl_defbool_val(info->diskbuf))
-rds->device_kind_flags |= (1 << LIBXL__DEVICE_KIND_VBD);
-
-rds->ao = ao;
-rds->domid = dss->domid;
-rds->callback = remus_setup_done;
-
-dss->sws.checkpoint_callback = remus_checkpoint_stream_written;
-
-callbacks->suspend = libxl__remus_domain_suspend_callback;
-callbacks->postcopy = libxl__remus_domain_resume_callback;
-callbacks->checkpoint = libxl__remus_domain_save_checkpoint_callback;
-
-libxl__remus_devices_setup(egc, rds);
-return;
-
-out:
-dss->callback(egc, dss, ERROR_FAIL);
-}
-
-static void remus_setup_done(libxl__egc *egc,
- libxl__remus_devices_state *rds, int rc)
-{
-libxl__domain_suspend_state *dss = CONTAINER_OF(rds, *dss, rds);
-STATE_AO_GC(dss->ao);
-
-if (!rc) {
-libxl__domain_save(egc, dss);
-return;
-}
-
-LOG(ERROR, "Remus: failed to setup device for guest with domid %u, rc %d",
-dss->domid, rc);
-rds->callback = remus_setup_failed;
-libxl__remus_devices_teardown(egc, rds);
-}
-
-static void remus_setup_failed(libxl__egc *egc,
-   libxl__remus_devices_state *rds, int rc)
-{
-libxl__domain_suspend_state *dss = CONTAINER_OF(rds, *dss, rds);
-STATE_AO_GC(dss->ao);
-
-if (rc)
-LOG(ERROR, "Remus: failed to teardown device after setup failed"
-" for guest with domid %u, rc %d", dss->domid, rc);
-
-dss->callback(egc, dss, rc);
-}
-
 static void remus_failover_cb(libxl__egc *egc,
   libxl__domain_suspend_state *dss, int rc)
 {
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 7293d0b..e421d36 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -709,38 +709,6 @@ static int store_libxl_entry(libxl__gc *gc, uint32_t domid,
 
libxl_device_model_version_to_string(b_info->device_model_version));
 }
 
-/*- remus asynchronous checkpoint callba

[Xen-devel] [PATCH v8 10/13] tools/libxl: adjust the indentation

[Wen Congyang]

This is just tidying up after the "tools/libxl: rename remus device
to checkpoint device" patch automatic renaming.

Signed-off-by: Yang Hongyang 
Signed-off-by: Wen Congyang 
Acked-by: Ian Campbell 
Reviewed-by: Konrad Rzeszutek Wilk 
Acked-by: Wei Liu 
---
 tools/libxl/libxl_checkpoint_device.c | 21 +++--
 tools/libxl/libxl_internal.h  | 19 +++
 2 files changed, 22 insertions(+), 18 deletions(-)

diff --git a/tools/libxl/libxl_checkpoint_device.c 
b/tools/libxl/libxl_checkpoint_device.c
index 109cd23..226f159 100644
--- a/tools/libxl/libxl_checkpoint_device.c
+++ b/tools/libxl/libxl_checkpoint_device.c
@@ -73,9 +73,9 @@ static void devices_teardown_cb(libxl__egc *egc,
 /* checkpoint device setup and teardown */
 
 static libxl__checkpoint_device* checkpoint_device_init(libxl__egc *egc,
-  libxl__checkpoint_devices_state 
*cds,
-  libxl__device_kind kind,
-  void *libxl_dev)
+libxl__checkpoint_devices_state *cds,
+libxl__device_kind kind,
+void *libxl_dev)
 {
 libxl__checkpoint_device *dev = NULL;
 
@@ -89,9 +89,10 @@ static libxl__checkpoint_device* 
checkpoint_device_init(libxl__egc *egc,
 }
 
 static void checkpoint_devices_setup(libxl__egc *egc,
-libxl__checkpoint_devices_state *cds);
+ libxl__checkpoint_devices_state *cds);
 
-void libxl__checkpoint_devices_setup(libxl__egc *egc, 
libxl__checkpoint_devices_state *cds)
+void libxl__checkpoint_devices_setup(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds)
 {
 int i, rc;
 
@@ -137,7 +138,7 @@ out:
 }
 
 static void checkpoint_devices_setup(libxl__egc *egc,
-libxl__checkpoint_devices_state *cds)
+ libxl__checkpoint_devices_state *cds)
 {
 int i, rc;
 
@@ -285,12 +286,12 @@ static void devices_checkpoint_cb(libxl__egc *egc,
 
 /* API implementations */
 
-#define define_checkpoint_api(api)\
-void libxl__checkpoint_devices_##api(libxl__egc *egc,\
-libxl__checkpoint_devices_state *cds)\
+#define define_checkpoint_api(api)  \
+void libxl__checkpoint_devices_##api(libxl__egc *egc,   \
+libxl__checkpoint_devices_state *cds)   \
 {   \
 int i;  \
-libxl__checkpoint_device *dev;   \
+libxl__checkpoint_device *dev;  \
 \
 STATE_AO_GC(cds->ao);   \
 \
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 630f048..bde7a15 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2818,7 +2818,8 @@ typedef struct libxl__save_helper_state {
  * Each device type needs to implement the interfaces specified in
  * the libxl__checkpoint_device_instance_ops if it wishes to support Remus.
  *
- * The high-level control flow through the checkpoint device layer is shown 
below:
+ * The high-level control flow through the checkpoint device layer is shown
+ * below:
  *
  * xl remus
  *  |->  libxl_domain_remus_start
@@ -2879,7 +2880,8 @@ int 
init_subkind_drbd_disk(libxl__checkpoint_devices_state *cds);
 void cleanup_subkind_drbd_disk(libxl__checkpoint_devices_state *cds);
 
 typedef void libxl__checkpoint_callback(libxl__egc *,
-   libxl__checkpoint_devices_state *, int rc);
+libxl__checkpoint_devices_state *,
+int rc);
 
 /*
  * State associated with a checkpoint invocation, including parameters
@@ -2887,7 +2889,7 @@ typedef void libxl__checkpoint_callback(libxl__egc *,
  * save/restore machinery.
  */
 struct libxl__checkpoint_devices_state {
-/* must be set by caller of libxl__checkpoint_device_(setup|teardown) 
*/
+/*-- must be set by caller of libxl__checkpoint_device_(setup|teardown) 
--*/
 
 libxl__ao *ao;
 uint32_t domid;
@@ -2900,7 +2902,8 @@ struct libxl__checkpoint_devices_state {
 /*
  * this array is allocated before setup the checkpoint devices by the
  * checkpoint abstract layer.
- * devs may be NULL, means there's no checkpoint devices that has been set 
up.
+ * devs may be NULL, means 

[Xen-devel] [PATCH v8 13/13] tools/libxl: seperate device init/cleanup from checkpoint device layer

[Wen Congyang]

we call (init|cleanup)_subkind_nic and (init|cleanup)_subkind_drbd_disk
directly in checkpoint device. Move them to libxl_remus.c, Call them before
calling libxl__checkpoint_devices_setup() or after calling
libxl__checkpoint_devices_teardown().
it is pure refactoring and no functional changes.

Signed-off-by: Wen Congyang 
Signed-off-by: Yang Hongyang 
Acked-by: Ian Campbell 
Reviewed-by: Konrad Rzeszutek Wilk 
Acked-by: Wei Liu 
---
 tools/libxl/libxl_checkpoint_device.c | 42 ++-
 tools/libxl/libxl_remus.c | 42 +++
 2 files changed, 44 insertions(+), 40 deletions(-)

diff --git a/tools/libxl/libxl_checkpoint_device.c 
b/tools/libxl/libxl_checkpoint_device.c
index bbc6dc4..0a16dbb 100644
--- a/tools/libxl/libxl_checkpoint_device.c
+++ b/tools/libxl/libxl_checkpoint_device.c
@@ -17,38 +17,6 @@
 
 #include "libxl_internal.h"
 
-/*- helper functions -*/
-
-static int init_device_subkind(libxl__checkpoint_devices_state *cds)
-{
-/* init device subkind-specific state in the libxl ctx */
-int rc;
-STATE_AO_GC(cds->ao);
-
-if (libxl__netbuffer_enabled(gc)) {
-rc = init_subkind_nic(cds);
-if (rc) goto out;
-}
-
-rc = init_subkind_drbd_disk(cds);
-if (rc) goto out;
-
-rc = 0;
-out:
-return rc;
-}
-
-static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
-{
-/* cleanup device subkind-specific state in the libxl ctx */
-STATE_AO_GC(cds->ao);
-
-if (libxl__netbuffer_enabled(gc))
-cleanup_subkind_nic(cds);
-
-cleanup_subkind_drbd_disk(cds);
-}
-
 /*- setup() and teardown() -*/
 
 /* callbacks */
@@ -86,14 +54,10 @@ static void checkpoint_devices_setup(libxl__egc *egc,
 void libxl__checkpoint_devices_setup(libxl__egc *egc,
  libxl__checkpoint_devices_state *cds)
 {
-int i, rc;
+int i;
 
 STATE_AO_GC(cds->ao);
 
-rc = init_device_subkind(cds);
-if (rc)
-goto out;
-
 cds->num_devices = 0;
 cds->num_nics = 0;
 cds->num_disks = 0;
@@ -126,7 +90,7 @@ void libxl__checkpoint_devices_setup(libxl__egc *egc,
 return;
 
 out:
-cds->callback(egc, cds, rc);
+cds->callback(egc, cds, 0);
 }
 
 static void checkpoint_devices_setup(libxl__egc *egc,
@@ -263,8 +227,6 @@ static void devices_teardown_cb(libxl__egc *egc,
 cds->disks = NULL;
 cds->num_disks = 0;
 
-cleanup_device_subkind(cds);
-
 cds->callback(egc, cds, rc);
 }
 
diff --git a/tools/libxl/libxl_remus.c b/tools/libxl/libxl_remus.c
index e83cdc9..54ec7de 100644
--- a/tools/libxl/libxl_remus.c
+++ b/tools/libxl/libxl_remus.c
@@ -26,6 +26,38 @@ static const libxl__checkpoint_device_instance_ops 
*remus_ops[] = {
 NULL,
 };
 
+/*- helper functions -*/
+
+static int init_device_subkind(libxl__checkpoint_devices_state *cds)
+{
+/* init device subkind-specific state in the libxl ctx */
+int rc;
+STATE_AO_GC(cds->ao);
+
+if (libxl__netbuffer_enabled(gc)) {
+rc = init_subkind_nic(cds);
+if (rc) goto out;
+}
+
+rc = init_subkind_drbd_disk(cds);
+if (rc) goto out;
+
+rc = 0;
+out:
+return rc;
+}
+
+static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
+{
+/* cleanup device subkind-specific state in the libxl ctx */
+STATE_AO_GC(cds->ao);
+
+if (libxl__netbuffer_enabled(gc))
+cleanup_subkind_nic(cds);
+
+cleanup_subkind_drbd_disk(cds);
+}
+
 /* Remus setup and teardown -*/
 
 static void remus_setup_done(libxl__egc *egc,
@@ -68,6 +100,12 @@ void libxl__remus_setup(libxl__egc *egc, libxl__remus_state 
*rs)
 cds->concrete_data = rs;
 rs->interval = info->interval;
 
+if (init_device_subkind(cds)) {
+LOG(ERROR, "Remus: failed to init device subkind for guest %u",
+dss->domid);
+goto out;
+}
+
 dss->sws.checkpoint_callback = remus_checkpoint_stream_written;
 
 callbacks->suspend = libxl__remus_domain_suspend_callback;
@@ -108,6 +146,8 @@ static void remus_setup_failed(libxl__egc *egc,
 LOG(ERROR, "Remus: failed to teardown device after setup failed"
 " for guest with domid %u, rc %d", dss->domid, rc);
 
+cleanup_device_subkind(cds);
+
 dss->callback(egc, dss, rc);
 }
 
@@ -142,6 +182,8 @@ static void remus_teardown_done(libxl__egc *egc,
 LOG(ERROR, "Remus: failed to teardown device for guest with domid %u,"
 " rc %d", dss->domid, rc);
 
+cleanup_device_subkind(cds);
+
 dss->callback(egc, dss, rc);
 }
 
-- 
2.5.0




___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel