Re: [Xen-devel] [PATCH 2/2] domain: use PGC_extra domheap page for shared_info

2020-03-06 Thread David Woodhouse
On Fri, 2020-03-06 at 13:29 +0100, Jan Beulich wrote: > How do you tell pages in use by domains from ones free to re-use? > Because of the overloading of struct page_info, I expect you can't > judge by just looking at a page's struct page_info instance. Are > you peeking into the migration streams

Re: [Xen-devel] [PATCH 2/2] domain: use PGC_extra domheap page for shared_info

2020-03-06 Thread David Woodhouse
On Fri, 2020-03-06 at 13:36 +0100, Jan Beulich wrote: > Oh, so really this is an optimization to allow the memory range to > not remain unused altogether by "old" Xen, i.e. unlike the kexec > range. Right. At the moment I just *don't* use the pages in the reserved region (and that includes initte

Re: [Xen-devel] [PATCH 2/2] domain: use PGC_extra domheap page for shared_info

2020-03-06 Thread David Woodhouse
On Fri, 2020-03-06 at 13:55 +0100, Jan Beulich wrote: > I'm sorry, but this doesn't really make things much more clear. > Further up you say "As it processes the live update information > ...", i.e. that's a case where you get positive indication that a > page is in use. You also have that reserved

Re: [Xen-devel] [PATCH 2/2] domain: use PGC_extra domheap page for shared_info

2020-03-06 Thread David Woodhouse
On Fri, 2020-03-06 at 14:10 +0100, Jan Beulich wrote: > Establishing of what the new separation rule and mechanism is going > to be (no matter how the two resulting pieces are going to be > named). Paul's opinion seemed to be that with secret hiding coming along and destroying the "xenheap is mapp

[Xen-devel] [PATCH] Add -MP to CFLAGS along with -MMD.

2020-03-17 Thread David Woodhouse
From: David Woodhouse This causes gcc (yes, and clang) to emit phony targets for each dependency. This means that when a header file is deleted, the C files which *used* to include it will no longer stop building with bogus out-of-date dependencies like this: make[5]: *** No rule to make

Re: [Xen-devel] [PATCH] Add -MP to CFLAGS along with -MMD.

2020-03-17 Thread David Woodhouse
On Tue, 2020-03-17 at 15:52 +0100, Jan Beulich wrote: > On 17.03.2020 15:34, David Woodhouse wrote: > > From: David Woodhouse > > > > This causes gcc (yes, and clang) to emit phony targets for each dependency. > > > > This means that when a header file is

Re: [Xen-devel] [PATCH] Add -MP to CFLAGS along with -MMD.

2020-03-17 Thread David Woodhouse
On Tue, 2020-03-17 at 16:59 +, Ian Jackson wrote: > David Woodhouse writes ("Re: [PATCH] Add -MP to CFLAGS along with -MMD."): > > On Tue, 2020-03-17 at 15:52 +0100, Jan Beulich wrote: > > > On 17.03.2020 15:34, David Woodhouse wrote: > > > > From: David

Re: [Xen-devel] [PATCH 1/2] xen/mm: fold PGC_broken into PGC_state bits

2020-03-17 Thread David Woodhouse
On Fri, 2020-02-07 at 20:27 +, Julien Grall wrote: > > +switch ( x & PGC_state ) > > { > > -nx &= ~PGC_state; > > -nx |= (((x & PGC_state) == PGC_state_free) > > - ? PGC_state_offlined : PGC_state_offlining); > > -} > > +

Re: [Xen-devel] [PATCH 1/2] xen/mm: fold PGC_broken into PGC_state bits

2020-03-17 Thread David Woodhouse
On Thu, 2020-02-20 at 12:10 +0100, Jan Beulich wrote: > On 07.02.2020 16:57, David Woodhouse wrote: > > @@ -1145,16 +1145,19 @@ static int reserve_offlined_page(struct > > page_info *head) > > > > for ( cur_head = head; cur_head < head + ( 1U

Re: [Xen-devel] [PATCH 2/2] xen/mm: Introduce PG_state_uninitialised

2020-03-17 Thread David Woodhouse
On Thu, 2020-02-20 at 12:59 +0100, Jan Beulich wrote: > On 07.02.2020 19:04, David Woodhouse wrote: > > --- a/xen/arch/x86/mm.c > > +++ b/xen/arch/x86/mm.c > > @@ -488,7 +488,8 @@ void share_xen_page_with_guest(struct page_info *page, > > struct domain *d, > >

Re: [Xen-devel] [PATCH 8/8] x86/setup: lift dom0 creation out into create_dom0() function

2020-03-17 Thread David Woodhouse
On Fri, 2020-02-21 at 18:06 +0100, Jan Beulich wrote: > On 01.02.2020 01:33, David Woodhouse wrote: > > --- a/xen/arch/x86/setup.c > > +++ b/xen/arch/x86/setup.c > > @@ -678,6 +678,92 @@ static unsigned int __init copy_bios_e820(struct > > e820entry *map, unsign

[Xen-devel] [PATCH 0/2] x86/setup: Dom0 creation cleanups

2020-03-18 Thread David Woodhouse
nup in its own right in the meantime. David Woodhouse (2): x86/setup: simplify handling of initrdidx when no initrd present x86/setup: lift dom0 creation out into create_dom0() function xen/arch/x86/setup.c | 175 --- 1 file changed, 95

[Xen-devel] [PATCH 1/2] x86/setup: simplify handling of initrdidx when no initrd present

2020-03-18 Thread David Woodhouse
From: David Woodhouse Remove a ternary operator that made my brain hurt. Replace it with something simpler that makes it somewhat clearer that the check for initrdidx < mbi->mods_count is because larger values are what find_first_bit() will return when it doesn't find anything. Al

[Xen-devel] [PATCH 2/2] x86/setup: lift dom0 creation out into create_dom0() function

2020-03-18 Thread David Woodhouse
From: David Woodhouse The creation of dom0 can be relatively self-contained. Shift it into a separate function and simplify __start_xen() a little bit. This is a cleanup in its own right, but will be even more desireable when live update provides an alternative path through __start_xen() that

Re: [Xen-devel] [PATCH 2/2] xen/mm: Introduce PG_state_uninitialised

2020-03-18 Thread David Woodhouse
On Wed, 2020-03-18 at 11:03 +0100, Jan Beulich wrote: > On 17.03.2020 23:15, David Woodhouse wrote: > > On Thu, 2020-02-20 at 12:59 +0100, Jan Beulich wrote: > > > On 07.02.2020 19:04, David Woodhouse wrote: > > > > ASSERT((pg[i].count_info &a

Re: [Xen-devel] [PATCH 1/2] x86/setup: simplify handling of initrdidx when no initrd present

2020-03-18 Thread David Woodhouse
On Wed, 2020-03-18 at 12:51 +0100, Jan Beulich wrote: > On 18.03.2020 12:46, David Woodhouse wrote: > > From: David Woodhouse > > > > Remove a ternary operator that made my brain hurt. > > My position towards this hasn't changed, just ftr. Your position

Re: [Xen-devel] [PATCH 1/2] xen/mm: fold PGC_broken into PGC_state bits

2020-03-18 Thread David Woodhouse
On Wed, 2020-03-18 at 12:31 +, Julien Grall wrote: > On 18/03/2020 09:56, Jan Beulich wrote: > > On 17.03.2020 22:52, David Woodhouse wrote: > > > On Thu, 2020-02-20 at 12:10 +0100, Jan Beulich wrote: > > > > > @@ -1699,14 +1714,14 @@ unsigned int online_p

Re: [Xen-devel] [PATCH] Add -MP to CFLAGS along with -MMD.

2020-03-18 Thread David Woodhouse
On Tue, 2020-03-17 at 16:59 +, Ian Jackson wrote: > David Woodhouse writes ("Re: [PATCH] Add -MP to CFLAGS along with -MMD."): > > On Tue, 2020-03-17 at 15:52 +0100, Jan Beulich wrote: > > > On 17.03.2020 15:34, David Woodhouse wrote: > > > > From: David

Re: [Xen-devel] [PATCH] Add -MP to CFLAGS along with -MMD.

2020-03-18 Thread David Woodhouse
On Wed, 2020-03-18 at 18:19 +, Ian Jackson wrote: > David Woodhouse writes ("Re: [PATCH] Add -MP to CFLAGS along with -MMD."): > > OK... I have attempted to address my frustration in a more coherent and > > hopefully productive way (qv), rather than resorting to m

Re: [Xen-devel] [PATCH 1/2] xen/mm: fold PGC_broken into PGC_state bits

2020-03-19 Thread David Woodhouse
On Thu, 2020-03-19 at 09:49 +0100, Jan Beulich wrote: > On 18.03.2020 18:13, David Woodhouse wrote: > > On Wed, 2020-03-18 at 12:31 +, Julien Grall wrote: > > > On 18/03/2020 09:56, Jan Beulich wrote: > > > > On 17.03.2020 22:52, David Woodhouse wrote: > &g

Re: [Xen-devel] [PATCH 1/2] xen/mm: fold PGC_broken into PGC_state bits

2020-03-19 Thread David Woodhouse
On Thu, 2020-03-19 at 12:59 +0100, Jan Beulich wrote: > > Read that message again from the point of view of a contributor. > > Pretend it isn't even me; pretend it's someone attempting to make > > their first, trivial, improvement to the Xen ecosystem. > > > > I hope you'll understand why my initi

[Xen-devel] [PATCH 1/2] tools/xenstore: Do not abort xenstore-ls if a node disappears while iterating

2020-03-19 Thread David Woodhouse
From: David Woodhouse The do_ls() function has somewhat inconsistent handling of errors. If reading the node's contents with xs_read() fails, then do_ls() will just quietly not display the contents. If reading the node's permissions with xs_get_permissions() fails, then do_ls() wi

[Xen-devel] [PATCH 2/2] tools/xenstore: Accumulate errors in xenstore-ls and exit appropriately

2020-03-19 Thread David Woodhouse
From: David Woodhouse Report only one error to stderr for each node, regardless of whether it's xs_read, xs_get_permissions or xs_directory on the child that fails. Always exit with a non-zero code if any failure happens, reporting the last error that occurred. Signed-off-by: David Wood

[Xen-devel] [PATCH 0/2] Handle

2020-03-19 Thread David Woodhouse
init_heap_pages(). David Woodhouse (2): xen/mm: fold PGC_broken into PGC_state bits xen/mm: Introduce PGC_state_uninitialised xen/arch/x86/domctl.c| 2 +- xen/arch/x86/mm.c| 3 +- xen/common/page_alloc.c | 110 +-- xen/include/asm

[Xen-devel] [PATCH 2/2] xen/mm: Introduce PGC_state_uninitialised

2020-03-19 Thread David Woodhouse
From: David Woodhouse It is possible for pages to enter general circulation without ever being process by init_heap_pages(). For example, pages of the multiboot module containing the initramfs may be assigned via assign_pages() to dom0 as it is created. And some code including map_pages_to_xen

[Xen-devel] [PATCH 1/2] xen/mm: fold PGC_broken into PGC_state bits

2020-03-19 Thread David Woodhouse
From: David Woodhouse Only PGC_state_offlining and PGC_state_offlined are valid in conjunction with PGC_broken. The other two states (free and inuse) were never valid for a broken page. By folding PGC_broken in, we can have three bits for PGC_state which allows up to 8 states, of which 6 are

Re: [Xen-devel] [PATCH 1/2] tools/xenstore: Do not abort xenstore-ls if a node disappears while iterating

2020-03-20 Thread David Woodhouse
On Fri, 2020-03-20 at 07:34 +0100, Jürgen Groß wrote: > Have you thought about the possibility to do the complete handling in a > single transaction? This would ensure a complete consistent picture > from the time the operation has started. Any inconsistency should be > reported as an error then.

Re: [Xen-devel] [PATCH 2/2] tools/xenstore: Accumulate errors in xenstore-ls and exit appropriately

2020-03-20 Thread David Woodhouse
On Fri, 2020-03-20 at 11:03 +, Paul Durrant wrote: > > +val = NULL; > > +/* Don't repeat an error message if xs_read() already > > failed */ > > +if (val) > > How can the code get here? The line above the comment always sets val to NULL. Oops,

Re: [Xen-devel] [PATCH 1/2] tools/xenstore: Do not abort xenstore-ls if a node disappears while iterating

2020-03-20 Thread David Woodhouse
On Fri, 2020-03-20 at 14:12 +, Ian Jackson wrote: > Jürgen Groß writes ("Re: [Xen-devel] [PATCH 1/2] tools/xenstore: Do not abort > xenstore-ls if a node disappears while iterating"): > > No, you just don't care for the transaction to succeed or fail (IMO it > > should never fail as you are re

Re: [Xen-devel] [PATCH 2/2] xen/mm: Introduce PGC_state_uninitialised

2020-03-20 Thread David Woodhouse
On Fri, 2020-03-20 at 13:33 +, Paul Durrant wrote: > > -Original Message- > > From: Xen-devel On Behalf Of David > > Woodhouse > > Sent: 19 March 2020 21:22 > > To: xen-devel@lists.xenproject.org > > Cc: Stefano Stabellini ; Julien Grall >

Re: [Xen-devel] [PATCH 2/2] xen/mm: Introduce PGC_state_uninitialised

2020-03-23 Thread David Woodhouse
On Mon, 2020-03-23 at 08:49 +, Paul Durrant wrote: > Ok, so deferring the call to free_heap_pages() (and consequently > init_heap_pages()) is safe to defer until the guest is torn down? (Or > is this only safe if the page is being assigned to the initial > domain?) It's intended to be safe in

Re: [Xen-devel] [PATCH 2/2] xen/mm: Introduce PGC_state_uninitialised

2020-03-23 Thread David Woodhouse
On Mon, 2020-03-23 at 09:34 +, Julien Grall wrote: > For liveupdate, we will need a way to initialize a page but mark it as > already inuse (i.e in the same state as they would be if allocated > normally). I am unconvinced of the veracity of this claim. We don't want to turn specific detail

Re: [Xen-devel] [PATCH 2/2] xen/mm: Introduce PGC_state_uninitialised

2020-03-24 Thread David Woodhouse
On Tue, 2020-03-24 at 10:08 +, Julien Grall wrote: > Hi David, > > On 23/03/2020 10:55, David Woodhouse wrote: > > On Mon, 2020-03-23 at 09:34 +, Julien Grall wrote: > > > For liveupdate, we will need a way to initialize a page but mark it as > > > alread

[Xen-devel] [RFC PATCH 2/3] x86/boot: Reserve live update boot memory

2020-01-08 Thread David Woodhouse
From: David Woodhouse For live update to work, it will need a region of memory that can be given to the boot allocator while it parses the state information from the previous Xen and works out which of the other pages of memory it can consume. Reserve that like the crashdump region, and accept

[Xen-devel] [RFC PATCH 3/3] Add KEXEC_RANGE_MA_LIVEUPDATE

2020-01-08 Thread David Woodhouse
From: David Woodhouse This allows kexec userspace to tell the next Xen where the range is, on its command line. Signed-off-by: David Woodhouse --- xen/arch/x86/machine_kexec.c | 15 --- xen/arch/x86/setup.c | 2 +- xen/include/public/kexec.h | 1 + 3 files changed, 14

[Xen-devel] [RFC PATCH 0/3] Live update boot memory management

2020-01-08 Thread David Woodhouse
ht be given to share_xen_page_with_guest() is allocated appropriately. The part which actually hands over the live update state isn't included yet, so this really does just *defer* the addition of the memory until a little bit later in __start_xen(). Actually taking ranges out of it will com

[Xen-devel] [RFC PATCH 1/3] x86/setup: Don't skip 2MiB underneath relocated Xen image

2020-01-08 Thread David Woodhouse
From: David Woodhouse Set 'e' correctly to reflect the location that Xen is actually relocated to from its default 2MiB location. Not 2MiB below that. Signed-off-by: David Woodhouse --- xen/arch/x86/setup.c | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git

Re: [Xen-devel] [RFC PATCH 1/3] x86/setup: Don't skip 2MiB underneath relocated Xen image

2020-01-10 Thread David Woodhouse
On Fri, 2020-01-10 at 11:15 +, Durrant, Paul wrote: > > -Original Message- > > From: Xen-devel On Behalf > > Of > > David Woodhouse > > Sent: 08 January 2020 17:25 > > To: Xen-devel > > Cc: Stefano Stabellini ; Julien Grall > > ; Wei

Re: [Xen-devel] [RFC PATCH 0/3] Live update boot memory management

2020-01-13 Thread David Woodhouse
On Wed, 2020-01-08 at 17:24 +, David Woodhouse wrote: > When doing a live update, Xen needs to be very careful not to scribble > on pages which contain guest memory or state information for the > domains which are being preserved. > > The information about which pages are in u

Re: [Xen-devel] [RFC PATCH V2 11/11] x86: tsc: avoid system instability in hibernation

2020-01-13 Thread David Woodhouse
On Mon, 2020-01-13 at 13:01 +, Andrew Cooper wrote: > On 13/01/2020 11:43, Singh, Balbir wrote: > > On Mon, 2020-01-13 at 11:16 +0100, Peter Zijlstra wrote: > > > On Fri, Jan 10, 2020 at 07:35:20AM -0800, Eduardo Valentin wrote: > > > > Hey Peter, > > > > > > > > On Wed, Jan 08, 2020 at 11:50:

Re: [Xen-devel] [RFC PATCH 0/3] Live update boot memory management

2020-01-14 Thread David Woodhouse
On Tue, 2020-01-14 at 14:15 +, Julien Grall wrote: > Hi David, > > On 13/01/2020 11:54, David Woodhouse wrote: > > On Wed, 2020-01-08 at 17:24 +, David Woodhouse wrote: > > > When doing a live update, Xen needs to be very careful not to scribble > > > on

Re: [Xen-devel] [RFC PATCH 0/3] Live update boot memory management

2020-01-14 Thread David Woodhouse
On Tue, 2020-01-14 at 15:00 +, Julien Grall wrote: > > On 14/01/2020 14:48, David Woodhouse wrote: > > On Tue, 2020-01-14 at 14:15 +, Julien Grall wrote: > > > Hi David, > > > > > > On 13/01/2020 11:54, David Woodhouse wrote: > > > >

Re: [Xen-devel] [RFC PATCH 0/3] Live update boot memory management

2020-01-14 Thread David Woodhouse
On Tue, 2020-01-14 at 16:29 +, Julien Grall wrote: > > That's the point in appending an IND_WRITE64 operation to the kimage > > stream. The actual write is done in the last gasp of kexec_reloc() > > after Xen#1 is quiescent, on the way into purgatory. > > I was not sure what you meant by IND_W

Re: [Xen-devel] [RFC PATCH 2/3] x86/boot: Reserve live update boot memory

2020-01-20 Thread David Woodhouse
On Mon, 2020-01-20 at 17:58 +0100, Jan Beulich wrote: > On 08.01.2020 18:24, David Woodhouse wrote: > > @@ -980,6 +1015,22 @@ void __init noreturn __start_xen(unsigned long mbi_p) > > set_kexec_crash_area_size((u64)nr_pages << PAGE_SHIFT); > > ke

[Xen-devel] [RFC PATCH v2 0/14] Live update: boot memory management, data stream handling

2020-01-22 Thread David Woodhouse
Expanding on the previous WIP patch set. As before, reserve a contiguous region of memory which can be safely used by the boot allocator in the new Xen, before the live update data stream has been processed and thus before the locations of all the other pages which contain live domain data are kno

[Xen-devel] [RFC PATCH v2 14/14] debug hacks

2020-01-22 Thread David Woodhouse
From: David Woodhouse --- xen/common/lu/restore.c | 7 +++ xen/common/lu/stream.c | 2 ++ xen/common/page_alloc.c | 6 ++ 3 files changed, 15 insertions(+) diff --git a/xen/common/lu/restore.c b/xen/common/lu/restore.c index 7f40513ef9..fa168bdac1 100644 --- a/xen/common/lu/restore.c

[Xen-devel] [RFC PATCH v2 04/14] Add KEXEC_RANGE_MA_LIVEUPDATE

2020-01-22 Thread David Woodhouse
From: David Woodhouse This allows kexec userspace to tell the next Xen where the range is, on its command line. Signed-off-by: David Woodhouse --- xen/arch/x86/machine_kexec.c | 13 ++--- xen/arch/x86/setup.c | 2 +- xen/include/public/kexec.h | 1 + 3 files changed, 12

[Xen-devel] [RFC PATCH v2 01/14] x86/setup: Don't skip 2MiB underneath relocated Xen image

2020-01-22 Thread David Woodhouse
From: David Woodhouse Set 'e' correctly to reflect the location that Xen is actually relocated to from its default 2MiB location. Not 2MiB below that. This is only vaguely a bug fix. The "missing" 2MiB would have been used in the end, and fed to the allocator. It's jus

[Xen-devel] [RFC PATCH v2 12/14] Don't add bad pages above HYPERVISOR_VIRT_END to the domheap

2020-01-22 Thread David Woodhouse
From: David Woodhouse Signed-off-by: David Woodhouse --- xen/common/page_alloc.c | 83 +++-- 1 file changed, 80 insertions(+), 3 deletions(-) diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index 20ef25d45a..2a20c12abb 100644 --- a/xen/common

[Xen-devel] [RFC PATCH v2 10/14] Put *something* into LU data...

2020-01-22 Thread David Woodhouse
From: David Woodhouse Signed-off-by: David Woodhouse --- xen/common/lu/save.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/xen/common/lu/save.c b/xen/common/lu/save.c index c767abd8f4..24d727ff47 100644 --- a/xen/common/lu/save.c +++ b/xen/common/lu/save.c @@ -7,7 +7,7

[Xen-devel] [RFC PATCH v2 05/14] Add KEXEC_TYPE_LIVE_UPDATE

2020-01-22 Thread David Woodhouse
From: David Woodhouse This is identical to the default case... for now. Signed-off-by: David Woodhouse --- xen/common/kexec.c | 18 ++ xen/include/public/kexec.h | 12 2 files changed, 26 insertions(+), 4 deletions(-) diff --git a/xen/common/kexec.c b/xen

[Xen-devel] [RFC PATCH v2 06/14] Add IND_WRITE64 primitive to kexec kimage

2020-01-22 Thread David Woodhouse
From: David Woodhouse This allows a single page-aligned physical address to be written to the current destination, intended to pass the location of the live update data stream from one Xen to the next. Signed-off-by: David Woodhouse --- xen/arch/x86/x86_64/kexec_reloc.S | 9 - xen

[Xen-devel] [RFC PATCH v2 03/14] Don't add unused parts of live update reserved bootmem to heap

2020-01-22 Thread David Woodhouse
From: David Woodhouse We need to promise that we won't use it for anything that will need to persist across live update. Until we can achieve that (which will build on of some of the work Hongyan is doing for secret hiding), the easy answer is just not to add that region to the main heap a

[Xen-devel] [RFC PATCH v2 09/14] Add basic lu_save_all() shell

2020-01-22 Thread David Woodhouse
From: David Woodhouse --- xen/common/kexec.c | 6 ++ xen/common/lu/Makefile | 2 +- xen/common/lu/save.c | 45 ++ xen/include/xen/lu.h | 3 +++ 4 files changed, 55 insertions(+), 1 deletion(-) create mode 100644 xen/common/lu/save.c diff

[Xen-devel] [RFC PATCH v2 07/14] Add kimage_add_live_update_data()

2020-01-22 Thread David Woodhouse
From: David Woodhouse Signed-off-by: David Woodhouse --- xen/common/kimage.c | 33 + xen/include/asm-x86/config.h | 2 ++ xen/include/xen/kimage.h | 3 +++ 3 files changed, 38 insertions(+) diff --git a/xen/common/kimage.c b/xen/common/kimage.c

[Xen-devel] [RFC PATCH v2 02/14] x86/boot: Reserve live update boot memory

2020-01-22 Thread David Woodhouse
From: David Woodhouse For live update to work, it will need a region of memory that can be given to the boot allocator while it parses the state information from the previous Xen and works out which of the other pages of memory it can consume. Reserve that like the crashdump region, and accept

[Xen-devel] [RFC PATCH v2 13/14] Basic shell of lu_reserve_all() from breadcrumb at boot

2020-01-22 Thread David Woodhouse
From: David Woodhouse This mostly just reserves pages, hence the name. The rest of the actual restoration will happen later, after the heap allocator works. Signed-off-by: David Woodhouse --- xen/arch/x86/setup.c| 20 +++ xen/common/lu/Makefile | 2 +- xen/common/lu

[Xen-devel] [RFC PATCH v2 08/14] Add basic live update stream creation

2020-01-22 Thread David Woodhouse
From: David Woodhouse Signed-off-by: David Woodhouse --- xen/common/Makefile| 1 + xen/common/lu/Makefile | 1 + xen/common/lu/stream.c | 108 + xen/include/xen/lu.h | 12 + 4 files changed, 122 insertions(+) create mode 100644 xen/common

[Xen-devel] [RFC PATCH v2 11/14] Don't panic if no multiboot modules are provided on live update boot

2020-01-22 Thread David Woodhouse
From: David Woodhouse Signed-off-by: David Woodhouse --- xen/arch/x86/setup.c | 11 --- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c index 521946dc27..bd65d6bf5d 100644 --- a/xen/arch/x86/setup.c +++ b/xen/arch/x86/setup.c

Re: [Xen-devel] [ANNOUNCE] Xen 4.14 Development Update

2020-01-29 Thread David Woodhouse
ies has been re-posted? Also, can the owners of any completed items > please respond so that the item can be moved into the 'Completed' section. > > = Projects = > > == Hypervisor == > > * Live-Updating Xen > - David Woodhouse Latest RFC patchset posted is

[Xen-devel] [RFC PATCH v3 03/22] Reserve live update memory regions

2020-01-30 Thread David Woodhouse
From: David Woodhouse The live update handover requires that a region of memory be reserved for the new Xen to use in its boot allocator. The original Xen may use that memory but not for any pages which are mapped to domains, or which would need to be preserved across the live update for any

[Xen-devel] [RFC PATCH v3 0/22] Live update: boot memory management, data stream handling, record format

2020-01-30 Thread David Woodhouse
t tree and we're working on cleaning it up for public consumption. David Woodhouse (21): x86/setup: Don't skip 2MiB underneath relocated Xen image x86/boot: Reserve live update boot memory Reserve live update memory regions Add KEXEC_RANGE_MA_LIVEUPDATE Add KEXEC

[Xen-devel] [RFC PATCH v3 14/22] Detect live update breadcrumb at boot and map data stream

2020-01-30 Thread David Woodhouse
From: David Woodhouse Signed-off-by: David Woodhouse --- xen/arch/x86/setup.c | 35 +-- xen/common/lu/stream.c | 34 ++ xen/include/xen/lu.h | 2 ++ 3 files changed, 69 insertions(+), 2 deletions(-) diff --git a/xen/arch/x86

[Xen-devel] [RFC PATCH v3 15/22] Start documenting the live update handover

2020-01-30 Thread David Woodhouse
From: David Woodhouse Signed-off-by: David Woodhouse --- docs/specs/libxc-migration-stream.pandoc | 19 +- docs/specs/live-update-handover.pandoc | 371 +++ 2 files changed, 388 insertions(+), 2 deletions(-) create mode 100644 docs/specs/live-update-handover.pandoc

[Xen-devel] [RFC PATCH v3 22/22] x86/setup: simplify handling of initrdidx when no initrd present

2020-01-30 Thread David Woodhouse
From: David Woodhouse Remove a ternary operator that made my brain hurt and replace it with something simpler that makes it clearer that the >= mbi->mods_count is because of what find_first_bit() returns when it doesn't find anything. Just have a simple condition to set initrdidx to z

[Xen-devel] [RFC PATCH v3 01/22] x86/setup: Don't skip 2MiB underneath relocated Xen image

2020-01-30 Thread David Woodhouse
From: David Woodhouse Set 'e' correctly to reflect the location that Xen is actually relocated to from its default 2MiB location. Not 2MiB below that. This is only vaguely a bug fix. The "missing" 2MiB would have been used in the end, and fed to the allocator. It's jus

[Xen-devel] [RFC PATCH v3 04/22] Add KEXEC_RANGE_MA_LIVEUPDATE

2020-01-30 Thread David Woodhouse
From: David Woodhouse This allows kexec userspace to tell the next Xen where the range is, on its command line. Signed-off-by: David Woodhouse --- xen/arch/x86/machine_kexec.c | 13 ++--- xen/include/public/kexec.h | 1 + 2 files changed, 11 insertions(+), 3 deletions(-) diff

[Xen-devel] [RFC PATCH v3 18/22] Add LU_VERSION and LU_END records to live update stream

2020-01-30 Thread David Woodhouse
From: David Woodhouse Signed-off-by: David Woodhouse --- xen/common/lu/save.c | 13 - xen/include/public/migration_stream.h | 9 + 2 files changed, 21 insertions(+), 1 deletion(-) diff --git a/xen/common/lu/save.c b/xen/common/lu/save.c index c43962c44e

[Xen-devel] [RFC PATCH v3 10/22] Don't add bad pages above HYPERVISOR_VIRT_END to the domheap

2020-01-30 Thread David Woodhouse
From: David Woodhouse Signed-off-by: David Woodhouse --- xen/common/page_alloc.c | 83 +++-- 1 file changed, 80 insertions(+), 3 deletions(-) diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c index a74bf02559..4ada4412dd 100644 --- a/xen/common

[Xen-devel] [RFC PATCH v3 07/22] Add basic live update stream creation

2020-01-30 Thread David Woodhouse
From: David Woodhouse Signed-off-by: David Woodhouse --- xen/common/Makefile| 1 + xen/common/lu/Makefile | 1 + xen/common/lu/stream.c | 135 + xen/include/xen/lu.h | 29 + 4 files changed, 166 insertions(+) create mode 100644 xen

[Xen-devel] [RFC PATCH v3 12/22] xen/vmap: allow vmap() to be called during early boot

2020-01-30 Thread David Woodhouse
From: David Woodhouse Signed-off-by: David Woodhouse --- xen/common/vmap.c | 23 +-- 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/xen/common/vmap.c b/xen/common/vmap.c index 37922f735b..8343460794 100644 --- a/xen/common/vmap.c +++ b/xen/common/vmap.c

[Xen-devel] [RFC PATCH v3 13/22] x86/setup: move vm_init() before end_boot_allocator()

2020-01-30 Thread David Woodhouse
From: David Woodhouse We would like to be able to use vmap() to map the live update data, and we need to do a first pass of the live update data before we prime the heap because we need to know which pages need to be preserved. Signed-off-by: David Woodhouse --- xen/arch/x86/setup.c | 8

[Xen-devel] [RFC PATCH v3 06/22] Add IND_WRITE64 primitive to kexec kimage

2020-01-30 Thread David Woodhouse
From: David Woodhouse This allows a single page-aligned physical address to be written to the current destination, intended to pass the location of the live update data stream from one Xen to the next. Signed-off-by: David Woodhouse --- xen/arch/x86/x86_64/kexec_reloc.S | 9 - xen

[Xen-devel] [RFC PATCH v3 21/22] x86/setup: finish plumbing in live update path through __start_xen()

2020-01-30 Thread David Woodhouse
From: David Woodhouse With this we are fairly much done hacking up __start_xen() to support live update. The live update functions themselves are still stubs, but now we can start populating those with actual save/restore of domain information. Signed-off-by: David Woodhouse --- xen/arch/x86

[Xen-devel] [RFC PATCH v3 09/22] Add basic lu_save_all() shell

2020-01-30 Thread David Woodhouse
From: David Woodhouse --- xen/common/kexec.c | 6 + xen/common/lu/Makefile | 2 +- xen/common/lu/save.c | 56 ++ xen/include/xen/lu.h | 3 +++ 4 files changed, 66 insertions(+), 1 deletion(-) create mode 100644 xen/common/lu/save.c diff

[Xen-devel] [RFC PATCH v3 19/22] Add shell of lu_reserve_pages()

2020-01-30 Thread David Woodhouse
From: David Woodhouse This currently only iterates over the records and prints the version of Xen that we're live updating from. In the fullness of time, it will also reserve the pages passed over as M2P as well as the pages belonging to preserved domains. Signed-off-by: David Wood

[Xen-devel] [RFC PATCH v3 16/22] Migrate migration stream definitions into Xen public headers

2020-01-30 Thread David Woodhouse
From: David Woodhouse Signed-off-by: David Woodhouse --- tools/libxc/xc_sr_common.c| 20 ++-- tools/libxc/xc_sr_common_x86.c| 4 +- tools/libxc/xc_sr_restore.c | 2 +- tools/libxc/xc_sr_restore_x86_hvm.c | 4 +- tools/libxc/xc_sr_restore_x86_pv.c| 8

[Xen-devel] [RFC PATCH v3 08/22] Add kimage_add_live_update_data()

2020-01-30 Thread David Woodhouse
From: David Woodhouse Signed-off-by: David Woodhouse --- xen/common/kimage.c | 34 ++ xen/include/xen/kimage.h | 3 +++ xen/include/xen/lu.h | 3 +++ 3 files changed, 40 insertions(+) diff --git a/xen/common/kimage.c b/xen/common/kimage.c index

[Xen-devel] [RFC PATCH v3 05/22] Add KEXEC_TYPE_LIVE_UPDATE

2020-01-30 Thread David Woodhouse
From: David Woodhouse This is identical to the default case... for now. Signed-off-by: David Woodhouse --- xen/common/kexec.c | 18 ++ xen/include/public/kexec.h | 12 2 files changed, 26 insertions(+), 4 deletions(-) diff --git a/xen/common/kexec.c b/xen

[Xen-devel] [RFC PATCH v3 17/22] Add lu_stream_{open, close, append}_record()

2020-01-30 Thread David Woodhouse
From: David Woodhouse Signed-off-by: David Woodhouse --- xen/common/lu/stream.c | 50 ++ xen/include/xen/lu.h | 5 + 2 files changed, 55 insertions(+) diff --git a/xen/common/lu/stream.c b/xen/common/lu/stream.c index 8c44a4eb37..2ee870e80a

[Xen-devel] [RFC PATCH v3 20/22] x86/setup: lift dom0 creation out into create_dom0 function

2020-01-30 Thread David Woodhouse
From: David Woodhouse It's about to become optional as __start_xen() grows a different path for live update, so move it out of the way. Signed-off-by: David Woodhouse --- xen/arch/x86/setup.c | 173 +++ 1 file changed, 94 insertions(+), 79 dele

[Xen-devel] [RFC PATCH v3 11/22] xen/vmap: allow vm_init_type to be called during early_boot

2020-01-30 Thread David Woodhouse
From: Wei Liu We want to move vm_init, which calls vm_init_type under the hood, to early boot stage. Add a path to get page from boot allocator instead. Add an emacs block to that file while I was there. Signed-off-by: Wei Liu --- xen/common/vmap.c | 22 +++--- 1 file changed,

[Xen-devel] [RFC PATCH v3 02/22] x86/boot: Reserve live update boot memory

2020-01-30 Thread David Woodhouse
From: David Woodhouse For live update to work, it will need a region of memory that can be given to the boot allocator while it parses the state information from the previous Xen and works out which of the other pages of memory it can consume. Reserve that like the crashdump region, and accept

[Xen-devel] [RFC PATCH v3 23/22] x86/smp: reset x2apic_enabled in smp_send_stop()

2020-01-31 Thread David Woodhouse
From: David Woodhouse Just before smp_send_stop() re-enables interrupts when shutting down for reboot or kexec, it calls __stop_this_cpu() which in turn calls disable_local_APIC(), which puts the APIC back in to the mode Xen found it in at boot. If that means turning x2APIC off and going back

[Xen-devel] [PATCH 2/8] x86/setup: Fix badpage= handling for memory above HYPERVISOR_VIRT_END

2020-01-31 Thread David Woodhouse
From: David Woodhouse Bad pages are identified by get_platform_badpages() and with badpage= on the command line. The boot allocator currently automatically elides these from the regions passed to it with init_boot_pages(). The xenheap is then initialised with the pages which are still marked as

[Xen-devel] [PATCH 0/8] Early cleanups and bug fixes in preparation for live update

2020-01-31 Thread David Woodhouse
came out of Wei's work on removing the directmap and is also needed for live update. Finally a little bit of preparation/cleanup of __setup_xen() to make way for what's to come, but which stands alone. David Woodhouse (7): x86/smp: reset x2apic_enabled in smp_send_stop()

[Xen-devel] [PATCH 7/8] x86/setup: simplify handling of initrdidx when no initrd present

2020-01-31 Thread David Woodhouse
From: David Woodhouse Remove a ternary operator that made my brain hurt. Replace it with something simpler that makes it somewhat clearer that the check for initrdidx < mbi->mods_count is because mbi->mods_count is what find_first_bit() will return when it doesn't find anything.

[Xen-devel] [PATCH 6/8] x86/setup: move vm_init() before end_boot_allocator()

2020-01-31 Thread David Woodhouse
From: David Woodhouse We would like to be able to use vmap() to map the live update data, and we need to do a first pass of the live update data before we prime the heap because we need to know which pages need to be preserved. The warning about ACPI code can be dropped, since that problem no

[Xen-devel] [PATCH 8/8] x86/setup: lift dom0 creation out into create_dom0() function

2020-01-31 Thread David Woodhouse
From: David Woodhouse The creation of dom0 can be relatively self-contained. Shift it into a separate function and simplify __start_xen() a little bit. This is a cleanup in its own right, but will be even more desireable when live update provides an alternative path through __start_xen() that

[Xen-devel] [PATCH 1/8] x86/smp: reset x2apic_enabled in smp_send_stop()

2020-01-31 Thread David Woodhouse
From: David Woodhouse Just before smp_send_stop() re-enables interrupts when shutting down for reboot or kexec, it calls __stop_this_cpu() which in turn calls disable_local_APIC(), which puts the APIC back in to the mode Xen found it in at boot. If that means turning x2APIC off and going back

[Xen-devel] [PATCH 4/8] xen/vmap: allow vm_init_type to be called during early_boot

2020-01-31 Thread David Woodhouse
From: Wei Liu We want to move vm_init, which calls vm_init_type under the hood, to early boot stage. Add a path to get page from boot allocator instead. Add an emacs block to that file while I was there. Signed-off-by: Wei Liu --- xen/common/vmap.c | 22 +++--- 1 file changed,

[Xen-devel] [PATCH 3/8] x86/setup: Don't skip 2MiB underneath relocated Xen image

2020-01-31 Thread David Woodhouse
From: David Woodhouse Set 'e' correctly to reflect the location that Xen is actually relocated to from its default 2MiB location. Not 2MiB below that. This is only vaguely a bug fix. The "missing" 2MiB would have been used in the end, and fed to the allocator. It's jus

[Xen-devel] [PATCH 5/8] xen/vmap: allow vmap() to be called during early boot

2020-01-31 Thread David Woodhouse
From: David Woodhouse Signed-off-by: David Woodhouse --- xen/common/vmap.c | 23 +-- 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/xen/common/vmap.c b/xen/common/vmap.c index 37922f735b..8343460794 100644 --- a/xen/common/vmap.c +++ b/xen/common/vmap.c

[Xen-devel] [PATCH] libxc: migrate migration stream definitions into Xen public headers

2020-01-31 Thread David Woodhouse
From: David Woodhouse These data structures will be used for live update, which is basically just live migration from one Xen to the next on the same machine via in-memory data structures, instead of across the network. Signed-off-by: David Woodhouse Well-excellent-carry-on-then-by: Ian

Re: [Xen-devel] [PATCH] libxc: migrate migration stream definitions into Xen public headers

2020-02-02 Thread David Woodhouse
On 1 February 2020 11:45:42 GMT, "Durrant, Paul" wrote: >> -Original Message- >> From: David Woodhouse >> Sent: 01 February 2020 00:40 >> To: xen-devel >> Cc: Ian Jackson ; Wei Liu ; >Andrew >> Cooper ; George Dunlap >> ;

Re: [Xen-devel] [PATCH] docs/designs: Add a design document for transparent live migration

2020-02-03 Thread David Woodhouse
On Mon, 2020-01-27 at 16:46 +, Ian Jackson wrote: > I don't like the term `transparent'. It is often abused in other > contexts. It can be clear to whom things are transparent. In a very > real sense existing migration is `transparent' to a domain's network > peers, for example. How about `

Re: [Xen-devel] [PATCH 6/8] x86/setup: move vm_init() before end_boot_allocator()

2020-02-03 Thread David Woodhouse
On Mon, 2020-02-03 at 11:10 +, Xia, Hongyan wrote: > Is there any problem to move vm_init() even earlier than this, like > right after init_frametable()? ACPI and NUMA functions need a couple of > things permanently mapped. You want it sooner than that, don't you? The code calls acpi_boot_tab

Re: [Xen-devel] [PATCH 8/8] x86/setup: lift dom0 creation out into create_dom0() function

2020-02-03 Thread David Woodhouse
On Mon, 2020-02-03 at 14:28 +, Julien Grall wrote: > The placement of printk shouldn't matter but the change feels a bit > out-of-context. Would you mind to explain it in the commit message? I didn't really intend to move the printk up; what I intended to do was move the setting of 'initrd' d

Re: [Xen-devel] [PATCH 5/8] xen/vmap: allow vmap() to be called during early boot

2020-02-03 Thread David Woodhouse
On Mon, 2020-02-03 at 14:00 +, Julien Grall wrote: > Hi David, > > On 01/02/2020 00:33, David Woodhouse wrote: > > From: David Woodhouse > > I am a bit concerned with this change, particularly the consequence this > have for the page-tables. There is an assumption t

Re: [Xen-devel] [PATCH 5/8] xen/vmap: allow vmap() to be called during early boot

2020-02-04 Thread David Woodhouse
On Tue, 2020-02-04 at 11:00 +, George Dunlap wrote: > On Mon, Feb 3, 2020 at 4:37 PM David Woodhouse wrote: > > > > On Mon, 2020-02-03 at 14:00 +, Julien Grall wrote: > > > Hi David, > > > > > > On 01/02/2020 00:33, David Woodhouse wrote: > &

<    1   2   3   4   5   6   7   8   9   10   >