[XEN PATCH v6 26/31] build: clean common temporary files from root makefile

2021-07-01 Thread Anthony PERARD
Signed-off-by: Anthony PERARD --- xen/Makefile | 1 + xen/scripts/Makefile.clean | 2 -- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/xen/Makefile b/xen/Makefile index fd002ecd52d0..6aac68750b4d 100644 --- a/xen/Makefile +++ b/xen/Makefile @@ -378,6 +378,7 @@ _clean

[XEN PATCH v6 29/31] build: build everything from the root dir, use obj=$subdir

2021-07-01 Thread Anthony PERARD
A subdirectory is now built by setting "$(obj)" instead of changing directory. "$(obj)" should always be set when using "Rules.mk" and thus a shortcut "$(build)" is introduced and should be used. A new variable "$(need-builtin)" is introduce. It is to be used whenever a "built_in.o" is wanted from

[XEN PATCH v6 22/31] build: fix $(TARGET).efi creation in arch/arm

2021-07-01 Thread Anthony PERARD
There is no need to try to guess a relative path to the "xen.efi" file, we can simply use $@. Also, there's no need to use `notdir`, make already do that work via $(@F). Signed-off-by: Anthony PERARD --- xen/arch/arm/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/xe

[XEN PATCH v6 30/31] build: introduce if_changed_deps

2021-07-01 Thread Anthony PERARD
This macro does compare command line like if_changed, but it also rewrite the dependencies generated by $(CC) in order to depend on a CONFIG_* as generated by kconfig instead of depending on autoconf.h. This allow to make a change in kconfig options and only rebuild the object that uses that CONFIG

[XEN PATCH v6 28/31] build: rework test/livepatch/Makefile

2021-07-01 Thread Anthony PERARD
This rework the livepatch/Makefile to make it less repetitive and make use of the facilities. All the targets to be built are now listed in $(extra-y) which will allow Rules.mk to build them without the need of a local target in a future patch. There are some changes/fixes in this patch: - when "x

[XEN PATCH v6 24/31] build: set ALL_OBJS to main Makefile; move prelink.o to main Makefile

2021-07-01 Thread Anthony PERARD
This is to avoid arch/$arch/Makefile having to recurse into parents directories. This avoid duplication of the logic to build prelink.o between arches. In order to do that, we cut the $(TARGET) target in the main Makefile in two, there is a "prepare" phase/target runned before starting to build "

[XEN PATCH v6 25/31] build: remove unneeded deps of x86_emulate.o

2021-07-01 Thread Anthony PERARD
Those two dependencies already exist so make doesn't need to know about them. The dependency will be generated by $(CC). Signed-off-by: Anthony PERARD --- xen/arch/x86/Makefile | 1 - 1 file changed, 1 deletion(-) diff --git a/xen/arch/x86/Makefile b/xen/arch/x86/Makefile index 3bd302d28ccf..fc

[XEN PATCH v6 31/31] build,riscv: tell the build system about riscv64/head.S

2021-07-01 Thread Anthony PERARD
This allows to `make arch/riscv/riscv64/head.o`. Example of rune on a fresh copy of the repository: make XEN_TARGET_ARCH=riscv64 CROSS_COMPILE=riscv64-linux-gnu- KBUILD_DEFCONFIG=tiny64_defconfig arch/riscv/riscv64/head.o Signed-off-by: Anthony PERARD --- xen/arch/riscv/riscv64/Makefile |

[XEN PATCH v6 23/31] build: fix arch/x86/node.o rule

2021-07-01 Thread Anthony PERARD
Avoid different spelling for the location of "xen-syms", and simply use the dependency variable. This avoid the assumption about $(TARGET) value. Signed-off-by: Anthony PERARD --- xen/arch/x86/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/xen/arch/x86/Makefile b/xe

[XEN PATCH v6 18/31] xen: move include/asm-* to include/arch-*/asm

2021-07-01 Thread Anthony PERARD
This avoid the need to create the symbolic link "include/asm". Signed-off-by: Anthony PERARD --- Other possible locations that I could think of: include/arch/*/asm arch/*/include/asm --- .gitignore| 5 ++- MAINTAINERS | 36 +

[XEN PATCH v6 27/31] build: prepare to always invoke $(MAKE) from xen/, use $(obj)

2021-07-01 Thread Anthony PERARD
In a future patch, when building a subdirectory, we will set "obj=$subdir" rather than change directory. Before that, we add "$(obj)" and "$(src)" in as many places as possible where we will need to know which subdirectory is been built. "$(obj)" is for files been generated during the build, and "

[XEN PATCH v6 21/31] build: set XEN_BUILD_EFI earlier

2021-07-01 Thread Anthony PERARD
We are going to need the variable XEN_BUILD_EFI earlier. This early check is using "try-run" to allow to have a temporary output file in case it is needed for $(CC) to build the *.c file. The "efi/check.o" file is still needed in "arch/x86/Makefile" so the check is currently duplicated. This pat

[XEN PATCH v6 19/31] build: rework .banner generation

2021-07-01 Thread Anthony PERARD
Avoid depending on Makefile but still allow to rebuild the banner when $(XEN_FULLVERSION) changes. Also add a dependency on tools/xen.flf, even if not expected to change. Signed-off-by: Anthony PERARD --- xen/Makefile | 20 +--- 1 file changed, 13 insertions(+), 7 deletions(-)

Re: [PATCH v20210701 03/40] xl: fix description of migrate --debug

2021-07-01 Thread Anthony PERARD
On Thu, Jul 01, 2021 at 11:55:58AM +0200, Olaf Hering wrote: > xl migrate --debug used to track every pfn in every batch of pages. > But these times are gone. The code in xc_domain_save is the consumer > of this knob, but it considers it only for the remus and colo case. > > Adjust the help text t

Re: [PATCH v20210701 03/40] xl: fix description of migrate --debug

2021-07-01 Thread Andrew Cooper
On 01/07/2021 10:55, Olaf Hering wrote: > xl migrate --debug used to track every pfn in every batch of pages. > But these times are gone. The code in xc_domain_save is the consumer > of this knob, but it considers it only for the remus and colo case. > > Adjust the help text to tell what --debug do

Re: [PATCH v20210701 03/40] xl: fix description of migrate --debug

2021-07-01 Thread Olaf Hering
Am Thu, 1 Jul 2021 15:33:47 +0100 schrieb Andrew Cooper : > Since when?  It was absolutely critical to debugging issues during the > development of migration v2. Well, I can find out if needed. What could would it enable today? Last time I looked there was none. Olaf pgpDXPbP6IGpy.pgp Descript

Re: [PATCH v20210701 03/40] xl: fix description of migrate --debug

2021-07-01 Thread Olaf Hering
Am Thu, 1 Jul 2021 16:40:55 +0200 schrieb Olaf Hering : > could "code", sorry. Olaf pgplVbaHz40oE.pgp Description: Digitale Signatur von OpenPGP

Re: [PATCH v20210701 03/40] xl: fix description of migrate --debug

2021-07-01 Thread Andrew Cooper
On 01/07/2021 15:41, Olaf Hering wrote: > Am Thu, 1 Jul 2021 16:40:55 +0200 > schrieb Olaf Hering : > >> could > "code", sorry. c/s 7449fb36c6c81d2ba10a40b59e61a9f420cd8450 was buggy and inverted the condition while making the code transformation. --debug makes no sense at all in a checkpointed s

Re: XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

2021-07-01 Thread Olaf Hering
Am Thu, 1 Jul 2021 12:19:24 +0100 schrieb Andrew Cooper : > Where is the 51 leaves coming in? I suspect the bug is this line, and the magic "0x1c" value: p->extd.max_leaf = min(p->extd.max_leaf, 0x1cu); In my testing basic remains at 13, while extd is truncated from 0x8008 to 28. I think

Re: [PATCH v20210701 03/40] xl: fix description of migrate --debug

2021-07-01 Thread Olaf Hering
Am Thu, 1 Jul 2021 15:49:08 +0100 schrieb Andrew Cooper : > --debug makes no sense at all in a checkpointed stream. This should probably have become "ctx->save.checkpointed == MIG_STREAM_NONE". But this still leaves the question what value this code branch has when verify_frames does not, and m

Re: XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

2021-07-01 Thread Andrew Cooper
On 01/07/2021 16:00, Olaf Hering wrote: > Am Thu, 1 Jul 2021 12:19:24 +0100 > schrieb Andrew Cooper : > >> Where is the 51 leaves coming in? > I suspect the bug is this line, and the magic "0x1c" value: > > p->extd.max_leaf = min(p->extd.max_leaf, 0x1cu); > > In my testing basic remains at 13, whi

Re: XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

2021-07-01 Thread Olaf Hering
Am Thu, 1 Jul 2021 16:13:21 +0100 schrieb Andrew Cooper : > That logic is definitely incorrect, but using a different constant isn't > right. Yeah, that occurred to me after sending this email. I think in the end the values should probably get closer to 51, not smaller than the current one. Ol

Re: XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

2021-07-01 Thread Andrew Cooper
On 01/07/2021 16:18, Olaf Hering wrote: > Am Thu, 1 Jul 2021 16:13:21 +0100 > schrieb Andrew Cooper : > >> That logic is definitely incorrect, but using a different constant isn't >> right. > Yeah, that occurred to me after sending this email. > I think in the end the values should probably get clo

[xtf test] 163212: all pass - PUSHED

2021-07-01 Thread osstest service owner
flight 163212 xtf real [real] http://logs.test-lab.xenproject.org/osstest/logs/163212/ Perfect :-) All tests in this flight passed as required version targeted for testing: xtf cdef9568c1b1f8831f3b8f644073e635d47f0ec4 baseline version: xtf 93b29b886e8665e368598c

[XEN PATCH] MAINTAINERS: Updating after change to tools/include/

2021-07-01 Thread Anthony PERARD
The LIBS section doesn't mention the headers associated with the libraries, same for LIBXENLIGHT section. They aren't any ':' in other section names, so remove it. Fixes: 4664034cdc72 ("tools/libs: move official headers to common directory") Fixes: f7079d7ef69f ("MAINTAINERS: add myself as tools/

Re: XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

2021-07-01 Thread Olaf Hering
Am Thu, 1 Jul 2021 16:25:10 +0100 schrieb Andrew Cooper : > -    p->extd.max_leaf = min(p->extd.max_leaf, 0x1cu); > +    p->extd.max_leaf = 0x8000 | min(p->extd.max_leaf & 0x, > 0x1cu); Yes, this fixes it for me. Thanks. Olaf pgpaVhBDW9H21.pgp Description: Digitale Signatur von

Re: XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

2021-07-01 Thread Andrew Cooper
On 01/07/2021 17:23, Olaf Hering wrote: > Am Thu, 1 Jul 2021 16:25:10 +0100 > schrieb Andrew Cooper : > >> -    p->extd.max_leaf = min(p->extd.max_leaf, 0x1cu); >> +    p->extd.max_leaf = 0x8000 | min(p->extd.max_leaf & 0x, >> 0x1cu); > Yes, this fixes it for me. Thanks. Fantastic.

Re: XEN_SYSCTL_cpu_policy_hvm_default truncates leaf count

2021-07-01 Thread Olaf Hering
Am Thu, 1 Jul 2021 17:28:25 +0100 schrieb Andrew Cooper : > 0x801c Dues this need to be tagged as unsigned with a trailing U? Olaf pgpUDLCgMImch.pgp Description: Digitale Signatur von OpenPGP

Re: [PATCH v20210701 33/40] tools: change struct precopy_stats to precopy_stats_t

2021-07-01 Thread Anthony PERARD
On Thu, Jul 01, 2021 at 11:56:28AM +0200, Olaf Hering wrote: > This will help libxl_save_msgs_gen.pl to copy the struct as a region of > memory. > > No change in behavior intented. > > Signed-off-by: Olaf Hering > --- > tools/include/xensaverestore.h | 7 +++ > tools/libs/saverestore/comm

Re: [PATCH v20210701 33/40] tools: change struct precopy_stats to precopy_stats_t

2021-07-01 Thread Olaf Hering
Am Thu, 1 Jul 2021 17:45:12 +0100 schrieb Anthony PERARD : > But can't libxl_save_msgs_gen.pl been able to deal with > thing like 'struct precopy_stats' ? It seems to be able to deal with > 'unsigned long'. Yes, this is apparently possible. I have to check why I thought it was required to turn th

Re: [XEN PATCH v6 18/31] xen: move include/asm-* to include/arch-*/asm

2021-07-01 Thread Paul Durrant
On 01/07/2021 15:09, Anthony PERARD wrote: This avoid the need to create the symbolic link "include/asm". Signed-off-by: Anthony PERARD Acked-by: Paul Durrant

Re: [PATCH v6 8/9] docs: hypercalls sphinx skeleton for generated html

2021-07-01 Thread Stefano Stabellini
On Thu, 1 Jul 2021, Luca Fancellu wrote: > > On 24 Jun 2021, at 00:34, Stefano Stabellini wrote: > > > > On Mon, 10 May 2021, Luca Fancellu wrote: > >> Create a skeleton for the documentation about hypercalls > >> > >> Signed-off-by: Luca Fancellu > >> --- > >> v6 changes: > >> - Now every plat

Re: [XEN PATCH v6 18/31] xen: move include/asm-* to include/arch-*/asm

2021-07-01 Thread Bob Eshleman
Hey Anthony, > diff --git a/MAINTAINERS b/MAINTAINERS > index 8a52a03969fe..03a5553116a8 100644 > --- a/MAINTAINERS > +++ b/MAINTAINERS > -F: xen/include/asm-arm/ > +F: xen/include/arch-arm/ ... snip ... > -F: xen/include/asm-x86/ > +F: xen/include/arch-x86/ It looks like riscv will al

[ovmf test] 163210: regressions - FAIL

2021-07-01 Thread osstest service owner
flight 163210 ovmf real [real] http://logs.test-lab.xenproject.org/osstest/logs/163210/ Regressions :-( Tests which did not succeed and are blocking, including tests which could not be run: test-amd64-i386-xl-qemuu-ovmf-amd64 12 debian-hvm-install fail REGR. vs. 162359 test-amd64-amd64-xl-qemuu

Re: [PATCH v6 6/9] docs: add doxygen preprocessor and related files

2021-07-01 Thread Stefano Stabellini
On Thu, 1 Jul 2021, Luca Fancellu wrote: > > On 23 Jun 2021, at 23:03, Stefano Stabellini wrote: > > > > On Mon, 10 May 2021, Luca Fancellu wrote: > >> Add preprocessor called by doxygen before parsing headers, > >> it will include in every header a doxygen_include.h file > >> that provides missi

Re: [PATCH v6 7/9] docs: Change Makefile and sphinx configuration for doxygen

2021-07-01 Thread Stefano Stabellini
On Thu, 1 Jul 2021, Luca Fancellu wrote: > > On 24 Jun 2021, at 00:33, Stefano Stabellini wrote: > > > > On Mon, 10 May 2021, Luca Fancellu wrote: > >> Modify docs/Makefile to call doxygen and generate sphinx > >> html documentation given the doxygen XML output. > >> > >> Modify docs/conf.py sph

Re: [PATCH v6 9/9] docs/doxygen: doxygen documentation for grant_table.h

2021-07-01 Thread Stefano Stabellini
On Thu, 1 Jul 2021, Luca Fancellu wrote: > Hi Stefano, > > > On 24 Jun 2021, at 00:34, Stefano Stabellini wrote: > > > > On Mon, 10 May 2021, Luca Fancellu wrote: > >> Modification to include/public/grant_table.h: > >> > >> 1) Add doxygen tags to: > >> - Create Grant tables section > >> - inclu

Re: [XEN PATCH v6 29/31] build: build everything from the root dir, use obj=$subdir

2021-07-01 Thread Bob Eshleman
On 7/1/21 7:10 AM, Anthony PERARD wrote: > A subdirectory is now built by setting "$(obj)" instead of changing > directory. "$(obj)" should always be set when using "Rules.mk" and > thus a shortcut "$(build)" is introduced and should be used. > > A new variable "$(need-builtin)" is introduce. It i

Re: [XEN PATCH v6 31/31] build,riscv: tell the build system about riscv64/head.S

2021-07-01 Thread Bob Eshleman
On 7/1/21 7:10 AM, Anthony PERARD wrote: > This allows to `make arch/riscv/riscv64/head.o`. > > Example of rune on a fresh copy of the repository: > make XEN_TARGET_ARCH=riscv64 CROSS_COMPILE=riscv64-linux-gnu- > KBUILD_DEFCONFIG=tiny64_defconfig arch/riscv/riscv64/head.o > > Signed-off-by:

[xen-unstable test] 163206: tolerable FAIL

2021-07-01 Thread osstest service owner
flight 163206 xen-unstable real [real] http://logs.test-lab.xenproject.org/osstest/logs/163206/ Failures :-/ but no regressions. Tests which are failing intermittently (not blocking): test-amd64-amd64-xl-qemut-debianhvm-i386-xsm 12 debian-hvm-install fail in 163190 pass in 163206 test-armhf-ar

[linux-linus test] 163207: regressions - FAIL

2021-07-01 Thread osstest service owner
flight 163207 linux-linus real [real] http://logs.test-lab.xenproject.org/osstest/logs/163207/ Regressions :-( Tests which did not succeed and are blocking, including tests which could not be run: test-amd64-i386-qemut-rhel6hvm-intel 7 xen-install fail REGR. vs. 152332 test-amd64-i386-xl-

[ovmf test] 163214: regressions - FAIL

2021-07-01 Thread osstest service owner
flight 163214 ovmf real [real] http://logs.test-lab.xenproject.org/osstest/logs/163214/ Regressions :-( Tests which did not succeed and are blocking, including tests which could not be run: test-amd64-i386-xl-qemuu-ovmf-amd64 12 debian-hvm-install fail REGR. vs. 162359 test-amd64-amd64-xl-qemuu

[ovmf test] 163216: regressions - FAIL

2021-07-01 Thread osstest service owner
flight 163216 ovmf real [real] http://logs.test-lab.xenproject.org/osstest/logs/163216/ Regressions :-( Tests which did not succeed and are blocking, including tests which could not be run: test-amd64-i386-xl-qemuu-ovmf-amd64 12 debian-hvm-install fail REGR. vs. 162359 test-amd64-amd64-xl-qemuu

[qemu-mainline test] 163213: regressions - FAIL

2021-07-01 Thread osstest service owner
flight 163213 qemu-mainline real [real] flight 163218 qemu-mainline real-retest [real] http://logs.test-lab.xenproject.org/osstest/logs/163213/ http://logs.test-lab.xenproject.org/osstest/logs/163218/ Regressions :-( Tests which did not succeed and are blocking, including tests which could not be

[ovmf test] 163217: regressions - FAIL

2021-07-01 Thread osstest service owner
flight 163217 ovmf real [real] http://logs.test-lab.xenproject.org/osstest/logs/163217/ Regressions :-( Tests which did not succeed and are blocking, including tests which could not be run: test-amd64-i386-xl-qemuu-ovmf-amd64 12 debian-hvm-install fail REGR. vs. 162359 test-amd64-amd64-xl-qemuu

Re: [PATCH v15 12/12] of: Add plumbing for restricted DMA pool

2021-07-01 Thread Guenter Roeck
pecific files for 20210701 # good: [62fb9874f5da54fdb243003b386128037319b219] Linux 5.13 git bisect start 'HEAD' 'v5.13' # bad: [f63c4fda987a19b1194cc45cb72fd5bf968d9d90] Merge remote-tracking branch 'rdma/for-next' git bisect bad f63c4fda987a19b1194cc45cb72fd5bf968d9d90 #

[linux-linus test] 163215: regressions - FAIL

2021-07-01 Thread osstest service owner
flight 163215 linux-linus real [real] http://logs.test-lab.xenproject.org/osstest/logs/163215/ Regressions :-( Tests which did not succeed and are blocking, including tests which could not be run: test-amd64-i386-qemut-rhel6hvm-intel 7 xen-install fail REGR. vs. 152332 test-amd64-i386-xl-

Re: [XEN PATCH v6 31/31] build,riscv: tell the build system about riscv64/head.S

2021-07-01 Thread Alistair Francis
On Fri, Jul 2, 2021 at 12:22 AM Anthony PERARD wrote: > > This allows to `make arch/riscv/riscv64/head.o`. > > Example of rune on a fresh copy of the repository: > make XEN_TARGET_ARCH=riscv64 CROSS_COMPILE=riscv64-linux-gnu- > KBUILD_DEFCONFIG=tiny64_defconfig arch/riscv/riscv64/head.o > > S

Re: [PATCH v15 06/12] swiotlb: Use is_swiotlb_force_bounce for swiotlb data bouncing

2021-07-01 Thread Will Deacon
On Wed, Jun 30, 2021 at 08:56:51AM -0700, Nathan Chancellor wrote: > On Wed, Jun 30, 2021 at 12:43:48PM +0100, Will Deacon wrote: > > On Wed, Jun 30, 2021 at 05:17:27PM +0800, Claire Chang wrote: > > > `BUG: unable to handle page fault for address: 003a8290` and > > > the fact it crashed at

Re: [PATCH v3 10/10] arm64: Change type of hsr, cpsr, spsr_el1 to uint64_t

2021-07-01 Thread Michal Orzel
Hi Julien, On 07.06.2021 15:31, Julien Grall wrote: > Hi, > > On 05/05/2021 08:43, Michal Orzel wrote: >> AArch64 registers are 64bit whereas AArch32 registers >> are 32bit or 64bit. MSR/MRS are expecting 64bit values thus >> we should get rid of helpers READ/WRITE_SYSREG32 >> in favour of using

Re: [PATCH 5/9] xen: introduce assign_pages_nr

2021-07-01 Thread Jan Beulich
On 30.06.2021 20:29, Julien Grall wrote: > On 10/06/2021 10:49, Jan Beulich wrote: >> On 07.06.2021 04:43, Penny Zheng wrote: >>> Introduce new interface assign_pages_nr to deal with when page number is >>> not in a power-of-two, which will save the trouble each time user needs >>> to split the siz

[ovmf test] 163205: regressions - FAIL

2021-07-01 Thread osstest service owner
flight 163205 ovmf real [real] http://logs.test-lab.xenproject.org/osstest/logs/163205/ Regressions :-( Tests which did not succeed and are blocking, including tests which could not be run: test-amd64-i386-xl-qemuu-ovmf-amd64 12 debian-hvm-install fail REGR. vs. 162359 test-amd64-amd64-xl-qemuu

[libvirt test] 163208: regressions - FAIL

2021-07-01 Thread osstest service owner
flight 163208 libvirt real [real] http://logs.test-lab.xenproject.org/osstest/logs/163208/ Regressions :-( Tests which did not succeed and are blocking, including tests which could not be run: build-amd64-libvirt 6 libvirt-buildfail REGR. vs. 151777 build-armhf-libvirt

Re: [PATCH 5/9] xen: introduce assign_pages_nr

2021-07-01 Thread Julien Grall
Hi Jan, On 01/07/2021 09:26, Jan Beulich wrote: On 30.06.2021 20:29, Julien Grall wrote: On 10/06/2021 10:49, Jan Beulich wrote: On 07.06.2021 04:43, Penny Zheng wrote: Introduce new interface assign_pages_nr to deal with when page number is not in a power-of-two, which will save the trouble

Re: [PATCH] libxl/x86: check return value of SHADOW_OP_SET_ALLOCATION domctl

2021-07-01 Thread Anthony PERARD
On Mon, Jun 28, 2021 at 01:47:03PM +0200, Jan Beulich wrote: > The hypervisor may not have enough memory to satisfy the request. > > Requested-by: Andrew Cooper > Signed-off-by: Jan Beulich > --- > Especially if the request was mostly fulfilled, guests may have done > fine despite the failure, s

JULY COMMUNITY CALL POSTPONED AND RESCHEDULED

2021-07-01 Thread George Dunlap
Sorry everyone for not including xen-devel in this announcement earlier. THERE WILL BE NO COMMUNITY CALL TODAY. We will have the call next week, at a time to be announced later today. We are in the process of finding a new time (a survey was already sent out to people on the community call sig

[PATCH v20210701 00/40] leftover from 2020

2021-07-01 Thread Olaf Hering
Various unreviewed changes, rebase to f95b7b37cf. Olaf Hering (40): hotplug/Linux: fix starting of xenstored with restarting systemd tools: add API to work with sevaral bits at once xl: fix description of migrate --debug tools: use integer division in convert-legacy-stream tools: handle

[PATCH v20210701 02/40] tools: add API to work with sevaral bits at once

2021-07-01 Thread Olaf Hering
Introduce new API to test if a fixed number of bits is clear or set, and clear or set them all at once. The caller has to make sure the input bitnumber is a multiple of BITS_PER_LONG. This API avoids the loop over each bit in a known range just to see if all of them are either clear or set. Sign

[PATCH v20210701 09/40] tools: add readv_exact to libxenctrl

2021-07-01 Thread Olaf Hering
Read a batch of iovec's. Short reads are the common case, finish the trailing iov with read_exact. Signed-off-by: Olaf Hering v2: - add comment to short-read handling --- tools/libs/ctrl/xc_private.c | 57 +++- tools/libs/ctrl/xc_private.h | 1 + 2 files change

[PATCH v20210701 01/40] hotplug/Linux: fix starting of xenstored with restarting systemd

2021-07-01 Thread Olaf Hering
A hard to trigger race with another unrelated systemd service and xenstored.service unveiled a bug in the way how xenstored is launched with systemd. launch-xenstore may start either a daemon or a domain. In case a domain is used, systemd-notify was called. If another service triggered a restart o

[PATCH v20210701 04/40] tools: use integer division in convert-legacy-stream

2021-07-01 Thread Olaf Hering
A single slash gives a float, a double slash gives an int. bitmap = unpack_exact("Q" * ((max_id/64) + 1)) TypeError: can't multiply sequence by non-int of type 'float' Signed-off-by: Olaf Hering v02: - import division to remain compatible with python2.7 (andrew) - white space in max_id chun

[PATCH v20210701 06/40] tools: fix Python3.4 TypeError in format string

2021-07-01 Thread Olaf Hering
Using the first element of a tuple for a format specifier fails with python3.4 as included in SLE12: b = b"string/%x" % (i, ) TypeError: unsupported operand type(s) for %: 'bytes' and 'tuple' It happens to work with python 2.7 and 3.6. Use a syntax that is handled by all three variants. Signe

[PATCH v20210701 03/40] xl: fix description of migrate --debug

2021-07-01 Thread Olaf Hering
xl migrate --debug used to track every pfn in every batch of pages. But these times are gone. The code in xc_domain_save is the consumer of this knob, but it considers it only for the remus and colo case. Adjust the help text to tell what --debug does today: Nothing. Signed-off-by: Olaf Hering R

[PATCH v20210701 18/40] tools: save: move errors array

2021-07-01 Thread Olaf Hering
Remove allocation from hotpath, move errors array into preallocated space. Signed-off-by: Olaf Hering Reviewed-by: Juergen Gross --- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/save.c | 7 ++- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/tools/libs/sav

[PATCH v20210701 11/40] tools: use sr_is_known_page_type

2021-07-01 Thread Olaf Hering
Verify pfn type on sending side, also verify incoming batch of pfns. Signed-off-by: Olaf Hering Reviewed-by: Juergen Gross v02: - use sr_is_known_page_type instead of xc_is_known_page_type --- tools/libs/saverestore/restore.c | 3 +-- tools/libs/saverestore/save.c| 6 ++ 2 files change

[PATCH v20210701 08/40] MAINTAINERS: add myself as saverestore maintainer

2021-07-01 Thread Olaf Hering
I touched it last. Signed-off-by: Olaf Hering --- MAINTAINERS | 6 ++ 1 file changed, 6 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 8a52a03969..36dc634958 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -381,6 +381,12 @@ R: Juergen Gross S: Supported F: tools/libs/

[PATCH v20210701 10/40] tools: add xc_is_known_page_type to libxenctrl

2021-07-01 Thread Olaf Hering
Users of xc_get_pfn_type_batch may want to sanity check the data returned by Xen. Add a simple helper for this purpose. Signed-off-by: Olaf Hering v02: - rename xc_is_known_page_type to sr_is_known_page_type - move from ctrl/xc_private.h to saverestore/common.h (jgross) --- tools/libs/saveresto

[PATCH v20210701 15/40] tools: prepare to allocate saverestore arrays once

2021-07-01 Thread Olaf Hering
The hotpath 'send_dirty_pages' is supposed to do just one thing: sending. The other end 'handle_page_data' is supposed to do just receiving. But instead both do other costly work like memory allocations and data moving. Do the allocations once, the array sizes are a compiletime constant. Avoid unn

[PATCH v20210701 05/40] tools: handle libxl__physmap_info.name properly in convert-legacy-stream

2021-07-01 Thread Olaf Hering
The trailing member name[] in libxl__physmap_info is written as a cstring into the stream. The current code does a sanity check if the last byte is zero. This attempt fails with python3.4 because name[-1] returns a type int. As a result the comparison with byte(\00) fails: File "/usr/lib/xen/bin

[PATCH v20210701 21/40] tools: save: move guest_data array

2021-07-01 Thread Olaf Hering
Remove allocation from hotpath, move guest_data array into preallocated space. Because this was allocated with calloc: Adjust the loop to clear unused entries as needed. Signed-off-by: Olaf Hering Reviewed-by: Juergen Gross --- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/sa

[PATCH v20210701 26/40] tools: restore: move mfns array in populate_pfns

2021-07-01 Thread Olaf Hering
Remove allocation from hotpath, move populate_pfns mfns array into preallocated space. Use some prefix to avoid conflict with an array used in handle_page_data. Signed-off-by: Olaf Hering Reviewed-by: Juergen Gross --- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/restore.c |

[PATCH v20210701 12/40] tools: unify type checking for data pfns in migration stream

2021-07-01 Thread Olaf Hering
Introduce a helper which decides if a given pfn in the migration stream is backed by memory. This specifically deals with type XEN_DOMCTL_PFINFO_XALLOC, which was a synthetic toolstack-only type used in Xen 4.2 to 4.5. It indicated a dirty page on the sending side for which no data will be send in

[PATCH v20210701 14/40] tools: show migration transfer rate in send_dirty_pages

2021-07-01 Thread Olaf Hering
Show how fast domU pages are transferred in each iteration. The relevant data is how fast the pfns travel, not so much how much protocol overhead exists. So the reported MiB/sec is just for pfns. Signed-off-by: Olaf Hering v02: - rearrange MiB_sec calculation (jgross) --- tools/libs/saverestor

[PATCH v20210701 22/40] tools: save: move local_pages array

2021-07-01 Thread Olaf Hering
Remove allocation from hotpath, move local_pages array into preallocated space. Adjust the code to use the src page as is in case of HVM. In case of PV the page may need to be normalised, use a private memory area for this purpose. Signed-off-by: Olaf Hering Reviewed-by: Juergen Gross --- tool

[PATCH v20210701 13/40] tools: unify type checking for data pfns in migration stream

2021-07-01 Thread Olaf Hering
Introduce a helper which decides if a given pfn type has data in the migration stream. No change in behavior intended. Signed-off-by: Olaf Hering --- tools/libs/saverestore/common.h | 18 ++ tools/libs/saverestore/restore.c | 29 +++-- tools/libs/saveres

[PATCH v20210701 31/40] tools: recognize LIBXL_API_VERSION for 4.16

2021-07-01 Thread Olaf Hering
This is required by upcoming API changes. Signed-off-by: Olaf Hering --- tools/include/libxl.h | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/tools/include/libxl.h b/tools/include/libxl.h index ae7fe27c1f..29931626a2 100644 --- a/tools/include/libxl.h +++ b/tools/include/l

[PATCH v20210701 39/40] tools: use xg_sr_bitmap for populated_pfns

2021-07-01 Thread Olaf Hering
Signed-off-by: Olaf Hering v02: - remove xg_ prefix from called functions --- tools/libs/saverestore/common.h | 21 +++- tools/libs/saverestore/restore.c | 69 tools/libs/saverestore/restore_x86_hvm.c | 9 tools/libs/saverestore/restore_x86_pv.

[PATCH v20210701 24/40] tools: restore: move mfns array

2021-07-01 Thread Olaf Hering
Remove allocation from hotpath, move mfns array into preallocated space. Signed-off-by: Olaf Hering Reviewed-by: Juergen Gross --- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/restore.c | 5 ++--- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/tools/libs/saver

[PATCH v20210701 17/40] tools: save: move types array

2021-07-01 Thread Olaf Hering
Remove allocation from hotpath, move types array into preallocated space. Signed-off-by: Olaf Hering Reviewed-by: Juergen Gross --- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/save.c | 7 ++- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/tools/libs/save

[PATCH v20210701 36/40] tools: add --min_remaining to libxl_domain_suspend

2021-07-01 Thread Olaf Hering
The decision to stop+move a domU to the new host must be based on two factors: - the available network bandwidth for the migration stream - the maximum time a workload within a domU can be savely suspended Both values define how many dirty pages a workload may produce prior the final stop+move. T

[PATCH v20210701 37/40] tools: add --abort_if_busy to libxl_domain_suspend

2021-07-01 Thread Olaf Hering
Provide a knob to the host admin to abort the live migration of a running domU if the downtime during final transit will be too long for the workload within domU. Adjust error reporting. Add ERROR_MIGRATION_ABORTED to allow callers of libxl_domain_suspend to distinguish between errors and the requ

[PATCH v20210701 35/40] tools: add --max_iters to libxl_domain_suspend

2021-07-01 Thread Olaf Hering
Migrating a large, and potentially busy, domU will take more time than neccessary due to excessive number of copying iterations. Allow to host admin to control the number of iterations which copy cumulated domU dirty pages to the target host. The default remains 5, which means one initial iterati

[PATCH v20210701 40/40] tools/libxc: use superpages during restore of HVM guest

2021-07-01 Thread Olaf Hering
During creating of a HVM domU meminit_hvm() tries to map superpages. After save/restore or migration this mapping is lost, everything is allocated in single pages. This causes a performance degradation after migration. Add neccessary code to preallocate a superpage for an incoming chunk of pfns. I

[PATCH v20210701 20/40] tools: save: move rec_pfns array

2021-07-01 Thread Olaf Hering
Remove allocation from hotpath, move rec_pfns array into preallocated space. Signed-off-by: Olaf Hering Reviewed-by: Juergen Gross --- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/save.c | 11 +-- 2 files changed, 3 insertions(+), 10 deletions(-) diff --git a/tools

[PATCH v20210701 34/40] tools: add callback to libxl for precopy_policy and precopy_stats_t

2021-07-01 Thread Olaf Hering
This duplicates simple_precopy_policy. To recap its purpose: - do up to 5 iterations of copying dirty domU memory to target, including the initial copying of all domU memory, excluding the final copying while the domU is suspended - do fewer iterations in case the domU dirtied less than 50 page

[PATCH v20210701 33/40] tools: change struct precopy_stats to precopy_stats_t

2021-07-01 Thread Olaf Hering
This will help libxl_save_msgs_gen.pl to copy the struct as a region of memory. No change in behavior intented. Signed-off-by: Olaf Hering --- tools/include/xensaverestore.h | 7 +++ tools/libs/saverestore/common.h | 2 +- tools/libs/saverestore/save.c | 6 +++--- 3 files changed, 7 inse

[PATCH v20210701 30/40] tools: restore: write data directly into guest

2021-07-01 Thread Olaf Hering
Read incoming migration stream directly into the guest memory. This avoids the memory allocation and copying, and the resulting performance penalty. Signed-off-by: Olaf Hering --- tools/libs/saverestore/common.h | 1 + tools/libs/saverestore/restore.c | 132 ++- 2

[PATCH v20210701 07/40] tools: create libxensaverestore

2021-07-01 Thread Olaf Hering
Move all save/restore related code from libxenguest.so into a separate library libxensaverestore.so. The only consumer is libxl-save-helper. There is no need to have the moved code mapped all the time in binaries where libxenguest.so is used. According to size(1) the change is: textdata

[PATCH v20210701 16/40] tools: save: move mfns array

2021-07-01 Thread Olaf Hering
Remove allocation from hotpath, move mfns array into preallocated space. Signed-off-by: Olaf Hering Reviewed-by: Juergen Gross --- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/save.c | 7 ++- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/tools/libs/saver

[PATCH v20210701 28/40] tools: restore: split record processing

2021-07-01 Thread Olaf Hering
handle_page_data must be able to read directly into mapped guest memory. This will avoid unneccesary memcpy calls for data which can be consumed verbatim. Rearrange the code to allow decisions based on the incoming record. This change is preparation for future changes in handle_page_data, no cha

[PATCH v20210701 27/40] tools: restore: move pfns array in populate_pfns

2021-07-01 Thread Olaf Hering
Remove allocation from hotpath, move populate_pfns' pfns array into preallocated space. Use some prefix to avoid conflict with an array used in handle_page_data. Signed-off-by: Olaf Hering Reviewed-by: Juergen Gross --- tools/libs/saverestore/common.h | 1 + tools/libs/saverestore/restore.c

[PATCH v20210701 23/40] tools: restore: move types array

2021-07-01 Thread Olaf Hering
Remove allocation from hotpath, move types array into preallocated space. Signed-off-by: Olaf Hering Reviewed-by: Juergen Gross --- tools/libs/saverestore/common.h | 1 + tools/libs/saverestore/restore.c | 12 +--- 2 files changed, 2 insertions(+), 11 deletions(-) diff --git a/tools/

[PATCH v20210701 38/40] tools: add API for expandable bitmaps

2021-07-01 Thread Olaf Hering
Since the incoming migration stream lacks info about what the highest pfn will be, some data structures can not be allocated upfront. Add an API for expandable bitmaps, loosely based on pfn_set_populated. Signed-off-by: Olaf Hering --- tools/libs/saverestore/common.c | 39 +++ t

[PATCH v20210701 19/40] tools: save: move iov array

2021-07-01 Thread Olaf Hering
Remove allocation from hotpath, move iov array into preallocated space. Signed-off-by: Olaf Hering Reviewed-by: Juergen Gross --- tools/libs/saverestore/common.h | 2 ++ tools/libs/saverestore/save.c | 7 ++- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/tools/libs/savere

[PATCH v20210701 25/40] tools: restore: move map_errs array

2021-07-01 Thread Olaf Hering
Remove allocation from hotpath, move map_errs array into preallocated space. Signed-off-by: Olaf Hering Reviewed-by: Juergen Gross --- tools/libs/saverestore/common.h | 1 + tools/libs/saverestore/restore.c | 12 +--- 2 files changed, 2 insertions(+), 11 deletions(-) diff --git a/too

[PATCH v20210701 29/40] tools: restore: split handle_page_data

2021-07-01 Thread Olaf Hering
handle_page_data must be able to read directly into mapped guest memory. This will avoid unneccesary memcpy calls for data that can be consumed verbatim. Split the various steps of record processing: - move processing to handle_buffered_page_data - adjust xenforeignmemory_map to set errno in case

[PATCH v20210701 32/40] tools: adjust libxl_domain_suspend to receive a struct props

2021-07-01 Thread Olaf Hering
Upcoming changes will pass more knobs down to xc_domain_save. Adjust the libxl_domain_suspend API to allow easy adding of additional knobs. No change in behavior intented. Signed-off-by: Olaf Hering Acked-by: Christian Lindig --- tools/include/libxl.h| 26 ++

Re: [PATCH 5/9] xen: introduce assign_pages_nr

2021-07-01 Thread Jan Beulich
On 01.07.2021 11:24, Julien Grall wrote: > On 01/07/2021 09:26, Jan Beulich wrote: >> On 30.06.2021 20:29, Julien Grall wrote: >>> On 10/06/2021 10:49, Jan Beulich wrote: On 07.06.2021 04:43, Penny Zheng wrote: > Introduce new interface assign_pages_nr to deal with when page number is

[PATCH v1] tools: fix comment typo in libxl__cpuid_legacy

2021-07-01 Thread Olaf Hering
Replace emualted with emulated. Signed-off-by: Olaf Hering --- tools/libs/light/libxl_cpuid.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/libs/light/libxl_cpuid.c b/tools/libs/light/libxl_cpuid.c index eb6feaa96d..b2c673841a 100644 --- a/tools/libs/light/libxl_cpuid

Re: [PATCH v1] tools: fix comment typo in libxl__cpuid_legacy

2021-07-01 Thread Andrew Cooper
On 01/07/2021 11:30, Olaf Hering wrote: > Replace emualted with emulated. > > Signed-off-by: Olaf Hering Acked-by: Andrew Cooper

  1   2   >