Re: [Xen-devel] Live migrate with Linux >= 4.13 domU causes kernel time jumps and TCP connection stalls.

[Hans van Kranenburg]

Hi,

On 12/21/18 6:54 PM, Hans van Kranenburg wrote:
> Hi,
> 
> We've been tracking down a live migration bug during the last three days
> here at work, and here's what we found so far.
> 
> 1. Xen version and dom0 linux kernel version don't matter.
> 2. DomU kernel is >= Linux 4.13.
> 
> When using live migrate to another dom0, this often happens:
> 
> [   37.511305] Freezing user space processes ... (elapsed 0.001 seconds)
> done.
> [   37.513316] OOM killer disabled.
> [   37.513323] Freezing remaining freezable tasks ... (elapsed 0.001
> seconds) done.
> [   37.514837] suspending xenstore...
> [   37.515142] xen:grant_table: Grant tables using version 1 layout
> [18446744002.593711] OOM killer enabled.
> [18446744002.593726] Restarting tasks ... done.
> [18446744002.604527] Setting capacity to 6291456

Tonight, I've been through 29 bisect steps to figure out a bit more. A
make defconfig with enabling Xen PV for domU reproduces the problem
already, so a complete cycle with compiling and testing had only to take
about 7 minutes.

So, it appears that this 18 gazillion seconds of uptime is a thing that
started happening earlier than the TCP situation already. All of the
test scenarios resulted in these huge uptime numbers in dmesg. Not all
of them result in TCP connections hanging.

> As a side effect, all open TCP connections stall, because the timestamp
> counters of packets sent to the outside world are affected:
> 
> https://syrinx.knorrie.org/~knorrie/tmp/tcp-stall.png

This is happening since:

commit 9a568de4818dea9a05af141046bd3e589245ab83
Author: Eric Dumazet 
Date:   Tue May 16 14:00:14 2017 -0700

tcp: switch TCP TS option (RFC 7323) to 1ms clock

In order to find out, the first 13 bisect steps were to figure out that
live migration was totally broken between...

commit bf22ff45bed664aefb5c4e43029057a199b7070c
Author: Jeffy Chen 
Date:   Mon Jun 26 19:33:34 2017 +0800

genirq: Avoid unnecessary low level irq function calls

...and...

commit bb68cfe2f5a7f43058aed299fdbb73eb281734ed
Author: Thomas Gleixner 
Date:   Mon Jul 31 22:07:09 2017 +0200

x86/hpet: Cure interface abuse in the resume path

In between are 12k+ commits. So, I restarted bisect and used either
revert of the first commit or cherry-pick of the fix to get a working
test case every single time.

http://paste.debian.net/plainh/be91aabd

> [...]
> 
> 3. Since this is related to time and clocks, the last thing today we
> tried was, instead of using default settings, put "clocksource=tsc
> tsc=stable:socket" on the xen command line and "clocksource=tsc" on the
> domU linux kernel line. What we observed after doing this, is that the
> failure happens less often, but still happens. Everything else applies.

Actually, it seems that the important thing is that uptime of the dom0s
is not very close to each other. After rebooting all four back without
tsc options, and then a few hours later rebooting one of them again, I
could easily reproduce again when live migrating to the later rebooted
server.

> Additional question:
> 
> It's 2018, should we have these "clocksource=tsc tsc=stable:socket" on
> Xen and "clocksource=tsc" anyways now, for Xen 4.11 and Linux 4.19
> domUs? All our hardware has 'TscInvariant = true'.
> 
> Related: https://news.ycombinator.com/item?id=13813079

This is still interesting.

 >8 

Now, the next question is... is 9a568de481 bad, or shouldn't there be 18
gazillion whatever uptime already... In Linux 4.9, this doesn't happen,
so next task will be to find out where that started.

to be continued...

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Live migrate with Linux >= 4.13 domU causes kernel time jumps and TCP connection stalls.

[Hans van Kranenburg]

So,

On 12/24/18 1:32 AM, Hans van Kranenburg wrote:
> 
> On 12/21/18 6:54 PM, Hans van Kranenburg wrote:
>>
>> We've been tracking down a live migration bug during the last three days
>> here at work, and here's what we found so far.
>>
>> 1. Xen version and dom0 linux kernel version don't matter.
>> 2. DomU kernel is >= Linux 4.13.
>>
>> When using live migrate to another dom0, this often happens:
>>
>> [   37.511305] Freezing user space processes ... (elapsed 0.001 seconds)
>> done.
>> [   37.513316] OOM killer disabled.
>> [   37.513323] Freezing remaining freezable tasks ... (elapsed 0.001
>> seconds) done.
>> [   37.514837] suspending xenstore...
>> [   37.515142] xen:grant_table: Grant tables using version 1 layout
>> [18446744002.593711] OOM killer enabled.
>> [18446744002.593726] Restarting tasks ... done.
>> [18446744002.604527] Setting capacity to 6291456
> 
> Tonight, I've been through 29 bisect steps to figure out a bit more. A
> make defconfig with enabling Xen PV for domU reproduces the problem
> already, so a complete cycle with compiling and testing had only to take
> about 7 minutes.
> 
> So, it appears that this 18 gazillion seconds of uptime is a thing that
> started happening earlier than the TCP situation already. All of the
> test scenarios resulted in these huge uptime numbers in dmesg. Not all
> of them result in TCP connections hanging.
> 
>> As a side effect, all open TCP connections stall, because the timestamp
>> counters of packets sent to the outside world are affected:
>>
>> https://syrinx.knorrie.org/~knorrie/tmp/tcp-stall.png
> 
> This is happening since:
> 
> commit 9a568de4818dea9a05af141046bd3e589245ab83
> Author: Eric Dumazet 
> Date:   Tue May 16 14:00:14 2017 -0700
> 
> tcp: switch TCP TS option (RFC 7323) to 1ms clock
> 
> [...]
> 
>> [...]
>>
>> 3. Since this is related to time and clocks, the last thing today we
>> tried was, instead of using default settings, put "clocksource=tsc
>> tsc=stable:socket" on the xen command line and "clocksource=tsc" on the
>> domU linux kernel line. What we observed after doing this, is that the
>> failure happens less often, but still happens. Everything else applies.
> 
> Actually, it seems that the important thing is that uptime of the dom0s
> is not very close to each other. After rebooting all four back without
> tsc options, and then a few hours later rebooting one of them again, I
> could easily reproduce again when live migrating to the later rebooted
> server.
> 
>> Additional question:
>>
>> It's 2018, should we have these "clocksource=tsc tsc=stable:socket" on
>> Xen and "clocksource=tsc" anyways now, for Xen 4.11 and Linux 4.19
>> domUs? All our hardware has 'TscInvariant = true'.
>>
>> Related: https://news.ycombinator.com/item?id=13813079
> 
> This is still interesting.
> 
>  >8 
> 
> Now, the next question is... is 9a568de481 bad, or shouldn't there be 18
> gazillion whatever uptime already... In Linux 4.9, this doesn't happen,
> so next task will be to find out where that started.

And that's...

commit f94c8d116997597fc00f0812b0ab9256e7b0c58f
Author: Peter Zijlstra 
Date:   Wed Mar 1 15:53:38 2017 +0100

sched/clock, x86/tsc: Rework the x86 'unstable' sched_clock() interface

a.k.a. v4.11-rc2~30^2

Before this commit, time listed in dmesg seems to follow uptime of the
domU, and after it, time in dmesg seems to jump around up and down when
live migrating to different dom0s, with the occasional/frequent jump to
a number above 180 which then also shows the TCP timestamp
breakage since 9a568de4.

So, next question is... what now? Any ideas appreciated.

Can anyone else reproduce this? I have super-common HP DL360 hardware
and mostly default settings, so it shouldn't be that hard.

Should I mail some other mailinglist with a question? Which one? Does
any of you Xen developers have more experience with time keeping code?

Regards,
Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Live migrate with Linux >= 4.13 domU causes kernel time jumps and TCP connection stalls.

[Hans van Kranenburg]

On 12/28/18 11:15 AM, Juergen Gross wrote:
> On 27/12/2018 22:12, Hans van Kranenburg wrote:
>> So,
>>
>> On 12/24/18 1:32 AM, Hans van Kranenburg wrote:
>>>
>>> On 12/21/18 6:54 PM, Hans van Kranenburg wrote:
>>>>
>>>> We've been tracking down a live migration bug during the last three days
>>>> here at work, and here's what we found so far.
>>>>
>>>> 1. Xen version and dom0 linux kernel version don't matter.
>>>> 2. DomU kernel is >= Linux 4.13.
>>>>
>>>> When using live migrate to another dom0, this often happens:
>>>>
>>>> [   37.511305] Freezing user space processes ... (elapsed 0.001 seconds)
>>>> done.
>>>> [   37.513316] OOM killer disabled.
>>>> [   37.513323] Freezing remaining freezable tasks ... (elapsed 0.001
>>>> seconds) done.
>>>> [   37.514837] suspending xenstore...
>>>> [   37.515142] xen:grant_table: Grant tables using version 1 layout
>>>> [18446744002.593711] OOM killer enabled.
>>>> [18446744002.593726] Restarting tasks ... done.
>>>> [18446744002.604527] Setting capacity to 6291456
>>>
>>> Tonight, I've been through 29 bisect steps to figure out a bit more. A
>>> make defconfig with enabling Xen PV for domU reproduces the problem
>>> already, so a complete cycle with compiling and testing had only to take
>>> about 7 minutes.
>>>
>>> So, it appears that this 18 gazillion seconds of uptime is a thing that
>>> started happening earlier than the TCP situation already. All of the
>>> test scenarios resulted in these huge uptime numbers in dmesg. Not all
>>> of them result in TCP connections hanging.
>>>
>>>> As a side effect, all open TCP connections stall, because the timestamp
>>>> counters of packets sent to the outside world are affected:
>>>>
>>>> https://syrinx.knorrie.org/~knorrie/tmp/tcp-stall.png
>>>
>>> This is happening since:
>>>
>>> commit 9a568de4818dea9a05af141046bd3e589245ab83
>>> Author: Eric Dumazet 
>>> Date:   Tue May 16 14:00:14 2017 -0700
>>>
>>> tcp: switch TCP TS option (RFC 7323) to 1ms clock
>>>
>>> [...]
>>>
>>>> [...]
>>>>
>>>> 3. Since this is related to time and clocks, the last thing today we
>>>> tried was, instead of using default settings, put "clocksource=tsc
>>>> tsc=stable:socket" on the xen command line and "clocksource=tsc" on the
>>>> domU linux kernel line. What we observed after doing this, is that the
>>>> failure happens less often, but still happens. Everything else applies.
>>>
>>> Actually, it seems that the important thing is that uptime of the dom0s
>>> is not very close to each other. After rebooting all four back without
>>> tsc options, and then a few hours later rebooting one of them again, I
>>> could easily reproduce again when live migrating to the later rebooted
>>> server.
>>>
>>>> Additional question:
>>>>
>>>> It's 2018, should we have these "clocksource=tsc tsc=stable:socket" on
>>>> Xen and "clocksource=tsc" anyways now, for Xen 4.11 and Linux 4.19
>>>> domUs? All our hardware has 'TscInvariant = true'.
>>>>
>>>> Related: https://news.ycombinator.com/item?id=13813079
>>>
>>> This is still interesting.
>>>
>>>  >8 
>>>
>>> Now, the next question is... is 9a568de481 bad, or shouldn't there be 18
>>> gazillion whatever uptime already... In Linux 4.9, this doesn't happen,
>>> so next task will be to find out where that started.
>>
>> And that's...
>>
>> commit f94c8d116997597fc00f0812b0ab9256e7b0c58f
>> Author: Peter Zijlstra 
>> Date:   Wed Mar 1 15:53:38 2017 +0100
>>
>> sched/clock, x86/tsc: Rework the x86 'unstable' sched_clock() interface
>>
>> a.k.a. v4.11-rc2~30^2
>>
>> Before this commit, time listed in dmesg seems to follow uptime of the
>> domU, and after it, time in dmesg seems to jump around up and down when
>> live migrating to different dom0s, with the occasional/frequent jump to
>> a number above 180 which then also shows the TCP timestamp
>> breakage since 9a568de4.
>>
>> So, next question is... what now? Any ideas appreciated.
>>
>> Can anyone else reproduc

Re: [Xen-devel] Live migrate with Linux >= 4.13 domU causes kernel time jumps and TCP connection stalls.

[Hans van Kranenburg]

On 1/7/19 1:04 PM, Juergen Gross wrote:
> On 28/12/2018 15:41, Hans van Kranenburg wrote:
>> On 12/28/18 11:15 AM, Juergen Gross wrote:
>>
>> [...]
>> So that explains the 18446742891.874140 number, which just corresponds
>> to something near to 'minus 23 minutes'.
> 
> I have a local reproducer for the issue now: instead of using live
> migration I'm just doing a "xl save" after the guest running for some
> minutes. The I reboot the host and do a "xl restore" as soon as
> possible.
> 
> Another note: HVM domains (and probably PVH, too) show the huge time
> info ("[18446742937.583537] ..."), while PV domains seem to show just
> a small jump backwards in time:
> 
> [  224.719316] Freezing user space processes ... (elapsed 0.001 seconds)
> done.
> [  224.720443] OOM killer disabled.
> [  224.720448] Freezing remaining freezable tasks ... (elapsed 0.001
> seconds) done.
> [  224.721678] PM: freeze of devices complete after 0.107 msecs
> [  224.721687] suspending xenstore...
> [  224.721726] PM: late freeze of devices complete after 0.037 msecs
> [  224.736062] PM: noirq freeze of devices complete after 14.325 msecs
> [  224.736155] xen:grant_table: Grant tables using version 1 layout
> [4.404026] Suspended for 187.219 seconds

And if you cause a time difference that lets it go down below zero?

I can just as easily reproduce with PV, and don't see much difference in
behavior with PVH. Actually, all the bisect steps to find it were done
using PV.

I haven't tried HVM, since I'm not using that at all.

Hans
___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH] xen: Fix x86 sched_clock() interface for xen

[Hans van Kranenburg]

On 1/10/19 11:07 AM, Juergen Gross wrote:
> Commit f94c8d11699759 ("sched/clock, x86/tsc: Rework the x86 'unstable'
> sched_clock() interface") broke Xen guest time handling across
> migration:
> 
> [  187.249951] Freezing user space processes ... (elapsed 0.001 seconds) done.
> [  187.251137] OOM killer disabled.
> [  187.251137] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) 
> done.
> [  187.252299] suspending xenstore...
> [  187.266987] xen:grant_table: Grant tables using version 1 layout
> [18446743811.706476] OOM killer enabled.
> [18446743811.706478] Restarting tasks ... done.
> [18446743811.720505] Setting capacity to 16777216

I tried this on top of 4.19.14 (together with "x86/mm: Fix guard hole
handling", which still doesn't seem to be in 4.19) and I can confirm the
fix:

Using a PV domU,

4.19 without patch:

[  646.199018] Freezing user space processes ... (elapsed 0.002 seconds)
done.
[  646.201305] OOM killer disabled.
[  646.201311] Freezing remaining freezable tasks ... (elapsed 0.001
seconds) done.
[  646.202699] suspending xenstore...
[  646.203005] xen:grant_table: Grant tables using version 1 layout
[18446200797.089367] OOM killer enabled.
[18446200797.089382] Restarting tasks ... done.


4.19 with patch, doing live migration movement between exactly the same
set of physical servers:

[   74.878062] Freezing user space processes ... (elapsed 0.002 seconds)
done.
[   74.880308] OOM killer disabled.
[   74.880314] Freezing remaining freezable tasks ... (elapsed 0.000
seconds) done.
[   74.881681] suspending xenstore...
[   74.887497] xen:grant_table: Grant tables using version 1 layout
[   74.942011] OOM killer enabled.
[   74.942025] Restarting tasks ... done.
[   74.947688] Setting capacity to 6291456
[   74.950833] Setting capacity to 10485760

Tested-by: Hans van Kranenburg 

Thanks!

> 
> Fix that by setting xen_sched_clock_offset at resume time to ensure a
> monotonic clock value.
> 
> Fixes: f94c8d11699759 ("sched/clock, x86/tsc: Rework the x86 'unstable' 
> sched_clock() interface")
> Cc:  # 4.11
> Reported-by: Hans van Kranenburg 
> Signed-off-by: Juergen Gross 
> ---
>  arch/x86/xen/suspend.c |  4 
>  arch/x86/xen/time.c| 11 +++
>  arch/x86/xen/xen-ops.h |  2 ++
>  3 files changed, 17 insertions(+)
> 
> diff --git a/arch/x86/xen/suspend.c b/arch/x86/xen/suspend.c
> index 1d83152c761b..45fc9caf3880 100644
> --- a/arch/x86/xen/suspend.c
> +++ b/arch/x86/xen/suspend.c
> @@ -67,6 +67,8 @@ void xen_arch_resume(void)
>  {
>   int cpu;
>  
> + xen_clocksource_resume();
> +
>   on_each_cpu(xen_vcpu_notify_restore, NULL, 1);
>  
>   for_each_online_cpu(cpu)
> @@ -81,4 +83,6 @@ void xen_arch_suspend(void)
>   xen_pmu_finish(cpu);
>  
>   on_each_cpu(xen_vcpu_notify_suspend, NULL, 1);
> +
> + xen_clocksource_suspend();
>  }
> diff --git a/arch/x86/xen/time.c b/arch/x86/xen/time.c
> index 72bf446c3fee..117ce958ffe6 100644
> --- a/arch/x86/xen/time.c
> +++ b/arch/x86/xen/time.c
> @@ -32,6 +32,7 @@
>  #define TIMER_SLOP   10
>  
>  static u64 xen_sched_clock_offset __read_mostly;
> +static u64 xen_clock_value_saved;
>  
>  /* Get the TSC speed from Xen */
>  static unsigned long xen_tsc_khz(void)
> @@ -54,6 +55,16 @@ static u64 xen_clocksource_read(void)
>   return ret;
>  }
>  
> +void xen_clocksource_suspend(void)
> +{
> + xen_clock_value_saved = xen_clocksource_read() - xen_sched_clock_offset;
> +}
> +
> +void xen_clocksource_resume(void)
> +{
> + xen_sched_clock_offset = xen_clocksource_read() - xen_clock_value_saved;
> +}
> +
>  static u64 xen_clocksource_get_cycles(struct clocksource *cs)
>  {
>   return xen_clocksource_read();
> diff --git a/arch/x86/xen/xen-ops.h b/arch/x86/xen/xen-ops.h
> index 0e60bd918695..a17d3bdab6b8 100644
> --- a/arch/x86/xen/xen-ops.h
> +++ b/arch/x86/xen/xen-ops.h
> @@ -63,6 +63,8 @@ void __init xen_build_dynamic_phys_to_machine(void);
>  void __init xen_vmalloc_p2m_tree(void);
>  
>  void xen_init_irq_ops(void);
> +void xen_clocksource_suspend(void);
> +void xen_clocksource_resume(void);
>  void xen_setup_timer(int cpu);
>  void xen_setup_runstate_info(int cpu);
>  void xen_teardown_timer(int cpu);
> 


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v2] xen: Fix x86 sched_clock() interface for xen

[Hans van Kranenburg]

Hi,

On 1/11/19 1:08 PM, Juergen Gross wrote:
> Commit f94c8d11699759 ("sched/clock, x86/tsc: Rework the x86 'unstable'
> sched_clock() interface") broke Xen guest time handling across
> migration:
> 
> [  187.249951] Freezing user space processes ... (elapsed 0.001 seconds) done.
> [  187.251137] OOM killer disabled.
> [  187.251137] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) 
> done.
> [  187.252299] suspending xenstore...
> [  187.266987] xen:grant_table: Grant tables using version 1 layout
> [18446743811.706476] OOM killer enabled.
> [18446743811.706478] Restarting tasks ... done.
> [18446743811.720505] Setting capacity to 16777216
> 
> Fix that by setting xen_sched_clock_offset at resume time to ensure a
> monotonic clock value.
> 
> [...]

I'm throwing around a PV domU over a bunch of test servers with live
migrate now, and in between the kernel logging, I'm seeing this:

[Fri Jan 11 13:58:42 2019] Freezing user space processes ... (elapsed
0.002 seconds) done.
[Fri Jan 11 13:58:42 2019] OOM killer disabled.
[Fri Jan 11 13:58:42 2019] Freezing remaining freezable tasks ...
(elapsed 0.000 seconds) done.
[Fri Jan 11 13:58:42 2019] suspending xenstore...
[Fri Jan 11 13:58:42 2019] [ cut here ]
[Fri Jan 11 13:58:42 2019] Current state: 1
[Fri Jan 11 13:58:42 2019] WARNING: CPU: 3 PID: 0 at
kernel/time/clockevents.c:133 clockevents_switch_state+0x48/0xe0
[Fri Jan 11 13:58:42 2019] Modules linked in:
[Fri Jan 11 13:58:42 2019] CPU: 3 PID: 0 Comm: swapper/3 Not tainted
4.19.14+ #1
[Fri Jan 11 13:58:42 2019] RIP: e030:clockevents_switch_state+0x48/0xe0
[Fri Jan 11 13:58:42 2019] Code: 8b 0c cd 40 ee 00 82 e9 d6 5b d1 00 80
3d 8e 8d 43 01 00 75 17 89 c6 48 c7 c7 92 62 1f 82 c6 05 7c 8d 43 01 01
e8 f8 22 f8 ff <0f> 0b 5b 5d f3 c3 83 e2 01 74 f7 48 8b 47 48 48 85 c0
74 69 48 89
[Fri Jan 11 13:58:42 2019] RSP: e02b:c9787e30 EFLAGS: 00010082
[Fri Jan 11 13:58:42 2019] RAX:  RBX: 88805df94d80
RCX: 0006
[Fri Jan 11 13:58:42 2019] RDX: 0007 RSI: 0001
RDI: 88805df963f0
[Fri Jan 11 13:58:42 2019] RBP: 0004 R08: 
R09: 0119
[Fri Jan 11 13:58:42 2019] R10: 0020 R11: 82af4e2d
R12: 88805df9ca40
[Fri Jan 11 13:58:42 2019] R13: 000dd28d6ca6 R14: 
R15: 
[Fri Jan 11 13:58:42 2019] FS:  7f34193ce040()
GS:88805df8() knlGS:
[Fri Jan 11 13:58:42 2019] CS:  e033 DS: 002b ES: 002b CR0: 80050033
[Fri Jan 11 13:58:42 2019] CR2: 7f6220be50e1 CR3: 5ce5c000
CR4: 2660
[Fri Jan 11 13:58:42 2019] Call Trace:
[Fri Jan 11 13:58:42 2019]  tick_program_event+0x4b/0x70
[Fri Jan 11 13:58:42 2019]  hrtimer_try_to_cancel+0xa8/0x100
[Fri Jan 11 13:58:42 2019]  hrtimer_cancel+0x10/0x20
[Fri Jan 11 13:58:42 2019]  __tick_nohz_idle_restart_tick+0x45/0xd0
[Fri Jan 11 13:58:42 2019]  tick_nohz_idle_exit+0x93/0xa0
[Fri Jan 11 13:58:42 2019]  do_idle+0x149/0x260
[Fri Jan 11 13:58:42 2019]  cpu_startup_entry+0x6a/0x70
[Fri Jan 11 13:58:42 2019] ---[ end trace 519c07d1032908f8 ]---
[Fri Jan 11 13:58:42 2019] xen:grant_table: Grant tables using version 1
layout
[Fri Jan 11 13:58:42 2019] OOM killer enabled.
[Fri Jan 11 13:58:42 2019] Restarting tasks ... done.
[Fri Jan 11 13:58:42 2019] Setting capacity to 6291456
[Fri Jan 11 13:58:42 2019] Setting capacity to 10485760

This always happens on every *first* live migrate that I do after
starting the domU.

OTOH, With v1 I've indeed also observed the 'leftover' time jump on the
first line when resuming. It didn't happen always, but it seems to not
happen anymore at all now, so that's good.

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v2] xen: Fix x86 sched_clock() interface for xen

[Hans van Kranenburg]

On 1/11/19 3:01 PM, Juergen Gross wrote:
> On 11/01/2019 14:12, Hans van Kranenburg wrote:
>> Hi,
>>
>> On 1/11/19 1:08 PM, Juergen Gross wrote:
>>> Commit f94c8d11699759 ("sched/clock, x86/tsc: Rework the x86 'unstable'
>>> sched_clock() interface") broke Xen guest time handling across
>>> migration:
>>>
>>> [  187.249951] Freezing user space processes ... (elapsed 0.001 seconds) 
>>> done.
>>> [  187.251137] OOM killer disabled.
>>> [  187.251137] Freezing remaining freezable tasks ... (elapsed 0.001 
>>> seconds) done.
>>> [  187.252299] suspending xenstore...
>>> [  187.266987] xen:grant_table: Grant tables using version 1 layout
>>> [18446743811.706476] OOM killer enabled.
>>> [18446743811.706478] Restarting tasks ... done.
>>> [18446743811.720505] Setting capacity to 16777216
>>>
>>> Fix that by setting xen_sched_clock_offset at resume time to ensure a
>>> monotonic clock value.
>>>
>>> [...]
>>
>> I'm throwing around a PV domU over a bunch of test servers with live
>> migrate now, and in between the kernel logging, I'm seeing this:
>>
>> [Fri Jan 11 13:58:42 2019] Freezing user space processes ... (elapsed
>> 0.002 seconds) done.
>> [Fri Jan 11 13:58:42 2019] OOM killer disabled.
>> [Fri Jan 11 13:58:42 2019] Freezing remaining freezable tasks ...
>> (elapsed 0.000 seconds) done.
>> [Fri Jan 11 13:58:42 2019] suspending xenstore...
>> [Fri Jan 11 13:58:42 2019] [ cut here ]
>> [Fri Jan 11 13:58:42 2019] Current state: 1
>> [Fri Jan 11 13:58:42 2019] WARNING: CPU: 3 PID: 0 at
>> kernel/time/clockevents.c:133 clockevents_switch_state+0x48/0xe0
>> [Fri Jan 11 13:58:42 2019] Modules linked in:
>> [Fri Jan 11 13:58:42 2019] CPU: 3 PID: 0 Comm: swapper/3 Not tainted
>> 4.19.14+ #1
>> [Fri Jan 11 13:58:42 2019] RIP: e030:clockevents_switch_state+0x48/0xe0
>> [Fri Jan 11 13:58:42 2019] Code: 8b 0c cd 40 ee 00 82 e9 d6 5b d1 00 80
>> 3d 8e 8d 43 01 00 75 17 89 c6 48 c7 c7 92 62 1f 82 c6 05 7c 8d 43 01 01
>> e8 f8 22 f8 ff <0f> 0b 5b 5d f3 c3 83 e2 01 74 f7 48 8b 47 48 48 85 c0
>> 74 69 48 89
>> [Fri Jan 11 13:58:42 2019] RSP: e02b:c9787e30 EFLAGS: 00010082
>> [Fri Jan 11 13:58:42 2019] RAX:  RBX: 88805df94d80
>> RCX: 0006
>> [Fri Jan 11 13:58:42 2019] RDX: 0007 RSI: 0001
>> RDI: 88805df963f0
>> [Fri Jan 11 13:58:42 2019] RBP: 0004 R08: 
>> R09: 0119
>> [Fri Jan 11 13:58:42 2019] R10: 0020 R11: 82af4e2d
>> R12: 88805df9ca40
>> [Fri Jan 11 13:58:42 2019] R13: 000dd28d6ca6 R14: 
>> R15: 
>> [Fri Jan 11 13:58:42 2019] FS:  7f34193ce040()
>> GS:88805df8() knlGS:
>> [Fri Jan 11 13:58:42 2019] CS:  e033 DS: 002b ES: 002b CR0: 80050033
>> [Fri Jan 11 13:58:42 2019] CR2: 7f6220be50e1 CR3: 5ce5c000
>> CR4: 2660
>> [Fri Jan 11 13:58:42 2019] Call Trace:
>> [Fri Jan 11 13:58:42 2019]  tick_program_event+0x4b/0x70
>> [Fri Jan 11 13:58:42 2019]  hrtimer_try_to_cancel+0xa8/0x100
>> [Fri Jan 11 13:58:42 2019]  hrtimer_cancel+0x10/0x20
>> [Fri Jan 11 13:58:42 2019]  __tick_nohz_idle_restart_tick+0x45/0xd0
>> [Fri Jan 11 13:58:42 2019]  tick_nohz_idle_exit+0x93/0xa0
>> [Fri Jan 11 13:58:42 2019]  do_idle+0x149/0x260
>> [Fri Jan 11 13:58:42 2019]  cpu_startup_entry+0x6a/0x70
>> [Fri Jan 11 13:58:42 2019] ---[ end trace 519c07d1032908f8 ]---
>> [Fri Jan 11 13:58:42 2019] xen:grant_table: Grant tables using version 1
>> layout
>> [Fri Jan 11 13:58:42 2019] OOM killer enabled.
>> [Fri Jan 11 13:58:42 2019] Restarting tasks ... done.
>> [Fri Jan 11 13:58:42 2019] Setting capacity to 6291456
>> [Fri Jan 11 13:58:42 2019] Setting capacity to 10485760
>>
>> This always happens on every *first* live migrate that I do after
>> starting the domU.
> 
> Yeah, its a WARN_ONCE().
> 
> And you didn't see it with v1 of the patch?

No.

> On the first glance this might be another bug just being exposed by
> my patch.
> 
> I'm investigating further, but this might take some time. Could you
> meanwhile verify the same happens with kernel 5.0-rc1? That was the
> one I tested with and I didn't spot that WARN.

I have Linux 5.0-rc1 with v2 on top now, which gives me this on live
migrate:

[   51.845967] xen:grant_table: Grant tables using version 1 layout
[   51.871076] BUG: unable to handle 

Re: [Xen-devel] [PATCH v2] xen: Fix x86 sched_clock() interface for xen

[Hans van Kranenburg]

On 1/11/19 4:57 PM, Hans van Kranenburg wrote:
> On 1/11/19 3:01 PM, Juergen Gross wrote:
>> On 11/01/2019 14:12, Hans van Kranenburg wrote:
>>> Hi,
>>>
>>> On 1/11/19 1:08 PM, Juergen Gross wrote:
>>>> Commit f94c8d11699759 ("sched/clock, x86/tsc: Rework the x86 'unstable'
>>>> sched_clock() interface") broke Xen guest time handling across
>>>> migration:
>>>>
>>>> [  187.249951] Freezing user space processes ... (elapsed 0.001 seconds) 
>>>> done.
>>>> [  187.251137] OOM killer disabled.
>>>> [  187.251137] Freezing remaining freezable tasks ... (elapsed 0.001 
>>>> seconds) done.
>>>> [  187.252299] suspending xenstore...
>>>> [  187.266987] xen:grant_table: Grant tables using version 1 layout
>>>> [18446743811.706476] OOM killer enabled.
>>>> [18446743811.706478] Restarting tasks ... done.
>>>> [18446743811.720505] Setting capacity to 16777216
>>>>
>>>> Fix that by setting xen_sched_clock_offset at resume time to ensure a
>>>> monotonic clock value.
>>>>
>>>> [...]
>>>
>>> I'm throwing around a PV domU over a bunch of test servers with live
>>> migrate now, and in between the kernel logging, I'm seeing this:
>>>
>>> [Fri Jan 11 13:58:42 2019] Freezing user space processes ... (elapsed
>>> 0.002 seconds) done.
>>> [Fri Jan 11 13:58:42 2019] OOM killer disabled.
>>> [Fri Jan 11 13:58:42 2019] Freezing remaining freezable tasks ...
>>> (elapsed 0.000 seconds) done.
>>> [Fri Jan 11 13:58:42 2019] suspending xenstore...
>>> [Fri Jan 11 13:58:42 2019] [ cut here ]
>>> [Fri Jan 11 13:58:42 2019] Current state: 1
>>> [Fri Jan 11 13:58:42 2019] WARNING: CPU: 3 PID: 0 at
>>> kernel/time/clockevents.c:133 clockevents_switch_state+0x48/0xe0
>>> [...]
>>>
>>> This always happens on every *first* live migrate that I do after
>>> starting the domU.
>>
>> Yeah, its a WARN_ONCE().

Ok, false alarm. It's there, but not caused by this patch.

I changed the WARN_ONCE to WARN for funs, and now I get it a lot more
already (v2):

https://paste.debian.net/plainh/d535a379

>> And you didn't see it with v1 of the patch?
> 
> No.

I was wrong. I tried a bit more, and I can also reproduce without v1 or
v2 patch at all, and I can reproduce it with v4.19.9. Just sometimes
needs a dozen times more live migrating it before it shows up.

I cannot make it happen to show up with the Debian 4.19.9 distro kernel,
that's why it was new for me.

So, let's ignore it in this thread now.

>> On the first glance this might be another bug just being exposed by
>> my patch.
>>
>> I'm investigating further, but this might take some time. Could you
>> meanwhile verify the same happens with kernel 5.0-rc1? That was the
>> one I tested with and I didn't spot that WARN.
> 
> I have Linux 5.0-rc1 with v2 on top now, which gives me this on live
> migrate:
> 
> [...]
> [   51.871076] BUG: unable to handle kernel NULL pointer dereference at
> 0098
> [   51.871091] #PF error: [normal kernel read fault]
> [   51.871100] PGD 0 P4D 0
> [   51.871109] Oops:  [#1] SMP NOPTI
> [   51.871117] CPU: 0 PID: 36 Comm: xenwatch Not tainted 5.0.0-rc1 #1
> [   51.871132] RIP: e030:blk_mq_map_swqueue+0x103/0x270
> [...]

Dunno about all the 5.0-rc1 crashes yet. I can provide more feedback
about that if you want, but not in here I presume.

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v3] xen: Fix x86 sched_clock() interface for xen

[Hans van Kranenburg]

Hi,

On 1/14/19 1:44 PM, Juergen Gross wrote:
> Commit f94c8d11699759 ("sched/clock, x86/tsc: Rework the x86 'unstable'
> sched_clock() interface") broke Xen guest time handling across
> migration:
> 
> [  187.249951] Freezing user space processes ... (elapsed 0.001 seconds) done.
> [  187.251137] OOM killer disabled.
> [  187.251137] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) 
> done.
> [  187.252299] suspending xenstore...
> [  187.266987] xen:grant_table: Grant tables using version 1 layout
> [18446743811.706476] OOM killer enabled.
> [18446743811.706478] Restarting tasks ... done.
> [18446743811.720505] Setting capacity to 16777216
> 
> Fix that by setting xen_sched_clock_offset at resume time to ensure a
> monotonic clock value.
> 
> [...]

With v3 of the patch, I see the time jump in one log line happen, but
only when using PVH.

[   49.486453] Freezing user space processes ... (elapsed 0.002 seconds)
done.
[   49.488743] OOM killer disabled.
[   49.488764] Freezing remaining freezable tasks ... (elapsed 0.001
seconds) done.
[   49.491117] suspending xenstore...
[2000731.388722] xen:events: Xen HVM callback vector for event delivery
is enabled
[   49.491750] xen:grant_table: Grant tables using version 1 layout
[   49.810722] OOM killer enabled.
[   49.810744] Restarting tasks ... done.
[   49.856263] Setting capacity to 6291456
[   50.006002] Setting capacity to 10485760

If I start as PV, it never seems to happen.

Up to you to decide how important this is. :)

FYI this is with v3 on top of the Debian stretch-backports 4.19 kernel,
which I'm starting to use now to reboot things with.

-# uname -a
Linux appnode-kylie 4.19.0-0.bpo.1-cloud-amd64 #1 SMP Debian
4.19.12-1~bpo9+1+mendix1 (2019-01-15) x86_64 GNU/Linux

https://salsa.debian.org/knorrie-guest/linux/commits/mendix/stretch-backports

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v3] xen: Fix x86 sched_clock() interface for xen

[Hans van Kranenburg]

Hi Boris,

On 1/14/19 2:54 PM, Boris Ostrovsky wrote:
> On 1/14/19 7:44 AM, Juergen Gross wrote:
>> Commit f94c8d11699759 ("sched/clock, x86/tsc: Rework the x86 'unstable'
>> sched_clock() interface") broke Xen guest time handling across
>> migration:
>>
>> [  187.249951] Freezing user space processes ... (elapsed 0.001 seconds) 
>> done.
>> [  187.251137] OOM killer disabled.
>> [  187.251137] Freezing remaining freezable tasks ... (elapsed 0.001 
>> seconds) done.
>> [  187.252299] suspending xenstore...
>> [  187.266987] xen:grant_table: Grant tables using version 1 layout
>> [18446743811.706476] OOM killer enabled.
>> [18446743811.706478] Restarting tasks ... done.
>> [18446743811.720505] Setting capacity to 16777216
>>
>> Fix that by setting xen_sched_clock_offset at resume time to ensure a
>> monotonic clock value.
>>
>> Fixes: f94c8d11699759 ("sched/clock, x86/tsc: Rework the x86 'unstable' 
>> sched_clock() interface")
>> Cc:  # 4.11
>> Reported-by: Hans van Kranenburg 
>> Signed-off-by: Juergen Gross 
> 
> Reviewed-by: Boris Ostrovsky 

Can you please change the address to my work email?

Reported-by: Hans van Kranenburg 

Also (see other email):

Tested-by: Hans van Kranenburg 

Thanks,
Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] [PATCH] tools/misc: Remove obsolete xen-bugtool

[Hans van Kranenburg]

xen-bugtool relies on code that has been removed in commit 9e8672f1c3
"tools: remove xend and associated python modules", more than 5 years
ago. Remove it, since it confuses users.

-$ /usr/sbin/xen-bugtool
Traceback (most recent call last):
  File "/usr/sbin/xen-bugtool", line 9, in 
from xen.util import bugtool
ImportError: No module named xen.util

Signed-off-by: Hans van Kranenburg 
Link: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=866380
Cc: Ian Jackson 
Cc: Wei Liu 
---
 docs/README.xen-bugtool | 16 
 tools/misc/Makefile |  2 --
 tools/misc/xen-bugtool  | 17 -
 3 files changed, 35 deletions(-)
 delete mode 100644 docs/README.xen-bugtool
 delete mode 100644 tools/misc/xen-bugtool

diff --git a/docs/README.xen-bugtool b/docs/README.xen-bugtool
deleted file mode 100644
index a7e95ef4ce..00
--- a/docs/README.xen-bugtool
+++ /dev/null
@@ -1,16 +0,0 @@
-xen-bugtool
-===
-
-The xen-bugtool command line application will collate the Xen dmesg output,
-details of the hardware configuration of your machine, information about the
-build of Xen that you are using, plus, if you allow it, various logs.
-
-The information collated can either be posted to a Xen Bugzilla bug (this bug
-must already exist in the system, and you must be a registered user there), or
-it can be saved as a .tar.bz2 for sending or archiving.
-
-The collated logs may contain private information, and if you are at all
-worried about that, you should not use this tool, or you should explicitly
-exclude those logs from the archive.
-
-xen-bugtool is wholly interactive, so simply run it, and answer the questions.
diff --git a/tools/misc/Makefile b/tools/misc/Makefile
index eaa28793ef..fd91202950 100644
--- a/tools/misc/Makefile
+++ b/tools/misc/Makefile
@@ -17,7 +17,6 @@ INSTALL_BIN+= xencov_split
 INSTALL_BIN += $(INSTALL_BIN-y)
 
 # Everything to be installed in regular sbin/
-INSTALL_SBIN   += xen-bugtool
 INSTALL_SBIN-$(CONFIG_MIGRATE) += xen-hptool
 INSTALL_SBIN-$(CONFIG_X86) += xen-hvmcrash
 INSTALL_SBIN-$(CONFIG_X86) += xen-hvmctx
@@ -41,7 +40,6 @@ INSTALL_PRIVBIN+= xenpvnetboot
 TARGETS_ALL := $(INSTALL_BIN) $(INSTALL_SBIN) $(INSTALL_PRIVBIN)
 
 # Everything which only needs copying to install
-TARGETS_COPY += xen-bugtool
 TARGETS_COPY += xen-ringwatch
 TARGETS_COPY += xencons
 TARGETS_COPY += xencov_split
diff --git a/tools/misc/xen-bugtool b/tools/misc/xen-bugtool
deleted file mode 100644
index a3742b4787..00
--- a/tools/misc/xen-bugtool
+++ /dev/null
@@ -1,17 +0,0 @@
-#!/usr/bin/env python
-
-#  -*- mode: python; -*-
-
-# Copyright (c) 2005, XenSource Ltd.
-
-import sys
-
-from xen.util import bugtool
-
-
-if __name__ == "__main__":
-try:
-sys.exit(bugtool.main())
-except KeyboardInterrupt:
-print "\nInterrupted."
-sys.exit(1)
-- 
2.20.1


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH] docs/sphinx: Introduction

[Hans van Kranenburg]

Hi, (dropped most of the Cc:)

On 8/7/19 9:41 PM, Andrew Cooper wrote:
> Put together an introduction page for the Sphinx/RST docs, along with a
> glossary which will accumulate over time.
> [...]
I've been using Xen for 13+ years now, so I'm not really able to provide
feedback about how someone new to it would experience things.

But, I think there's some feedback I can provide.

The first page, admin-guide/introduction is very well written, it's
short and it sets a very good frame of reference. Keep it like this.

When writing documentation, you're not only providing information. You
can "steer" a lot of things. By consciously deciding about the exact
technical level of stuff that you provide on the "front page", you'll
automatically cause a selection of audience that you want to stay or
want to browse/scare away because it's not for them.

And, I think this first introduction page exactly does that, but I this
is where I'm of course biased. I think it will "work" for a new user who
already knows what a NIC and a Kernel is, and it will work for an
interested developer, looking to help figuring out to get it running on
$whatever hardware not fully supported.

The other pages already available seem to be developer documentation
instead of user documentation.

One of the things on my wishlist is to help cleaning up the end user
documentation for Xen. Xen itself has a wiki with a lot of horrible
outdated information, there's a Debian wiki with even more horrible
outated information, and so on. As a new user, you completely get lost
because you have no idea what's still relevant or not. It's a shame that
we lose potential new users for which the product would be a good fit
because of that.

(With a group of motivated people, a few with technical knowledge and a
few with decent tech. writer skills we could do a "sprint" getting some
big hammers and chainsaws out to do some huge spring cleaning and
restructure it.) A wiki is a great solution for short-term low-barrier
gathering of information by anyone, but in the long term it's just a big
accumulation of cruft without a clear maintainer.

Instead of fully starting to hijack this thread now, my last feedback
would be: in your git commit message, you don't mention what your target
audience is. I'm interested to hear what it is.

Thanks,
Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [Pkg-xen-devel] [admin] [BUG] task jbd2/xvda4-8:174 blocked for more than 120 seconds.

[Hans van Kranenburg]

Hi,

Upstream mailing list is at:
  xen-devel@lists.xenproject.org

Apparently,
  x...@packages.debian.org
results in a message to
  pkg-xen-de...@lists.alioth.debian.org

On 2/8/19 6:13 PM, Samuel Thibault wrote:
> 
> Sacha, le ven. 08 févr. 2019 18:00:22 +0100, a ecrit:
>> On  Debian GNU/Linux 9.7 (stretch) amd64, we have a bug on the last Xen
>> Hypervisor version:
>>
>> xen-hypervisor-4.8-amd64 4.8.5+shim4.10.2+xsa282
> 
> (Read: 4.8.5+shim4.10.2+xsa282-1+deb9u11)
> 
>> The rollback on the previous package version corrected the problem:
>>
>> xen-hypervisor-4.8-amd64 4.8.4+xsa273+shim4.10.1+xsa273-1+deb9u10

Since this is the first message arriving about this in my inbox, can you
explain what "the problem" is?

> (Only the hypervisor needed to be downgraded to fix the issue)
> 
>> The errors are on the domU a frozen file system until a kernel panic.

Do you have a reproducable case that shows success with the previous Xen
hypervisor package and failure with the new one, while keeping all other
things the same?

This seems like an upstream thing, because for 4.8, the Debian package
updates are almost exclusively shipping upstream stable udpates.

> (not really a kernel panic, just a warning that processes are stuck for
> more than 2m waiting for fs I/O).
> 
> So it looks like the deb9u11 update brought issues with the vbd
> behavior, did anybody experience this?
> 
> The dom0 and domU kernels are linux-image-4.9.0-8-amd64 4.9.130-2.

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [admin] [Pkg-xen-devel] [BUG] task jbd2/xvda4-8:174 blocked for more than 120 seconds.

[Hans van Kranenburg]

Hi,

On 2/9/19 12:16 AM, Samuel Thibault wrote:
> 
> Hans van Kranenburg, le ven. 08 févr. 2019 20:18:44 +0100, a ecrit:
>> [...]
>>
>> On 2/8/19 6:13 PM, Samuel Thibault wrote:
>>>
>>> Sacha, le ven. 08 févr. 2019 18:00:22 +0100, a ecrit:
>>>> On  Debian GNU/Linux 9.7 (stretch) amd64, we have a bug on the last Xen
>>>> Hypervisor version:
>>>>
>>>> xen-hypervisor-4.8-amd64 4.8.5+shim4.10.2+xsa282
>>>
>>> (Read: 4.8.5+shim4.10.2+xsa282-1+deb9u11)
>>>
>>>> The rollback on the previous package version corrected the problem:
>>>>
>>>> xen-hypervisor-4.8-amd64 4.8.4+xsa273+shim4.10.1+xsa273-1+deb9u10
>>
>> Since this is the first message arriving about this in my inbox, can you
>> explain what "the problem" is?
> 
> I have forwarded the original mail: all VM I/O get stuck, and thus the
> VM becomes unusable.

These are in many cases the symptoms of running out of "grant frames".
So let's verify first if this is the case or not.

Your xen-utils-4.8 packages contains a program at
/usr/lib/xen-4.8/bin/xen-diag that you can use in the dom0 to gather
information.

e.g.

  -# ./xen-diag  gnttab_query_size 5
  domid=5: nr_frames=11, max_nr_frames=32

If this nr_frames hits the max allowed, then randomly things will stall.
This does not have to happen directly after domU boot, but it likely
happens later, when disks/cpus are actually used. There is no useful
message/hint at all in the domU kernel (yet) abuot this when it happens.

Can you verify if this is happening?

With Xen 4.8, you can add gnttab_max_frames=64 (or another number, but
higher than the default 32) to the xen hypervisor command line and reboot.

For Xen 4.11 which will be in Buster, the default is 64 and the way to
configure higher values/limits for dom0 and domU have changed. There
will be some text about this recurring problem in the README.Debian
under known issues.

>>> (Only the hypervisor needed to be downgraded to fix the issue)
>>>
>>>> The errors are on the domU a frozen file system until a kernel panic.
>>
>> Do you have a reproducable case that shows success with the previous Xen
>> hypervisor package and failure with the new one, while keeping all other
>> things the same?
> 
> We have a production system which gets to hang within about a day. We
> don't know what exactly triggers the issue.
> 
>> This seems like an upstream thing, because for 4.8, the Debian package
>> updates are almost exclusively shipping upstream stable udpates.
> 
> Ok.

Related:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=880554

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [admin] [Pkg-xen-devel] [BUG] task jbd2/xvda4-8:174 blocked for more than 120 seconds.

[Hans van Kranenburg]

On 2/11/19 2:37 AM, Dongli Zhang wrote:
> 
> On 2/10/19 12:35 AM, Samuel Thibault wrote:
>>
>> Hans van Kranenburg, le sam. 09 févr. 2019 17:01:55 +0100, a ecrit:
>>>> I have forwarded the original mail: all VM I/O get stuck, and thus the
>>>> VM becomes unusable.
>>>
>>> These are in many cases the symptoms of running out of "grant frames".
>>
>> Oh!  That could be it indeed.  I'm wondering what could be monopolizing
>> them, though, and why +deb9u11 is affected while +deb9u10 is not.  I'm
>> afraid increasing the gnttab max size to 32 might just defer filling it
>> up.
>>
>>>   -# ./xen-diag  gnttab_query_size 5
>>>   domid=5: nr_frames=11, max_nr_frames=32
>>
>> The current value is 31 over max 32 indeed.
> 
> Assuming this is grant v1, there are still 4096/8=512 grant references 
> available
> (32-31=1 frame available). I do not think the I/O hang can be affected by the
> lack of grant entry.

I suspect that 31 measurement was taken when the domU was not hanging yet.

> If to increase the max frame to 64 takes effect, it is weird why the I/O would
> hang when there are still 512 entries available.
> 
>>> With Xen 4.8, you can add gnttab_max_frames=64 (or another number, but
>>> higher than the default 32) to the xen hypervisor command line and reboot.
>>
>> admin@: I made the modification in the grub config. We can probably try
>> to reboot with the newer hypervisor, and monitor that value.

K

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH for-next v2 v2 2/5] libxl: make python scripts work with python 2.6 and up

[Hans van Kranenburg]

On 3/6/19 6:52 PM, Wei Liu wrote:
> Go through the transformations suggested by 2to3 and pick the
> necessary ones.
> 
> Use sys.stderr.write to avoid importing from __future__.
> 
> Signed-off-by: Wei Liu 
> ---
>  tools/libxl/gentest.py  |  2 +-
>  tools/libxl/gentypes.py | 10 +-
>  tools/libxl/idl.py  | 13 ++---
>  3 files changed, 12 insertions(+), 13 deletions(-)
> 
> diff --git a/tools/libxl/gentest.py b/tools/libxl/gentest.py
> index 989959fc68..81e13b437c 100644
> --- a/tools/libxl/gentest.py
> +++ b/tools/libxl/gentest.py
> @@ -86,7 +86,7 @@ def gen_rand_init(ty, v, indent = "", parent = None):
>  
>  if __name__ == '__main__':
>  if len(sys.argv) < 3:
> -print >>sys.stderr, "Usage: gentest.py  "
> +sys.stderr.write("Usage: gentest.py  \n")
>  sys.exit(1)
>  
>  random.seed(os.getenv('LIBXL_TESTIDL_SEED'))
> diff --git a/tools/libxl/gentypes.py b/tools/libxl/gentypes.py
> index 88e5c5f30e..656c157c01 100644
> --- a/tools/libxl/gentypes.py
> +++ b/tools/libxl/gentypes.py
> @@ -576,14 +576,14 @@ def libxl_C_enum_from_string(ty, str, e, indent = "
> "):
>  
>  if __name__ == '__main__':
>  if len(sys.argv) != 6:
> -print >>sys.stderr, "Usage: gentypes.py   
>   "
> +sys.stderr.write("Usage: gentypes.py
>  \n")
>  sys.exit(1)
>  
>  (_, idlname, header, header_private, header_json, impl) = sys.argv
>  
>  (builtins,types) = idl.parse(idlname)
>  
> -print "outputting libxl type definitions to %s" % header
> +print("outputting libxl type definitions to %s" % header)

Note that print is *not* a function in python 2, unless you..

  from __future__ import print_function

...which should not be avoided.

It looks like print() just works as a function, but it's not, and I
would recommend against introducing this kind of misleading code.

In print("hallo"), the parentheses are part of an expression, like in
  ((1+ 1) * 2)

Other syntax with parentheses creates tuples.

Python 2.7.13 (>>> replaced with ===)

=== type(())

=== type((1+1))

=== type((1, 2))


=== print () <- tuple
()
=== print (1+1) <- expression
2
=== print (1, 2) <- tuple
(1, 2)

=== from __future__ import print_function
=== print()

=== print(1+1)
2
=== print(1, 2)
1 2

>  f = open(header, "w")
>  
> @@ -633,7 +633,7 @@ if __name__ == '__main__':
>  f.write("""#endif /* %s */\n""" % (header_define))
>  f.close()
>  
> -print "outputting libxl JSON definitions to %s" % header_json
> +print("outputting libxl JSON definitions to %s" % header_json)
>  
>  f = open(header_json, "w")
>  
> @@ -657,7 +657,7 @@ if __name__ == '__main__':
>  f.write("""#endif /* %s */\n""" % header_json_define)
>  f.close()
>  
> -print "outputting libxl type internal definitions to %s" % header_private
> +print("outputting libxl type internal definitions to %s" % 
> header_private)
>  
>  f = open(header_private, "w")
>  
> @@ -683,7 +683,7 @@ if __name__ == '__main__':
>  f.write("""#endif /* %s */\n""" % header_json_define)
>  f.close()
>  
> -print "outputting libxl type implementations to %s" % impl
> +print("outputting libxl type implementations to %s" % impl)
>  
>  f = open(impl, "w")
>  f.write("""
> diff --git a/tools/libxl/idl.py b/tools/libxl/idl.py
> index 2a7f3c44fe..b5bfc66b50 100644
> --- a/tools/libxl/idl.py
> +++ b/tools/libxl/idl.py
> @@ -11,7 +11,7 @@ DIR_BOTH = 3
>  _default_namespace = ""
>  def namespace(s):
>  if type(s) != str:
> -raise TypeError, "Require a string for the default namespace."
> +raise TypeError("Require a string for the default namespace.")
>  global _default_namespace
>  _default_namespace = s
>  
> @@ -346,7 +346,7 @@ class OrderedDict(dict):
>  return [(x,self[x]) for x in self.__ordered]
>  
>  def parse(f):
> -print >>sys.stderr, "Parsing %s" % f
> +sys.stderr.write("Parsing %s\n" % f)

And if you have it from future anyway...

print("Parsing", f, file=sys.stderr)

>  globs = {}
>  locs = OrderedDict()
> @@ -362,11 +362,10 @@ def parse(f):
>  globs[n] = t
>  
>  try:
> -execfile(f, globs, locs)
> -except SyntaxError,e:
> -raise SyntaxError, \
> -  "Errors were found at line %d while processing %s:\n\t%s"\
> -  %(e.lineno,f,e.text)
> +exec(compile(open(f).read(), f, 'exec'), globs, locs)
> +except SyntaxError as e:
> +raise SyntaxError("Errors were found at line %d while processing 
> %s:\n\t%s"\
> +  %(e.lineno,f,e.text))
>  
>  types = [t for t in locs.ordered_values() if isinstance(t,Type)]
>  
> 

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] python3 issues

[Hans van Kranenburg]

On 3/26/19 7:18 PM, M A Young wrote:
> On Tue, 26 Mar 2019, Wei Liu wrote:
> 
>> On Tue, Mar 26, 2019 at 01:16:35PM +, Wei Liu wrote:
>>> On Mon, Mar 25, 2019 at 10:20:05PM +, YOUNG, MICHAEL A. wrote:
  if ty.init_fn is not None:
 --- xen-4.12.0-rc6/tools/pygrub/src/GrubConf.py.orig   2019-03-24 
 22:44:05.502581989 +
 +++ xen-4.12.0-rc6/tools/pygrub/src/GrubConf.py2019-03-24 
 22:49:14.025934786 +
 @@ -230,10 +230,10 @@
  def _get_default(self):
  return self._default
  def _set_default(self, val):
 -if val == "saved":
 +if val == "saved" or not val.isdecimal():
>>
>> Your change suggested there could be a non-decimal string that is not
>> "saved" -- is this really needed?
> 
> It is getting set to ${next_entry} presumably from the clause 
> 
> if [ "${next_entry}" ] ; then
>set default="${next_entry}"
>set next_entry=
>save_env next_entry
>set boot_once=true
> else
>set default="${saved_entry}"
> fi
> 
> in the grub.cfg file giving the error
> 
>   File "/usr/lib64/python3.7/site-packages/grub/GrubConf.py", line 239, in 
> _set_default
> if self._default < 0:
> TypeError: '<' not supported between instances of 'str' and 'int'
> 
> I didn't see this with python 2 before the patch so I assume python3 is 
> more fussy.

Comparison is also used for sorting.

In Python 2, numeric values always sort before strings. So, for example:

== sorted(['a','b','c', 1, 2, 3])
[1, 2, 3, 'a', 'b', 'c']

However, this behavior also easily leads to bugs, for example when
someone forgets to convert strings that hold numeric values to actual
numbers and still compares things, which silently introduces unintended
behavior.

So, the above if self._default < 0 will just always be False if
self._default is a string. I didn't read the context of these lines, but
this looks like an actual bug currently.

Python 3 no longer allows comparing string and int, because it doesn't
make sense.

== sorted([1,2,3,'a','b','c'])
Traceback (most recent call last):
  File "", line 1, in 
TypeError: unorderable types: str() < int()

So I guess that if the contents are expected to be "saved" or a number
as string like "1", "2", then:

1. check for "saved"
2. try: int(val) except: blah
3. etc

Also, python 2 does not have isdecimal() on strings.

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] python3 issues

[Hans van Kranenburg]

On 3/26/19 2:16 PM, Wei Liu wrote:
> On Mon, Mar 25, 2019 at 10:20:05PM +, YOUNG, MICHAEL A. wrote:
>> [...]
>> --- xen-4.12.0-rc6/tools/pygrub/src/pygrub.orig  2019-03-24 
>> 22:44:05.503582025 +
>> +++ xen-4.12.0-rc6/tools/pygrub/src/pygrub   2019-03-24 22:48:24.446113809 
>> +
>> @@ -457,7 +457,7 @@
>>  # limit read size to avoid pathological cases
>>  buf = f.read(FS_READ_MAX)
>>  del f
>> -self.cf.parse(buf)
>> +self.cf.parse(buf.decode())
> 
> Hmm... This could be a bit problematic for 2 compatibility. I will need
> some time to check the documents.

The exact moment when you're typing .decode() or .encode() to try fix a
python2/3 problem, you're entering a world of pain (tm).

[insert big lebowski walter pointing gun gif here]

Some of the python 2 and 3 compatibility things can be fixed easily by
importing things from the future, but when dealing with strings and
bytes, you're entering the "is it really worth it" department.

In python 2, strings are bytes, and there's some added duct tape
available to do something with unicode and utf-8.

In python 3, strings are unicode and bytes are bytes, which is sooo
much more convenient. If the strings have to be stored in some location
that stores bytes, you can encode them. If you have some incoming byte
stream, you can decode it.

I don't really have recommendations to make both of them work with the
same lines of code since I tried and gave up when doing it myself.

What you're looking at is how the in-memory storage for a 'string' is
done and how to connect input and output to that, while looking at
python 2 and 3 functions with the same name, doing different things in
a different context. There is no sane way to do this that works with
python 2 and 3.

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH for-4.13 2/2] Rationalize max_grant_frames and max_maptrack_frames handling

[Hans van Kranenburg]

Hi all,

On 11/27/19 12:13 PM, Durrant, Paul wrote:
>> -Original Message-
>> From: Ian Jackson 
>> Sent: 27 November 2019 11:10
>> [...]
>> Subject: RE: [Xen-devel] [PATCH for-4.13 2/2] Rationalize max_grant_frames
>> and max_maptrack_frames handling
>>
>> Durrant, Paul writes ("RE: [Xen-devel] [PATCH for-4.13 2/2] Rationalize
>> max_grant_frames and max_maptrack_frames handling"):
 -Original Message-
 From: Xen-devel  On Behalf Of
>> Ian
 Jackson
 I have seen reports of users who ran out of grant/maptrack frames
 because of updates to use multiring protocols etc.  The error messages
 are not very good and the recommended workaround has been to increase
 the default limit on the hypervisor command line.

 It is important that we don't break that workaround!
>>>
>>> Alas it has apparently been broken for several releases now :-(
>>
>> I guess at least in Debian (where I have seen this) we haven't
>> released with any affected versions yet...
> 
> I believe the problem was introduce in 4.10, so I think it would be prudent 
> to also back-port the final fix to stable trees from then on.

Yes, the max grant frame issue has historically always been a painful
experience for end users, and Xen 4.11 which we now have in the current
Debian stable has made it worse compared to previous versions indeed.

Changing the hypervisor command line worked in the past, and now that
value is overwritten again by a lower value in the toolstack, which
requires setting per-domU settings, or, what I did, just additionally
also setting max_grant_frames in /etc/xen/xl.conf to the same value as
the hypervisor command line.

This change is very welcome, even to 4.11-stable if possible, since it
will not break existing configuration of users.

If changing only the value of the hypervisor command line works again,
then old information that shows up when the users searches the web will
be useful again, which is good.

Hans

P.S. Now I'm curious to figure out what a maptrack frame is, didn't hear
about that one before.

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH for-4.13 2/2] Rationalize max_grant_frames and max_maptrack_frames handling

[Hans van Kranenburg]

On 11/28/19 3:54 PM, George Dunlap wrote:
> On 11/27/19 10:32 PM, Hans van Kranenburg wrote:
>> Hi all,
>>
>> On 11/27/19 12:13 PM, Durrant, Paul wrote:
>>>> -Original Message-
>>>> From: Ian Jackson 
>>>> Sent: 27 November 2019 11:10
>>>> [...]
>>>> Subject: RE: [Xen-devel] [PATCH for-4.13 2/2] Rationalize max_grant_frames
>>>> and max_maptrack_frames handling
>>>>
>>>> Durrant, Paul writes ("RE: [Xen-devel] [PATCH for-4.13 2/2] Rationalize
>>>> max_grant_frames and max_maptrack_frames handling"):
>>>>>> -Original Message-
>>>>>> From: Xen-devel  On Behalf Of
>>>> Ian
>>>>>> Jackson
>>>>>> I have seen reports of users who ran out of grant/maptrack frames
>>>>>> because of updates to use multiring protocols etc.  The error messages
>>>>>> are not very good and the recommended workaround has been to increase
>>>>>> the default limit on the hypervisor command line.
>>>>>>
>>>>>> It is important that we don't break that workaround!
>>>>>
>>>>> Alas it has apparently been broken for several releases now :-(
>>>>
>>>> I guess at least in Debian (where I have seen this) we haven't
>>>> released with any affected versions yet...
>>>
>>> I believe the problem was introduce in 4.10, so I think it would be prudent 
>>> to also back-port the final fix to stable trees from then on.
>>
>> Yes, the max grant frame issue has historically always been a painful
>> experience for end users, and Xen 4.11 which we now have in the current
>> Debian stable has made it worse compared to previous versions indeed.
> 
> This rather suggests that the default value isn't very well chosen.
> Ideally some investigation would be done to improve the default sizing;
> end-users shouldn't have to know anything about grant table frames.

Most of the problems started happening a few years ago when using a
newer Linux that got all kinds of multiqueue block stuff for disk and
network enabled on top of an older Xen. (e.g. in Debian using the Linux
4.9 backports kernel on top of Xen 4.4 in Jessie).

The default for the hypervisor option has already been doubled from 32
to 64, which I think is sufficient. However, having the toolstack revert
it back to 32 again is not very helpful, but that's what this thread is
about to solve. :)

A while ago I did some testing:
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=880554#119

I haven't been able to cause nr_frames to go over 64 in any test myself,
and also have never seen values that high in production use. The above
debian bug also does not contain any other report from anyone with a
number above 64. There are reports of users setting it to 256 and then
not caring about it any more, but they didn't report the xen_diag output
back after that, so there's no real data.

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v6 16/20] grub-module-verifier: Ignore all_video for xenpvh

[Hans van Kranenburg]

On 11/28/18 2:55 PM, Juergen Gross wrote:
> From: Hans van Kranenburg 
> 
> This solves the build failing with "Error: no symbol table and no
> .moddeps section"
> 
> Also see:
> - 6371e9c10433578bb236a8284ddb9ce9e201eb59
> - https://savannah.gnu.org/bugs/?49012
> 
> Signed-off-by: Hans van Kranenburg 
> Reviewed-by: Daniel Kiper 

Just a small detail... The xenpvh in the subject was not renamed to
xen_pvh. That can probably be fixed up while committing. :)

> ---
> V2: new patch
> Signed-off-by: Juergen Gross 
> ---
>  util/grub-module-verifier.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/util/grub-module-verifier.c b/util/grub-module-verifier.c
> index 03ba1ab43..97cb9 100644
> --- a/util/grub-module-verifier.c
> +++ b/util/grub-module-verifier.c
> @@ -129,6 +129,7 @@ struct platform_whitelist {
>  
>  static struct platform_whitelist whitelists[] = {
>{"i386", "xen", (const char *[]) {"all_video", 0}},
> +  {"i386", "xen_pvh", (const char *[]) {"all_video", 0}},
>{"x86_64", "xen", (const char *[]) {"all_video", 0}},
>{"sparc64", "ieee1275", (const char *[]) {"all_video", 0}},
>  
> 

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 0/2] Move RSDP address out of setup_header

[Hans van Kranenburg]

Hi,

On 11/20/18 8:25 AM, Juergen Gross wrote:
> Resend with Daniel's mailo address corrected
> 
> Instead of passing the RSDP address for Xen PVH guests from grub2 to
> the kernel in setup_header move it into the non-legacy part of the
> boot_params structure.
> 
> This patch series should be applied rather sooner than later in order
> to avoid shipping linux 4.20 with a corky boot protocol.
> 
> Juergen Gross (2):
>   x86/boot: revert commit ae7e1238e68f2a (Add ACPI RSDP address to
> setup_header)
>   x86/acpi, x86/boot: Take RSDP address from boot params if available
> 
>  Documentation/x86/boot.txt| 32 +---
>  arch/x86/boot/header.S|  6 +-
>  arch/x86/include/asm/x86_init.h   |  2 --
>  arch/x86/include/uapi/asm/bootparam.h |  7 ++-
>  arch/x86/kernel/acpi/boot.c   |  2 +-
>  arch/x86/kernel/head32.c  |  1 -
>  arch/x86/kernel/head64.c  |  2 --
>  arch/x86/kernel/setup.c   | 17 -
>  8 files changed, 5 insertions(+), 64 deletions(-)

FWIW, I can confirm that I can boot a Xen domU in PVH mode with the
following four patches added on top of Linux 4.19.5 (applied in this
order), in combination with grub master branch with patchset v5 for grub
on top:

Date:   Wed Oct 10 08:14:55 2018 +0200
x86/boot: Add ACPI RSDP address to setup_header

Date:   Wed Oct 10 08:14:56 2018 +0200
x86/acpi, x86/boot: Take RSDP address for boot params if available

Date:   Tue Nov 20 08:25:28 2018 +0100
x86/boot: revert commit ae7e1238e68f2a (Add ACPI RSDP address to
setup_header)

Date:   Tue Nov 20 08:25:29 2018 +0100
x86/acpi, x86/boot: Take RSDP address from boot params if available

[0.205884] Booting paravirtualized kernel on Xen PVH

Thanks,
Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] Linux 4.19.5 fails to boot as Xen dom0

[Hans van Kranenburg]

Hi,

As also seen at:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=914951

Attached there are two serial console output logs. One is starting with
Xen 4.11 (from debian unstable) as dom0, and the other one without Xen.

[2.085543] BUG: unable to handle kernel paging request at
888d9fffc000
[2.085610] PGD 200c067 P4D 200c067 PUD 0
[2.085674] Oops:  [#1] SMP NOPTI
[2.085736] CPU: 1 PID: 1 Comm: swapper/0 Not tainted
4.19.0-trunk-amd64 #1 Debian 4.19.5-1~exp1+pvh1
[2.085823] Hardware name: HP ProLiant DL360 G7, BIOS P68 05/21/2018
[2.085895] RIP: e030:ptdump_walk_pgd_level_core+0x1fd/0x490
[...]

The pti=off setting on the PV dom0 kernel is left behind from the time
when 4.9 failed to boot as Xen dom0 because of the bug handling that.

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v6 00/20] xen: add pvh guest support

[Hans van Kranenburg]

Hi Daniel,

On 11/29/18 1:22 PM, Daniel Kiper wrote:
> On Wed, Nov 28, 2018 at 02:55:10PM +0100, Juergen Gross wrote:
>> This patch series adds support for booting Linux as PVH guest.
>>
>> Similar to i386/xen and x86_64/xen platforms the new i386/xenpvh
>> platform grub is booted as a standalone image directly by Xen.
>>
>> For booting Linux kernel it is using the standard linux kernel
>> loader. The only modification of the linux loader is to pass the
>> ACPI RSDP address via boot parameters to the kernel, as that table
>> might not be located at the usual physical address just below 1MB.
>>
>> The related Linux kernel patches have been accepted for 4.20-rc4
> 
> Patch set LGTM. If there are no objections I will apply it in a week
> or so.
> 
> Hans, may I add "Tested-by: Hans van Kranenburg " to
> the patches?

Sure!

> Juergen, thank you for doing the work.

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Linux 4.19.5 fails to boot as Xen dom0

[Hans van Kranenburg]

On 11/29/18 4:06 PM, Kirill A. Shutemov wrote:
> On Thu, Nov 29, 2018 at 03:00:45PM +, Juergen Gross wrote:
>> On 29/11/2018 15:32, Kirill A. Shutemov wrote:
>>> On Thu, Nov 29, 2018 at 02:24:47PM +, Kirill A. Shutemov wrote:
>>>> On Thu, Nov 29, 2018 at 01:35:17PM +, Juergen Gross wrote:
>>>>> On 29/11/2018 14:26, Kirill A. Shutemov wrote:
>>>>>> On Thu, Nov 29, 2018 at 09:41:25AM +0000, Juergen Gross wrote:
>>>>>>> On 29/11/2018 02:22, Hans van Kranenburg wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> As also seen at:
>>>>>>>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=914951
>>>>>>>>
>>>>>>>> Attached there are two serial console output logs. One is starting with
>>>>>>>> Xen 4.11 (from debian unstable) as dom0, and the other one without Xen.
>>>>>>>>
>>>>>>>> [2.085543] BUG: unable to handle kernel paging request at
>>>>>>>> 888d9fffc000
>>>>>>>> [2.085610] PGD 200c067 P4D 200c067 PUD 0
>>>>>>>> [2.085674] Oops:  [#1] SMP NOPTI
>>>>>>>> [2.085736] CPU: 1 PID: 1 Comm: swapper/0 Not tainted
>>>>>>>> 4.19.0-trunk-amd64 #1 Debian 4.19.5-1~exp1+pvh1
>>>>>>>> [2.085823] Hardware name: HP ProLiant DL360 G7, BIOS P68 05/21/2018
>>>>>>>> [2.085895] RIP: e030:ptdump_walk_pgd_level_core+0x1fd/0x490
>>>>>>>> [...]
>>>>>>>
>>>>>>> The offending stable commit is 4074ca7d8a1832921c865d250bbd08f3441b3657
>>>>>>> ("x86/mm: Move LDT remap out of KASLR region on 5-level paging"), this
>>>>>>> is commit d52888aa2753e3063a9d3a0c9f72f94aa9809c15 upstream.
>>>>>>>
>>>>>>> Current upstream kernel is booting fine under Xen, so in general the
>>>>>>> patch should be fine. Using an upstream kernel built from above commit
>>>>>>> (with the then needed Xen fixup patch 1457d8cf7664f34c4ba534) is fine,
>>>>>>> too.
>>>>>>>
>>>>>>> Kirill, are you aware of any prerequisite patch from 4.20 which could be
>>>>>>> missing in 4.19.5?
>>>>>>
>>>>>> I'm not.
>>>>>>
>>>>>> Let me look into this.
>>>>>>
>>>>>
>>>>> What is making me suspicious is the failure happening just after
>>>>> releasing the init memory. Maybe there is an access to .init.data
>>>>> segment or similar? The native kernel booting could be related to the
>>>>> usage of 2M mappings not being available in a PV-domain.
>>>>
>>>> Ahh.. Could you test this:
>>>>
>>>> diff --git a/arch/x86/mm/dump_pagetables.c b/arch/x86/mm/dump_pagetables.c
>>>> index a12afff146d1..7dec63ec7aab 100644
>>>> --- a/arch/x86/mm/dump_pagetables.c
>>>> +++ b/arch/x86/mm/dump_pagetables.c
>>>> @@ -496,7 +496,7 @@ static inline bool is_hypervisor_range(int idx)
>>>> * 8000 - 87ff is reserved for
>>>> * the hypervisor.
>>>> */
>>>> -  return  (idx >= pgd_index(__PAGE_OFFSET) - 16) &&
>>>> +  return  (idx >= pgd_index(__PAGE_OFFSET) - 17) &&
>>>>(idx <  pgd_index(__PAGE_OFFSET));
>>>>  #else
>>>>return false;
>>>
>>> Or, better, this:
>>
>> That makes it boot again!
>>
>> Any idea why upstream doesn't need it?
> 
> Nope.
> 
> I'll prepare a proper fix.
> 

Thanks for looking into this.

In the meantime, I applied the "Or, better, this" change, and my dom0
boots again.

FYI, boot log now: (paste 90d valid)
https://paste.debian.net/plainh/48940826

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Linux 4.19.5 fails to boot as Xen dom0

[Hans van Kranenburg]

On 11/30/18 2:26 PM, Kirill A. Shutemov wrote:
> On Fri, Nov 30, 2018 at 01:11:56PM +0000, Hans van Kranenburg wrote:
>> On 11/29/18 4:06 PM, Kirill A. Shutemov wrote:
>>> On Thu, Nov 29, 2018 at 03:00:45PM +, Juergen Gross wrote:
>>>> On 29/11/2018 15:32, Kirill A. Shutemov wrote:
>>>>> On Thu, Nov 29, 2018 at 02:24:47PM +, Kirill A. Shutemov wrote:
>>>>>> On Thu, Nov 29, 2018 at 01:35:17PM +, Juergen Gross wrote:
>>>>>>> On 29/11/2018 14:26, Kirill A. Shutemov wrote:
>>>>>>>> On Thu, Nov 29, 2018 at 09:41:25AM +, Juergen Gross wrote:
>>>>>>>>> On 29/11/2018 02:22, Hans van Kranenburg wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> As also seen at:
>>>>>>>>>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=914951
>>>>>>>>>>
>>>>>>>>>> Attached there are two serial console output logs. One is starting 
>>>>>>>>>> with
>>>>>>>>>> Xen 4.11 (from debian unstable) as dom0, and the other one without 
>>>>>>>>>> Xen.
>>>>>>>>>>
>>>>>>>>>> [2.085543] BUG: unable to handle kernel paging request at
>>>>>>>>>> 888d9fffc000
>>>>>>>>>> [2.085610] PGD 200c067 P4D 200c067 PUD 0
>>>>>>>>>> [2.085674] Oops:  [#1] SMP NOPTI
>>>>>>>>>> [2.085736] CPU: 1 PID: 1 Comm: swapper/0 Not tainted
>>>>>>>>>> 4.19.0-trunk-amd64 #1 Debian 4.19.5-1~exp1+pvh1
>>>>>>>>>> [2.085823] Hardware name: HP ProLiant DL360 G7, BIOS P68 
>>>>>>>>>> 05/21/2018
>>>>>>>>>> [2.085895] RIP: e030:ptdump_walk_pgd_level_core+0x1fd/0x490
>>>>>>>>>> [...]
>>>>>>>>>
>>>>>>>>> The offending stable commit is 
>>>>>>>>> 4074ca7d8a1832921c865d250bbd08f3441b3657
>>>>>>>>> ("x86/mm: Move LDT remap out of KASLR region on 5-level paging"), this
>>>>>>>>> is commit d52888aa2753e3063a9d3a0c9f72f94aa9809c15 upstream.
>>>>>>>>>
>>>>>>>>> Current upstream kernel is booting fine under Xen, so in general the
>>>>>>>>> patch should be fine. Using an upstream kernel built from above commit
>>>>>>>>> (with the then needed Xen fixup patch 1457d8cf7664f34c4ba534) is fine,
>>>>>>>>> too.
>>>>>>>>>
>>>>>>>>> Kirill, are you aware of any prerequisite patch from 4.20 which could 
>>>>>>>>> be
>>>>>>>>> missing in 4.19.5?
>>>>>>>>
>>>>>>>> I'm not.
>>>>>>>>
>>>>>>>> Let me look into this.
>>>>>>>>
>>>>>>>
>>>>>>> What is making me suspicious is the failure happening just after
>>>>>>> releasing the init memory. Maybe there is an access to .init.data
>>>>>>> segment or similar? The native kernel booting could be related to the
>>>>>>> usage of 2M mappings not being available in a PV-domain.
>>>>>>
>>>>>> Ahh.. Could you test this:
>>>>>>
>>>>>> diff --git a/arch/x86/mm/dump_pagetables.c 
>>>>>> b/arch/x86/mm/dump_pagetables.c
>>>>>> index a12afff146d1..7dec63ec7aab 100644
>>>>>> --- a/arch/x86/mm/dump_pagetables.c
>>>>>> +++ b/arch/x86/mm/dump_pagetables.c
>>>>>> @@ -496,7 +496,7 @@ static inline bool is_hypervisor_range(int idx)
>>>>>>   * 8000 - 87ff is reserved for
>>>>>>   * the hypervisor.
>>>>>>   */
>>>>>> -return  (idx >= pgd_index(__PAGE_OFFSET) - 16) &&
>>>>>> +return  (idx >= pgd_index(__PAGE_OFFSET) - 17) &&
>>>>>>  (idx <  pgd_index(__PAGE_OFFSET));
>>>>>>  #else
>>>>>>  return false;
>>>>>
>>>>> Or, better, this:
>>>>
>>>> That makes it boot again!
>>>>
>>>> Any idea why upstream doesn't need it?
>>>
>>> Nope.
>>>
>>> I'll prepare a proper fix.
>>>
>>
>> Thanks for looking into this.
>>
>> In the meantime, I applied the "Or, better, this" change, and my dom0
>> boots again.
>>
>> FYI, boot log now: (paste 90d valid)
>> https://paste.debian.net/plainh/48940826
> 
> I forgot to CC you:
> 
> https://lkml.kernel.org/r/20181130121131.g3xvlvixv7mvl...@black.fi.intel.com
> 
> Please give it a try.

Ah, right, thanks. The xen-devel list is also not in Cc.

I'll slam it on top of my 4.19.5 debian package build and test.

Hans


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Linux 4.19.5 fails to boot as Xen dom0

[Hans van Kranenburg]

On 11/30/18 2:26 PM, Kirill A. Shutemov wrote:
> On Fri, Nov 30, 2018 at 01:11:56PM +0000, Hans van Kranenburg wrote:
>> On 11/29/18 4:06 PM, Kirill A. Shutemov wrote:
>>> On Thu, Nov 29, 2018 at 03:00:45PM +, Juergen Gross wrote:
>>>> On 29/11/2018 15:32, Kirill A. Shutemov wrote:
>>>>> On Thu, Nov 29, 2018 at 02:24:47PM +, Kirill A. Shutemov wrote:
>>>>>> On Thu, Nov 29, 2018 at 01:35:17PM +, Juergen Gross wrote:
>>>>>>> On 29/11/2018 14:26, Kirill A. Shutemov wrote:
>>>>>>>> On Thu, Nov 29, 2018 at 09:41:25AM +, Juergen Gross wrote:
>>>>>>>>> On 29/11/2018 02:22, Hans van Kranenburg wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> As also seen at:
>>>>>>>>>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=914951
>>>>>>>>>>
>>>>>>>>>> Attached there are two serial console output logs. One is starting 
>>>>>>>>>> with
>>>>>>>>>> Xen 4.11 (from debian unstable) as dom0, and the other one without 
>>>>>>>>>> Xen.
>>>>>>>>>>
>>>>>>>>>> [2.085543] BUG: unable to handle kernel paging request at
>>>>>>>>>> 888d9fffc000
>>>>>>>>>> [2.085610] PGD 200c067 P4D 200c067 PUD 0
>>>>>>>>>> [2.085674] Oops:  [#1] SMP NOPTI
>>>>>>>>>> [2.085736] CPU: 1 PID: 1 Comm: swapper/0 Not tainted
>>>>>>>>>> 4.19.0-trunk-amd64 #1 Debian 4.19.5-1~exp1+pvh1
>>>>>>>>>> [2.085823] Hardware name: HP ProLiant DL360 G7, BIOS P68 
>>>>>>>>>> 05/21/2018
>>>>>>>>>> [2.085895] RIP: e030:ptdump_walk_pgd_level_core+0x1fd/0x490
>>>>>>>>>> [...]
>>>>>>>>>
>>>>>>>>> The offending stable commit is 
>>>>>>>>> 4074ca7d8a1832921c865d250bbd08f3441b3657
>>>>>>>>> ("x86/mm: Move LDT remap out of KASLR region on 5-level paging"), this
>>>>>>>>> is commit d52888aa2753e3063a9d3a0c9f72f94aa9809c15 upstream.
>>>>>>>>>
>>>>>>>>> Current upstream kernel is booting fine under Xen, so in general the
>>>>>>>>> patch should be fine. Using an upstream kernel built from above commit
>>>>>>>>> (with the then needed Xen fixup patch 1457d8cf7664f34c4ba534) is fine,
>>>>>>>>> too.
>>>>>>>>>
>>>>>>>>> Kirill, are you aware of any prerequisite patch from 4.20 which could 
>>>>>>>>> be
>>>>>>>>> missing in 4.19.5?
>>>>>>>>
>>>>>>>> I'm not.
>>>>>>>>
>>>>>>>> Let me look into this.
>>>>>>>>
>>>>>>>
>>>>>>> What is making me suspicious is the failure happening just after
>>>>>>> releasing the init memory. Maybe there is an access to .init.data
>>>>>>> segment or similar? The native kernel booting could be related to the
>>>>>>> usage of 2M mappings not being available in a PV-domain.
>>>>>>
>>>>>> Ahh.. Could you test this:
>>>>>>
>>>>>> diff --git a/arch/x86/mm/dump_pagetables.c 
>>>>>> b/arch/x86/mm/dump_pagetables.c
>>>>>> index a12afff146d1..7dec63ec7aab 100644
>>>>>> --- a/arch/x86/mm/dump_pagetables.c
>>>>>> +++ b/arch/x86/mm/dump_pagetables.c
>>>>>> @@ -496,7 +496,7 @@ static inline bool is_hypervisor_range(int idx)
>>>>>>   * 8000 - 87ff is reserved for
>>>>>>   * the hypervisor.
>>>>>>   */
>>>>>> -return  (idx >= pgd_index(__PAGE_OFFSET) - 16) &&
>>>>>> +return  (idx >= pgd_index(__PAGE_OFFSET) - 17) &&
>>>>>>  (idx <  pgd_index(__PAGE_OFFSET));
>>>>>>  #else
>>>>>>  return false;
>>>>>
>>>>> Or, better, this:
>>>>
>>>> That makes it boot again!
>>>>
>>>> Any idea why upstream doesn't need it?
>>>
>>> Nope.
>>>
>>> I'll prepare a proper fix.
>>>
>>
>> Thanks for looking into this.
>>
>> In the meantime, I applied the "Or, better, this" change, and my dom0
>> boots again.
>>
>> FYI, boot log now: (paste 90d valid)
>> https://paste.debian.net/plainh/48940826
> 
> I forgot to CC you:
> 
> https://lkml.kernel.org/r/20181130121131.g3xvlvixv7mvl...@black.fi.intel.com
> 
> Please give it a try.

I'm not in that thread, so my response here...

You paste a v2-like patch into 'Re: [PATCH 1/2]'. Juergen says:
s/LDT_PGD_ENTRY/GUARD_HOLE_PGD_ENTRY/, then you say Ughh.., change it to
GUARD_HOLE_ENTRY, which does not exist, and then get a Reviewed-by from
Juergen.

I guess it has to be GUARD_HOLE_PGD_ENTRY after all...

arch/x86/include/asm/pgtable_64_types.h:116:31: error:
'GUARD_HOLE_ENTRY' undeclared (first use in this function); did you mean
'GUARD_HOLE_PGD_ENTRY'?

I'll test that instead.

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Linux 4.19.5 fails to boot as Xen dom0

[Hans van Kranenburg]

On 11/30/18 5:21 PM, Kirill A. Shutemov wrote:
> On Fri, Nov 30, 2018 at 02:53:50PM +0000, Hans van Kranenburg wrote:
>> On 11/30/18 2:26 PM, Kirill A. Shutemov wrote:
>>> On Fri, Nov 30, 2018 at 01:11:56PM +, Hans van Kranenburg wrote:
>>>> On 11/29/18 4:06 PM, Kirill A. Shutemov wrote:
>>>>> On Thu, Nov 29, 2018 at 03:00:45PM +, Juergen Gross wrote:
>>>>>> On 29/11/2018 15:32, Kirill A. Shutemov wrote:
>>>>>>> On Thu, Nov 29, 2018 at 02:24:47PM +, Kirill A. Shutemov wrote:
>>>>>>>> On Thu, Nov 29, 2018 at 01:35:17PM +, Juergen Gross wrote:
>>>>>>>>> On 29/11/2018 14:26, Kirill A. Shutemov wrote:
>>>>>>>>>> On Thu, Nov 29, 2018 at 09:41:25AM +, Juergen Gross wrote:
>>>>>>>>>>> On 29/11/2018 02:22, Hans van Kranenburg wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> As also seen at:
>>>>>>>>>>>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=914951
>>>>>>>>>>>>
>>>>>>>>>>>> Attached there are two serial console output logs. One is starting 
>>>>>>>>>>>> with
>>>>>>>>>>>> Xen 4.11 (from debian unstable) as dom0, and the other one without 
>>>>>>>>>>>> Xen.
>>>>>>>>>>>>
>>>>>>>>>>>> [2.085543] BUG: unable to handle kernel paging request at
>>>>>>>>>>>> 888d9fffc000
>>>>>>>>>>>> [2.085610] PGD 200c067 P4D 200c067 PUD 0
>>>>>>>>>>>> [2.085674] Oops:  [#1] SMP NOPTI
>>>>>>>>>>>> [2.085736] CPU: 1 PID: 1 Comm: swapper/0 Not tainted
>>>>>>>>>>>> 4.19.0-trunk-amd64 #1 Debian 4.19.5-1~exp1+pvh1
>>>>>>>>>>>> [2.085823] Hardware name: HP ProLiant DL360 G7, BIOS P68 
>>>>>>>>>>>> 05/21/2018
>>>>>>>>>>>> [2.085895] RIP: e030:ptdump_walk_pgd_level_core+0x1fd/0x490
>>>>>>>>>>>> [...]
>>>>>>>>>>>
>>>>>>>>>>> The offending stable commit is 
>>>>>>>>>>> 4074ca7d8a1832921c865d250bbd08f3441b3657
>>>>>>>>>>> ("x86/mm: Move LDT remap out of KASLR region on 5-level paging"), 
>>>>>>>>>>> this
>>>>>>>>>>> is commit d52888aa2753e3063a9d3a0c9f72f94aa9809c15 upstream.
>>>>>>>>>>>
>>>>>>>>>>> Current upstream kernel is booting fine under Xen, so in general the
>>>>>>>>>>> patch should be fine. Using an upstream kernel built from above 
>>>>>>>>>>> commit
>>>>>>>>>>> (with the then needed Xen fixup patch 1457d8cf7664f34c4ba534) is 
>>>>>>>>>>> fine,
>>>>>>>>>>> too.
>>>>>>>>>>>
>>>>>>>>>>> Kirill, are you aware of any prerequisite patch from 4.20 which 
>>>>>>>>>>> could be
>>>>>>>>>>> missing in 4.19.5?
>>>>>>>>>>
>>>>>>>>>> I'm not.
>>>>>>>>>>
>>>>>>>>>> Let me look into this.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> What is making me suspicious is the failure happening just after
>>>>>>>>> releasing the init memory. Maybe there is an access to .init.data
>>>>>>>>> segment or similar? The native kernel booting could be related to the
>>>>>>>>> usage of 2M mappings not being available in a PV-domain.
>>>>>>>>
>>>>>>>> Ahh.. Could you test this:
>>>>>>>>
>>>>>>>> diff --git a/arch/x86/mm/dump_pagetables.c 
>>>>>>>> b/arch/x86/mm/dump_pagetables.c
>>>>>>>> index a12afff146d1..7dec63ec7aab 100644
>>>>>>>> --- a/arch/x86/mm/dump_pagetables.c
>>>>>>>> +++ 

Re: [Xen-devel] Linux 4.19.5 fails to boot as Xen dom0

[Hans van Kranenburg]

On 12/6/18 12:31 PM, Juergen Gross wrote:
> On 06/12/2018 12:13, Greg KH wrote:
>> On Thu, Nov 29, 2018 at 02:35:17PM +0100, Juergen Gross wrote:
>>> On 29/11/2018 14:26, Kirill A. Shutemov wrote:
>>>> On Thu, Nov 29, 2018 at 09:41:25AM +, Juergen Gross wrote:
>>>>> On 29/11/2018 02:22, Hans van Kranenburg wrote:
>>>>>> Hi,
>>>>>>
>>>>>> As also seen at:
>>>>>> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=914951
>>>>>>
>>>>>> Attached there are two serial console output logs. One is starting with
>>>>>> Xen 4.11 (from debian unstable) as dom0, and the other one without Xen.
>>>>>>
>>>>>> [2.085543] BUG: unable to handle kernel paging request at
>>>>>> 888d9fffc000
>>>>>> [2.085610] PGD 200c067 P4D 200c067 PUD 0
>>>>>> [2.085674] Oops:  [#1] SMP NOPTI
>>>>>> [2.085736] CPU: 1 PID: 1 Comm: swapper/0 Not tainted
>>>>>> 4.19.0-trunk-amd64 #1 Debian 4.19.5-1~exp1+pvh1
>>>>>> [2.085823] Hardware name: HP ProLiant DL360 G7, BIOS P68 05/21/2018
>>>>>> [2.085895] RIP: e030:ptdump_walk_pgd_level_core+0x1fd/0x490
>>>>>> [...]
>>>>>
>>>>> The offending stable commit is 4074ca7d8a1832921c865d250bbd08f3441b3657
>>>>> ("x86/mm: Move LDT remap out of KASLR region on 5-level paging"), this
>>>>> is commit d52888aa2753e3063a9d3a0c9f72f94aa9809c15 upstream.
>>>>>
>>>>> Current upstream kernel is booting fine under Xen, so in general the
>>>>> patch should be fine. Using an upstream kernel built from above commit
>>>>> (with the then needed Xen fixup patch 1457d8cf7664f34c4ba534) is fine,
>>>>> too.
>>>>>
>>>>> Kirill, are you aware of any prerequisite patch from 4.20 which could be
>>>>> missing in 4.19.5?
>>>>
>>>> I'm not.
>>>>
>>>> Let me look into this.
>>>>
>>>
>>> What is making me suspicious is the failure happening just after
>>> releasing the init memory. Maybe there is an access to .init.data
>>> segment or similar? The native kernel booting could be related to the
>>> usage of 2M mappings not being available in a PV-domain.
>>
>> Did this ever get fixed anywhere that I can properly backport it to the
>> 4.19.y tree?
> 
> https://lore.kernel.org/lkml/20181130202328.65359-2-kirill.shute...@linux.intel.com/
> 
> Still pending upstream. Just pinged tglx.

And FYI, I actually use this patch on top of 4.19.5 now. It just applies
and works.

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] Live migrate with Linux >= 4.13 domU causes kernel time jumps and TCP connection stalls.

[Hans van Kranenburg]

Hi,

We've been tracking down a live migration bug during the last three days
here at work, and here's what we found so far.

1. Xen version and dom0 linux kernel version don't matter.
2. DomU kernel is >= Linux 4.13.

When using live migrate to another dom0, this often happens:

[   37.511305] Freezing user space processes ... (elapsed 0.001 seconds)
done.
[   37.513316] OOM killer disabled.
[   37.513323] Freezing remaining freezable tasks ... (elapsed 0.001
seconds) done.
[   37.514837] suspending xenstore...
[   37.515142] xen:grant_table: Grant tables using version 1 layout
[18446744002.593711] OOM killer enabled.
[18446744002.593726] Restarting tasks ... done.
[18446744002.604527] Setting capacity to 6291456

As a side effect, all open TCP connections stall, because the timestamp
counters of packets sent to the outside world are affected:

https://syrinx.knorrie.org/~knorrie/tmp/tcp-stall.png

"The problem seems to occur after the domU is resumed. The first packet
(#90) has wrong timestamp value (far from the past), marked red in the
image. Green is the normal sequence of timestamps from the server
(domU), acknowledged by the client. Once client receives the packet from
the past, it attempts re-sending everything from the start. As the
timestamp never reaches normal value, the client goes crazy thinking
that the server has not received anything, keeping the loop on. But they
just exist in different ages."

--- >8 ---

Ad 1. We reproduced this on different kinds of HP dl360 g7/8/9 gear,
both with Xen 4.11 / Linux 4.19.9 dom0 kernel and with Xen 4.4 / Linux
3.16 as dom0 kernel.

Ad 2. This was narrowed down by just grabbing old debian kernel images
from https://snapshot.debian.org/binary/?cat=l and trying them.

OK   linux-image-4.12.0-2-amd64_4.12.13-1_amd64.deb
FAIL linux-image-4.13.0-rc5-amd64_4.13~rc5-1~exp1_amd64.deb
FAIL linux-image-4.13.0-trunk-amd64_4.13.1-1~exp1_amd64.deb
FAIL linux-image-4.13.0-1-amd64_4.13.4-1_amd64.deb
FAIL linux-image-4.13.0-1-amd64_4.13.13-1_amd64.deb
FAIL linux-image-4.14.0-3-amd64_4.14.17-1_amd64.deb
FAIL linux-image-4.15.0-3-amd64_4.15.17-1_amd64.deb
FAIL linux-image-4.16.0-2-amd64_4.16.16-2_amd64.deb
FAIL ... everything up to 4.19.9 here

So, there seems to be a change introduced in 4.13 that makes this
behaviour appear. We didn't start compiling old kernels yet to be able
to bisect it further.

--- >8 ---

For the rest of the info, I'm focussing on a test environment for
reproduction, which is 4x identical HP DL360G7, named sirius, gamma,
omega and flopsy.

It's running the 4.11 packages from Debian, rebuilt for Stretch:
4.11.1~pre.20180911.5acdd26fdc+dfsg-5~bpo9+1
https://salsa.debian.org/xen-team/debian-xen/commits/stretch-backports

Dom0 kernel is 4.19.9 from Debian, rebuilt for Stretch:
https://salsa.debian.org/knorrie-guest/linux/commits/debian/4.19.9-1_mxbp9+1

xen_commandline : placeholder dom0_max_vcpus=1-4 dom0_mem=4G,max:4G
com2=115200,8n1 console=com2,vga noreboot xpti=no-dom0,domu smt=off

vendor_id   : GenuineIntel
cpu family  : 6
model   : 44
model name  : Intel(R) Xeon(R) CPU   X5675  @ 3.07GHz
stepping: 2
microcode   : 0x1f
cpu MHz : 3066.727

--- >8 ---

There are some interesting additional patterns:

1. consistent success / failure paths.

After rebooting all 4 physical servers, starting a domU with 4.19 kernel
and then live migrating it, it might first time fail, or it might
succeed. However, from the first time it fails, the specific direction
of movement keeps showing the failure every single time this combination
is used. Same goes for successful live migrate. E.g.:

sirius -> flopsy OK
sirius -> gamma OK
flopsy -> gamma OK
flopsy -> sirius OK
gamma -> flopsy FAIL
gamma -> sirius FAIL
omega -> flopsy FAIL

After rebooting all of the servers again, and restarting the whole test
procedure, the combinations and results change, but are again consistent
as soon as we start live migrating and seeing results.

2. TCP connections only hang when opened while "timestamp value in dmesg
is low", followed with a "time is 18 gazillion" situation. When opening
a TCP connection to the domU while it's at 18 gazillion seconds uptime,
the TCP connection keeps working all the time after subsequent live
migrations, even when it jumps up and down, following the OK and FAIL paths.

3. Since this is related to time and clocks, the last thing today we
tried was, instead of using default settings, put "clocksource=tsc
tsc=stable:socket" on the xen command line and "clocksource=tsc" on the
domU linux kernel line. What we observed after doing this, is that the
failure happens less often, but still happens. Everything else applies.

--- >8 ---

Additional question:

It's 2018, should we have these "clocksource=tsc tsc=stable:socket" on
Xen and "clocksource=tsc" anyways now, for Xen 4.11 and Linux 4.19
domUs? All our hardware has 'TscInvariant = true'.

Related: h

Re: [Xen-devel] [Xen-users] Future of 32-bit PV support

[Hans van Kranenburg]

On 08/17/2018 06:09 AM, Juergen Gross wrote:
> On 17/08/18 00:33, Andy Smith wrote:
>> Hi Juergen,
>>
>> [...] 
>> If so, could the final removal of 32-bit PV in the Linux kernel be
>> held off until there is:
>>
>> 1) a kernel shipping in Debian stable, Ubuntu LTS and CentOS that
>>boots under PVH, and;
>>
>> 2) support in grub2 so I can build a grub image that boots under
>>PVH?
> 
> I think this is a reasonable request.
> 
>> If grub PVH support is not going to happen, what is the roadmap for
>> user-specified guest kernels under PVH?
> 
> I have a patch series lying around for grub2 PVH support. It requires
> some rework and another kernel enhancement. I'll try to resume work on
> the patches soon.

+1, Yes please!

Do I understand correctly ('another kernel enhancement') that you found
more things which need to be done than the 32-bit memory map limitation
issue at the grub side?

>> [...]
Thanks,
Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH] tools/xentop : replace use of deprecated vwprintw

[Hans van Kranenburg]

Hi,

On 07/19/2018 08:53 AM, Wei Liu wrote:
> On Wed, Jul 18, 2018 at 03:22:17PM -0700, Christopher Clark wrote:
>> gcc-8.1 complains:
>>
>> | xentop.c: In function 'print':
>> | xentop.c:304:4: error: 'vwprintw' is deprecated 
>> [-Werror=deprecated-declarations]
>> | vwprintw(stdscr, (curses_str_t)fmt, args);
>> | ^~~~
>>
>> vw_printw (note the underscore) is a non-deprecated alternative.
>>
>> Signed-off-by: Christopher Clark 
> 
> Acked-by: Wei Liu 

Would it be possible to put this into 4.11-stable? It seems the change
is not that invasive and risky. :)

If so, I can drop a patch from the Debian packaging.

Thanks,
Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[Xen-devel] SMT/Hyperthreading detection not always correct

[Hans van Kranenburg]

Hi,

I ran into the situation where I got this banner in the xl dmesg...

(XEN) ***
(XEN) Booted on L1TF-vulnerable hardware with SMT/Hyperthreading
(XEN) enabled.  Please assess your configuration and choose an
(XEN) explicit 'smt=' setting.  See XSA-273.
(XEN) ***

...while hyperthreading is disabled in the bios settings.

Some lines from the POST messages:

ProLiant System BIOS - P68 (05/21/2018)
[...]
2 Processor(s) detected, 12 total cores enabled, Hyperthreading is disabled
Proc 1: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz
Proc 2: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz

More debug info below. See the thread '"Booted on L1TF-vulnerable
hardware with SMT/Hyperthreading enabled" .. or not?' in xen-users for
even more.

Reaction from Andrew on xen-users thread: "Looks like Xen does have a
bug identifying the topology.  Hyperthreading is definitely off, but
there is no MADT so information is probably being derived from the
legacy MP table.For now, set smt=0 if you want to skip the warning, and
can you email xen-devel with a bug report please? "

-# xl dmesg
(XEN) parameter "placeholder" unknown!
(XEN) Xen version 4.11.1-pre (Debian
4.11.1~pre+1.733450b39b-1~exp1~mxbp9+1) (h...@knorrie.org) (gcc (Debian
6.3.0-18+deb9u1) 6.3.0 20170516) debug=n  Wed Aug 22 15:26:44 UTC 2018
(XEN) Bootloader: GRUB 2.02~beta3-5
(XEN) Command line: placeholder dom0_max_vcpus=4 dom0_mem=4G,max:4G
com2=115200,8n1 console=com2,vga noreboot xpti=no-dom0 cpuinfo
dom0_vcpus_pin
(XEN) Xen image load base address: 0
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 2 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)   - 0009f400 (usable)
(XEN)  0009f400 - 000a (reserved)
(XEN)  000f - 0010 (reserved)
(XEN)  0010 - d762f000 (usable)
(XEN)  d762f000 - d763c000 (ACPI data)
(XEN)  d763c000 - d763d000 (usable)
(XEN)  d763d000 - dc00 (reserved)
(XEN)  fec0 - fee1 (reserved)
(XEN)  ff80 - 0001 (reserved)
(XEN)  0001 - 001b27fff000 (usable)
(XEN) New Xen image base address: 0xd700
(XEN) ACPI: RSDP 000F4F00, 0024 (r2 HP)
(XEN) ACPI: XSDT D7630140, 00B4 (r1 HP ProLiant2   � 162E)
(XEN) ACPI: FACP D7630240, 00F4 (r3 HP ProLiant2   � 162E)
(XEN) ACPI: DSDT D7630340, 20BD (r1 HP DSDT1 INTL 20030228)
(XEN) ACPI: FACS D762F100, 0040
(XEN) ACPI: SPCR D762F140, 0050 (r1 HP SPCRRBSU1   � 162E)
(XEN) ACPI: MCFG D762F1C0, 003C (r1 HP ProLiant1 0)
(XEN) ACPI: HPET D762F200, 0038 (r1 HP ProLiant2   � 162E)
(XEN) ACPI:  D762F240, 0064 (r2 HP ProLiant2   � 162E)
(XEN) ACPI: SPMI D762F2C0, 0040 (r5 HP ProLiant1   � 162E)
(XEN) ACPI: ERST D762F300, 01D0 (r1 HP ProLiant1   � 162E)
(XEN) ACPI: APIC D762F500, 015E (r1 HP ProLiant2 0)
(XEN) ACPI: SRAT D762F680, 0570 (r1 HP Proliant1   � 162E)
(XEN) ACPI:  D762FC00, 0176 (r1 HP ProLiant1   � 162E)
(XEN) ACPI: BERT D762FD80, 0030 (r1 HP ProLiant1   � 162E)
(XEN) ACPI: HEST D762FDC0, 00BC (r1 HP ProLiant1   � 162E)
(XEN) ACPI: DMAR D762FE80, 0146 (r1 HP ProLiant1   � 162E)
(XEN) ACPI: SSDT D7632400, 0125 (r3 HP  CRSPCI02   HP1)
(XEN) ACPI: SSDT D7632540, 01CF (r3 HP  riser1a2 INTL 20061109)
(XEN) ACPI: SSDT D7632740, 03BB (r1 HP  pcc1 INTL 20090625)
(XEN) ACPI: SSDT D7632B00, 0377 (r1 HP pmab1 INTL 20090625)
(XEN) ACPI: SSDT D7632E80, 2094 (r1  INTEL PPM RCM 1 INTL 20061109)
(XEN) System RAM: 110581MB (113235768kB)
(XEN) Domain heap initialised DMA width 32 bits
(XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec0, GSI 0-23
(XEN) IOAPIC[1]: apic_id 0, version 32, address 0xfec8, GSI 24-47
(XEN) Enabling APIC mode:  Phys.  Using 2 I/O APICs
(XEN) Failed to get Error Log Address Range.
(XEN) CPU: Physical Processor ID: 0
(XEN) CPU: Processor Core ID: 0
(XEN) CPU: L1 I cache: 32K, L1 D cache: 32K
(XEN) CPU: L2 cache: 256K
(XEN) CPU: L3 cache: 12288K
(XEN) Speculative mitigation facilities:
(XEN)   Hardware features: IBRS/IBPB STIBP L1D_FLUSH SSBD
(XEN)   Compiled-in support: INDIRECT_THUNK SHADOW_PAGING
(XEN)   Xen settings: BTI-Thunk RETPOLINE, SPEC_CTRL: IBRS- SSBD-,
Other: IBPB L1D_FLUSH
(XEN)   L1TF: believed vulnerable, maxphysaddr L1D 44, CPUID 40, Safe
address 100
(XEN)   Support for VMs: PV: MSR_SPEC_CTRL RSB EAGER_FPU, HVM:
MSR_SPEC_CTRL R

Re: [Xen-devel] SMT/Hyperthreading detection not always correct

[Hans van Kranenburg]

On 09/03/2018 03:13 PM, Jan Beulich wrote:
 On 03.09.18 at 13:55,  wrote:
>> I ran into the situation where I got this banner in the xl dmesg...
>>
>> (XEN) ***
>> (XEN) Booted on L1TF-vulnerable hardware with SMT/Hyperthreading
>> (XEN) enabled.  Please assess your configuration and choose an
>> (XEN) explicit 'smt=' setting.  See XSA-273.
>> (XEN) ***
>>
>> ...while hyperthreading is disabled in the bios settings.
>>
>> Some lines from the POST messages:
>>
>> ProLiant System BIOS - P68 (05/21/2018)
>> [...]
>> 2 Processor(s) detected, 12 total cores enabled, Hyperthreading is disabled
>> Proc 1: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz
>> Proc 2: Intel(R) Xeon(R) CPU X5675 @ 3.07GHz
>>
>> More debug info below. See the thread '"Booted on L1TF-vulnerable
>> hardware with SMT/Hyperthreading enabled" .. or not?' in xen-users for
>> even more.
>>
>> Reaction from Andrew on xen-users thread: "Looks like Xen does have a
>> bug identifying the topology.  Hyperthreading is definitely off, but
>> there is no MADT so information is probably being derived from the
>> legacy MP table.For now, set smt=0 if you want to skip the warning, and
>> can you email xen-devel with a bug report please? "
> 
> "There is no MADT information" could have been a hint to you that first
> of all you need ...
> 
>> -# xl dmesg
>> (XEN) parameter "placeholder" unknown!
>> (XEN) Xen version 4.11.1-pre (Debian
>> 4.11.1~pre+1.733450b39b-1~exp1~mxbp9+1) (h...@knorrie.org) (gcc (Debian
>> 6.3.0-18+deb9u1) 6.3.0 20170516) debug=n  Wed Aug 22 15:26:44 UTC 2018
>> (XEN) Bootloader: GRUB 2.02~beta3-5
>> (XEN) Command line: placeholder dom0_max_vcpus=4 dom0_mem=4G,max:4G
>> com2=115200,8n1 console=com2,vga noreboot xpti=no-dom0 cpuinfo
>> dom0_vcpus_pin
> 
> ... "loglvl=all" added so we can see all relevant information. In
> particular ...
> 
>> (XEN) ACPI: RSDP 000F4F00, 0024 (r2 HP)
>> (XEN) ACPI: XSDT D7630140, 00B4 (r1 HP ProLiant2162E)
>> (XEN) ACPI: FACP D7630240, 00F4 (r3 HP ProLiant2162E)
>> (XEN) ACPI: DSDT D7630340, 20BD (r1 HP DSDT1 INTL 20030228)
>> (XEN) ACPI: FACS D762F100, 0040
>> (XEN) ACPI: SPCR D762F140, 0050 (r1 HP SPCRRBSU1162E)
>> (XEN) ACPI: MCFG D762F1C0, 003C (r1 HP ProLiant1 0)
>> (XEN) ACPI: HPET D762F200, 0038 (r1 HP ProLiant2162E)
>> (XEN) ACPI:  D762F240, 0064 (r2 HP ProLiant2162E)
>> (XEN) ACPI: SPMI D762F2C0, 0040 (r5 HP ProLiant1162E)
>> (XEN) ACPI: ERST D762F300, 01D0 (r1 HP ProLiant1162E)
>> (XEN) ACPI: APIC D762F500, 015E (r1 HP ProLiant2 0)
>> (XEN) ACPI: SRAT D762F680, 0570 (r1 HP Proliant1162E)
>> (XEN) ACPI:  D762FC00, 0176 (r1 HP ProLiant1162E)
>> (XEN) ACPI: BERT D762FD80, 0030 (r1 HP ProLiant1162E)
>> (XEN) ACPI: HEST D762FDC0, 00BC (r1 HP ProLiant1162E)
>> (XEN) ACPI: DMAR D762FE80, 0146 (r1 HP ProLiant1162E)
>> (XEN) ACPI: SSDT D7632400, 0125 (r3 HP  CRSPCI02   HP1)
>> (XEN) ACPI: SSDT D7632540, 01CF (r3 HP  riser1a2 INTL 20061109)
>> (XEN) ACPI: SSDT D7632740, 03BB (r1 HP  pcc1 INTL 20090625)
>> (XEN) ACPI: SSDT D7632B00, 0377 (r1 HP pmab1 INTL 20090625)
>> (XEN) ACPI: SSDT D7632E80, 2094 (r1  INTEL PPM RCM 1 INTL 20061109)
> 
> ... this demonstrates that ACPI is available, and I very much
> hope your system isn't as screwed as there not being a MADT
> available despite all the other ACPI tables being present.

Now with loglvl=all:

-# xl dmesg
(XEN) parameter "placeholder" unknown!
(XEN) Xen version 4.11.1-pre (Debian
4.11.1~pre+1.733450b39b-1~exp1~mxbp9+1) (h...@knorrie.org) (gcc (Debian
6.3.0-18+deb9u1) 6.3.0 20170516) debug=n  Wed Aug 22 15:26:44 UTC 2018
(XEN) Bootloader: GRUB 2.02~beta3-5
(XEN) Command line: placeholder dom0_max_vcpus=4 dom0_mem=4G,max:4G
com2=115200,8n1 console=com2,vga noreboot xpti=no-dom0 cpuinfo
dom0_vcpus_pin loglvl=all
(XEN) Xen image load base address: 0
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: none; EDID transfer time: 2 seconds
(XEN)  EDID info not retrieved because no DDC retrieval method detected
(XEN) Disc information:
(XEN)  Found 1 MBR signatures
(XEN)  Found 1 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)   - 0009f400 (usable)
(XEN)  0009f400 - 000a (reserved)
(XEN)  000f - 0010 (reserved)
(XEN)  0010 - d762f000 (usable)
(XEN)  d762f000 - d763c000 (ACPI data)
(XEN)  d763c000 - d763d000 (usable)
(XEN)  d763d000 - dc00 (reserved)
(XEN)  fec0 - fee1 (reserved)
(XEN)  000

Re: [Xen-devel] SMT/Hyperthreading detection not always correct

[Hans van Kranenburg]

On 09/03/2018 03:35 PM, Jan Beulich wrote:
 On 03.09.18 at 15:24,  wrote:
>> (XEN) ACPI: LAPIC (acpi_id[0x00] lapic_id[0x00] enabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x10] lapic_id[0x20] enabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x08] lapic_id[0x10] enabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x18] lapic_id[0x30] enabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x04] enabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x14] lapic_id[0x24] enabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x0c] lapic_id[0x14] enabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x1c] lapic_id[0x34] enabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x02] enabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x12] lapic_id[0x22] enabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x0a] lapic_id[0x12] enabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x1a] lapic_id[0x32] enabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x06] lapic_id[0x06] disabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x16] lapic_id[0x26] disabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x0e] lapic_id[0x16] disabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x1e] lapic_id[0x36] disabled)
> 
> These look to be all the primary threads, the last few disabled due
> to being absent.
> 
>> (XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x01] disabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x11] lapic_id[0x21] disabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x09] lapic_id[0x11] disabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x19] lapic_id[0x31] disabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x05] lapic_id[0x05] disabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x15] lapic_id[0x25] disabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x0d] lapic_id[0x15] disabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x1d] lapic_id[0x35] disabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x03] disabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x13] lapic_id[0x23] disabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x0b] lapic_id[0x13] disabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x1b] lapic_id[0x33] disabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x07] lapic_id[0x07] disabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x17] lapic_id[0x27] disabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x0f] lapic_id[0x17] disabled)
>> (XEN) ACPI: LAPIC (acpi_id[0x1f] lapic_id[0x37] disabled)
> 
> And these are all the secondary threads afaict, which are all
> disabled. I'm unconvinced we can assume a system is not using
> hyperthreading simply based upon their MADT entries showing all
> of them as disabled - what if a HT-enabled CPU was later
> hotplugged?
> 
> I would have hoped that the topology related CPUID output
> would change when HT is (properly) disabled by the BIOS.
> 
> For the moment I can't see a way around you specifying "smt=0"
> on systems like this one.

The actual reason why I started the conversation about it was that for a
regular end user, the message telling the user that he has "hardware
with SMT/Hyperthreading enabled" is a confusing thing if you just made
sure that you have it disabled in the settings.

So, alternatively, instead of a technical solution, the warning could be
changed to something like "Booted on L1TF-vulnerable hardware and unable
to properly detect if SMT/Hyperthreading is enabled or disabled
Please choose explicit smt setting blah to make sure that it behaves how
you want."

As a user, that's less confusing, it just tells me that Xen likes me to
make explicit what I want.

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH 07/12] tools/xenstore: Re-introduce (fake) xs_restrict call to preserve ABI

[Hans van Kranenburg]

On 10/12/2018 05:12 PM, Ian Jackson wrote:
> From: Hans van Kranenburg 

No, this was in the changes that I copied back from Ubuntu, it was
written by Stefan Bader:

 >8 

Description: Re-introduce fake xs_restrict API call
 libxenstore cannot remove an API function without changing its version
 number. As long as we want to remain with 3.0 we have to keep it around.
 Debian might decide to increment the version at some point but we do not
 know how and when. So for now keep the version stable.

Author: Stefan Bader 

 >8 

> libxenstore3.0 in Xen 4.8 had this function.  We don't really want to
> bump the ABI version (soname) just for this, since we don't think
> there are actual callers anywhere.  But tools complain about the
> symbol going away.
> 
> So, provide a function xs_restrict which conforms to the original
> semantics, although it always fails.
> 
> Gbp-Pq: Topic xenstore
> Gbp-Pq: Name tools-fake-xs-restrict.patch
> Signed-off-by: Ian Jackson 
> ---
> v2: New in this version of the series
> ---
>  tools/xenstore/include/xenstore.h | 5 +
>  tools/xenstore/xs.c   | 6 ++
>  2 files changed, 11 insertions(+)
> 
> diff --git a/tools/xenstore/include/xenstore.h 
> b/tools/xenstore/include/xenstore.h
> index f460b8c5e5..0d95bb0e5c 100644
> --- a/tools/xenstore/include/xenstore.h
> +++ b/tools/xenstore/include/xenstore.h
> @@ -132,6 +132,11 @@ bool xs_mkdir(struct xs_handle *h, xs_transaction_t t,
>  bool xs_rm(struct xs_handle *h, xs_transaction_t t,
>  const char *path);
>  
> +/* Fake function which will always return false (required to let
> + * libxenstore remain at 3.0 version.
> + */
> +bool xs_restrict(struct xs_handle *h, unsigned domid);
> +
>  /* Get permissions of node (first element is owner, first perms is "other").
>   * Returns malloced array, or NULL: call free() after use.
>   */
> diff --git a/tools/xenstore/xs.c b/tools/xenstore/xs.c
> index 77700bff2b..cbcebb2bce 100644
> --- a/tools/xenstore/xs.c
> +++ b/tools/xenstore/xs.c
> @@ -796,6 +796,12 @@ unwind:
>   return false;
>  }
>  
> +/* Always return false a functionality has been removed in Xen 4.9 */
> +bool xs_restrict(struct xs_handle *h, unsigned domid)
> +{
> + return false;
> +}
> +
>  /* Watch a node for changes (poll on fd to detect, or call read_watch()).
>   * When the node (or any child) changes, fd will become readable.
>   * Token is returned when watch is read, to allow matching.
> 


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v2] libxencall: Improve linux syscall error messages

[Hans van Kranenburg]

On 10/15/2018 04:32 PM, Ian Jackson wrote:
> Make the bufdev and non-bufdev messages distinct, and always print the
> non-constant argument (ie, the size).

Ok, so I was doing live migration of a domU with 8GiB ram, with Xen 4.11
and Linux 4.18 in the dom0, and it consistently failed with:

xencall: error: alloc_pages: mmap failed: Invalid argument
xc: error: Unable to allocate memory for dirty bitmaps, batch pfns and
deferred pages: Internal error
xc: error: Save failed (12 = Cannot allocate memory): Internal error
libxl-save-helper: debug: complete r=-1: Cannot allocate memory

With Linux 4.17 it either succeeds, or crashes the hypervisor (oh yeah!).

Thanks to help in #xendevel a workaround was found (thanks Juergen), by
bumping the value of /sys/module/xen_privcmd/parameters/limit to
something higher than the default 64, which, using some obscure
calculation method, seems to mean 256k memory.

I could migrate this 8GiB domU when setting it to 128.

Andrew said:
17:13 < andyhhp> migration can and will use large buffers
17:13 < andyhhp> the total size of mappings is O(n) with the size of the
VM you are trying to migrate.

If I live migrate a domU with 64GiB of memory, how do I know what value
I need for this limit?

It feels a bit like a new kind of gnttab_max_frames setting that users
will run into when they do non-trivial things with Xen.

If this is the case, and sane low defaults are preferred, then I'd
really like to see some kind of error message that is actually helpful
for an end user. E.g. "Hey! You're trying to migrate a domU with XGiB of
memory. It's not gonna happen right now, but if you change the value for
/sys/module/xen_privcmd/parameters/limit to Y it will work. Don't worry,
your domU is still running fine on this host.", having Y calculated back
from memory size.

And then the live migration howto should contain a section with tunables
that describe how you should tune xen taking the max domU memory size
that you're using into account...

Just some thinking out loud... For normal users, these kind of errors
are really scary, and if they're not a bug, but an expected thing
because of settings, the way of presenting them to the user should not
look like a really serious problem.

Hans

> This assists diagnosis.
> 
> CC: Andrew Cooper 
> CC: Hans van Kranenburg 
> ---
> v2: Print sizes.
> ---
>  tools/libs/call/linux.c | 8 +---
>  1 file changed, 5 insertions(+), 3 deletions(-)
> 
> diff --git a/tools/libs/call/linux.c b/tools/libs/call/linux.c
> index d8a6306e04..51fa4899eb 100644
> --- a/tools/libs/call/linux.c
> +++ b/tools/libs/call/linux.c
> @@ -93,7 +93,8 @@ static void *alloc_pages_bufdev(xencall_handle *xcall, 
> size_t npages)
>   xcall->buf_fd, 0);
>  if ( p == MAP_FAILED )
>  {
> -PERROR("alloc_pages: mmap failed");
> +PERROR("alloc_pages: mmap (,%zu*%lu,...) [bufdev] failed",
> +   npages, (unsigned long)PAGE_SIZE);
>  p = NULL;
>  }
>  
> @@ -110,7 +111,7 @@ static void *alloc_pages_nobufdev(xencall_handle *xcall, 
> size_t npages)
>  p = mmap(NULL, size, PROT_READ|PROT_WRITE, 
> MAP_PRIVATE|MAP_ANONYMOUS|MAP_LOCKED, -1, 0);
>  if ( p == MAP_FAILED )
>  {
> -PERROR("alloc_pages: mmap failed");
> +PERROR("alloc_pages: mmap(,%zu,...) [nobufdev] failed", size);
>  return NULL;
>  }
>  
> @@ -119,7 +120,8 @@ static void *alloc_pages_nobufdev(xencall_handle *xcall, 
> size_t npages)
>  rc = madvise(p, npages * PAGE_SIZE, MADV_DONTFORK);
>  if ( rc < 0 )
>  {
> -PERROR("alloc_pages: madvise failed");
> +PERROR("alloc_pages: madvise (,%zu*%lu,) [nobufdev] failed",
> +   npages, (unsigned long)PAGE_SIZE);
>  goto out;
>  }
>  
> 


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v3 00/18] xen: add pvh guest support

[Hans van Kranenburg]

Hi,

On 10/24/2018 04:54 PM, Juergen Gross wrote:
> This patch series adds support for booting Linux as PVH guest.
> 
> Similar to i386/xen and x86_64/xen platforms the new i386/xenpvh
> platform grub is booted as a standalone image directly by Xen.
> 
> For booting Linux kernel it is using the standard linux kernel
> loader. The only modification of the linux loader is to pass the
> ACPI RSDP address via boot parameters to the kernel, as that table
> might not be located at the usual physical address just below 1MB.
> 
> The related Linux kernel patches have been accepted in the merge
> window for post 4.19 (4.20 or 5.0, whatever it will be called).
> 
> Changes in V3:
> - addressed lots of comments, most by Daniel and some by Roger,
>   details are mentioned in the individual patches
> 
> [...]

Thanks for the hard work.

FWIW, a test report from an end user.

I used the following for testing, which is derived from the instructions
Juergen provided earlier this year. Maybe it's useful for someone else
who wants to test.

git clone https://git.savannah.gnu.org/git/grub.git
git checkout -b xen_pvh
[apply patches from emails]
./autogen.sh
rm -rf knorrie/build
mkdir -p knorrie/build
cd knorrie/build
../../configure TARGET_LDFLAGS=-static --target=i386 --with-platform=xen_pvh
make

I just symlinked the test-kernel to vmlinuz-pvh here, so that I could
easily boot back into the regular kernel with pvgrub2, which is
hardcoded to look at /vmlinuz in my case (no grub with menu in the domU
etc).

cat 

Re: [Xen-devel] Xen Project Spectre/Meltdown FAQ

[Hans van Kranenburg]

On 01/05/2018 12:35 PM, Lars Kurth wrote:
> Hi all, this is a repost of
> https://blog.xenproject.org/2018/01/04/xen-project-spectremeltdown-faq/
> for xen-users/xen-devel. If you have questions, please reply to this
> thread and we will try and improve the FAQ based on questions. 
> Regards Lars

Thanks for the writeup.

The main reason for the reader to get confused is the amount of
different combinations of situations that are possible, which all again
have their own set of vulnerabilities and also their own (maybe even
different) set of possibilities to be used as environment for executing
an attack.

So let's help them by being more explicit.

> Google’s Project Zero announced several information leak
> vulnerabilities affecting all modern superscalar processors. Details
> can be found on their blog, and in the Xen Project Advisory 254 [1].
> To help our users understand the impact and our next steps forward,
> we put together the following FAQ.
> 
> Note that we will update the FAQ as new information surfaces.
> 
> = Is Xen impacted by Meltdown and Spectre? =
> 
> There are two angles to consider for this question:
> 
> * Can an untrusted guest attack the hypervisor using Meltdown or
> Spectre?
> * Can a guest user-space program attack a guest kernel using
> Meltdown or Spectre?

> Systems running Xen, like all operating systems and hypervisors, are
> potentially affected by Spectre (referred to as SP1 and SP2 in
> Advisory 254 [1]). For Arm Processors information, you can find which
> processors are impacted here [2].  In general, both the hypervisor
> and a guest kernel are vulnerable to attack via SP1 and SP2.
> 
> Only Intel processors are impacted by Meltdown (referred to as SP3 in
> Advisory 254 [1]).

> On Intel processors, only 64-bit PV mode guests can attack Xen.

"On Intel processors an attack at Xen using SP3 can only be done by
64-bit PV mode guests."

Even if it looks super-redundant, I think keeping explicit information
in every sentence is preferable, so they cannot be misinterpreted or
accidentally be taken out of context.

> Guests running in 32-bit PV mode, HVM mode, and PVH
> mode cannot attack the hypervisor using SP3. However, in 32-bit PV
> mode, HVM mode, and PVH mode, guest userspaces can attack guest
> kernels using SP3; so updating guest kernels is advisable.

> Interestingly, guest kernels running in 64-bit PV mode are not
> vulnerable to attack using SP3, because 64-bit PV guests already run
> in a KPTI-like mode.

Like Juergen already mentioned, additionally: "However, keep in mind
that a succesful attack on the hypervisor can still be used to recover
information about the same guest from physical memory."

> = Is there any risk of privilege escalation? =
> 
> Meltdown and Spectre are, by themselves, only information leaks.
> There is no suggestion that speculative execution can be used to
> modify memory or cause a system to do anything it might not have done
> already.
> 
> = Where can I find more information? =
> 
> We will update this blog post and Advisory 254 [1] as new information
> becomes available. Updates will also be published on xen-announce@.
> 
> We will also maintain a technical FAQ on our wiki [3] for answers to
> more detailed technical questions that emerge on xen-devel@ and other
> communication channels.
> 
> = Are there any patches for the vulnerability? =
> 
> We have prototype patches for a mitigation for Meltdown on Intel CPUs
> and a Mitigation for SP2/CVE-2017-5715, which are functional but have
> not undergone rigorous review and have not been backported to all
> supported Xen Project releases.
> 
> As information related to Meltdown and Spectre is now public,
> development will continue in public on xen-devel@ and patches will be
> posted and attached to Advisory 254 [1] as they become available in
> the next few days.
> 
> = Can SP1/SP2 be fixed at all? What plans are there to mitigate them?
> =
> 
> SP2 can be mitigated in two ways, both of which essentially prevent
> speculative execution of indirect branches. The first is to flush the
> branch prediction logic on entry into the hypervisor. This requires
> microcode updates, which Intel and AMD are in the process of
> preparing, as well as patches to the hypervisor which are also in
> process and should be available soon.
> 
> The second is to do indirect jumps in a way which is not subject to
> speculative execution. This requires the hypervisor to be recompiled
> with a compiler that contains special new features. These new
> compiler features are also in the process of being prepared for both
> gcc and clang, and should be available soon.
> 
> SP1 is much more difficult to mitigate. We have some ideas we’re
> exploring, but they’re still at the design stage at this point.
> 
> = Does Xen have any equivalent to Linux’s KPTI series? =
> 
> Linux’s KPTI series is designed to address SP3 only.

This one...

> For Xen guests, only 64-bit PV guests are affected by SP3.

...should be more expl

[Xen-devel] debian bugs: kaiser patch break pvh v1

[Hans van Kranenburg]

Hi,

At debian, we have reports of users that were apparently using PVH v1
with Xen 4.8 and Linux 4.9. After the debian kernel team included the
meltdown patches, PVH v1 breaks:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=886591

So, my questions are:

1. Should we just gently tell the users that they were not supposed to
do that in the first place, and tell them to look for alternatives?

2. Well, not really a question, just a heads-up, maybe there's something
in here that you find interesting to look into. I have no idea if e.g.
the thing can be related to any problem in 4.14 or newer.

Regards,
Hans van Kranenburg

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Radical proposal: ship not-fully-tidied shim as 4.10.1

[Hans van Kranenburg]

On 01/09/2018 07:22 PM, Rich Persaud wrote:
>>> On Jan 9, 2018, at 12:56, Stefano Stabellini  wrote:
>>>
>>> On Tue, 9 Jan 2018, Doug Goldstein wrote:
>>> On 1/9/18 11:33 AM, Jan Beulich wrote:
>>> On 09.01.18 at 18:23,  wrote:
> On Tue, Jan 9, 2018 at 8:52 AM, Stefano Stabellini
>  wrote:
 On Tue, 9 Jan 2018, George Dunlap wrote:
 On Mon, Jan 8, 2018 at 9:01 PM, Rich Persaud  wrote:
 On a similarly pragmatic note: would a variation of Anthony's vixen 
 patch
> series be suitable for pre-PVH Xen 4.6 - 4.9?  These versions are 
> currently 
> documented as security-supported (Oct 2018 - July 2020).
>>>
>>> Hmm, Ian's mail seems to be focusing on the idea of checking in a
>>> non-polished series to 4.10, rather than exctly what the content of
>>> that series would be.
>>>
>>> In the IRL conversation that preceeded this mail, the new short-term
>>> target we discussed was:
>>> 1. A 4.10-based shim that could boot either under HVM or PVH
>>> 2. A script that would take an existing PV config, and spit out a) a
>>> bootable ISO with the shim & whatever was needed, and b) a new config
>>> that would boot the same VM, but in HVM mode with the shim
>>>
>>> The script + a 4.10 shim binary *should* allow most PV guests to boot
>>> without any changes whatsoever for most older versions of Xen.
>>>
>>> There are a number of people for whom this won't work; I think we also
>>> need to provide a way to transparently change PV guests into PVshim
>>> guests.  But that will necessarily involve significant toolstack
>>> functionality, at which point you might as well backport PVH as well.
>>
>> Yes, there will be a number of people that won't be covered by this fix,
>> including those that can't use HVM/PVH mode because VT-x isn't available
>> at all in their environment. That is the only reason to run PV today.
>> Providing a way to transparently change PV guests into PVshim guests
>> won't cover any of these cases. A more complete workaround to SP3 is
>> along the lines of https://marc.info/?l=xen-devel&m=151509740625690.
>>
>> That said, I realize that we are only trying to do the best we can in a
>> very difficult situation, with very little time in our hands. I agree
>> with Ian that we should commit something unpolished and only partially
>> reviewed soon, even though it doesn't cover a good chunk of the userbase
>> for one reason or another. Even if migration doesn't work, it will still
>> help all that don't require it. It is only a partial fix by nature
>> anyway.
>
> Can people be a bit more explicit about what they think should be done 
> here?
>
> I'm happy to redirect effort to PVH shim if that's what the solution
> is going to be.
>
> I obviously prefer the HVM approach as it works on a broad range of Xen 
> versions
> without modification but I'm keen to get something done quickly and
> don't want to
> waste effort.

 From what I've read today, I have no reason to believe the PVH
 shim won't work in HVM mode. How would the HVM-only approach
 be better in that case?

 Jan
>>>
>>> I feel like I should state the obvious here. Its tested over a large
>>> data set.
>>
>> Right: if we are going to commit something unpolished and unreviewed,
>> let it be at least very well tested by the submitter. Honest question:
>> how much more dev&test we need on PVShim before we get it to similar
>> levels of confidence?
> 

> Since the primary audience for security fixes are production
> deployments of Xen where customer assets are at risk, is there an
> estimate for the percentage/size of Xen deployments where PVH (not
> only Xen 4.10) has already been deployed for production customers?
> That could give other customers more confidence in deploying PVH in
> production.
+1

I have been hearing mostly-very-positive stories around, except for the
missing pvgrub2 support. :)

But as a sysadmin who's also strongly considering to jump to 4.10 and
PVH I'd definitely like to hear more stories.

Hans

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen Project Spectre/Meltdown FAQ

[Hans van Kranenburg]

On 01/11/2018 10:15 AM, Lars Kurth wrote:
> I am wondering whether something like the attached table would make
> understanding the FAQ easier. Page 1 is clearly what is Xen specific and
> we definitely should cover.
> Page 2 in general covers Linux and guests. The first block is relatively
> straightforward.
> 
> The 2nd and 3rd block is based on information from Doug: as there are
> many gaps, I would be uneasy about publishing these somewhere prominent. 
> 
> Also
>> As this is really guest specific this information can't be provided by
>> Xen.
> which carries a risk that any analysis made by anyone might only apply
> to the context in which the analysis was done.
> 
> But the question keeps coming up, so making this clearer is maybe sensible.

Yes! This is a really good thing do do, since it's much more powerful
than trying to express the "multi-dimensional combinations" in  sentences.

When having this, the amount of text in the faq should just clearly
describe the categories, and cut out all the "X can but not if Y, but
also Y but not if Z" type sentences and then refer to the tables for the
end verdict for a specific users own situation.

  -- >8 --

The one thing I would want to point out again, which keeps to be a
non-obvious thing for users, is that in the short term with the pvshim
solution, a 64 bit PV guest in pvshim mode can still not be protected
against itself.

At  "Is a user space attack on the guest kernel possible (when running
in a Xen VM)"  there could be a [3] at 64 bit PV no, with the
explanation that while technically correct, this can again be
circumvented by exploiting the attack via Xen (see 'on other guest'
table) back to itself.

Or maybe adding an extra table "Is a user space attack via Xen back to
the guest itself possible (when running in a Xen VM)?" will help instead.

And to make it more complicated, a user would want to see how the tables
change when injecting the pvshim approach... For that, it might be
sufficient to add an extra row to all tables with "64 bit PV in pvhsim"
just below "64 bit PV".


Thanks,
Hans


[0] I also haven't see this info in any PR from AWS about PV guests?
Like, "hey, we protected ourselves and other customers against you, but
we can't do anything about your own business. Please stop using 64bit PV
instances for now if there's anything untrusted running inside."

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Radical proposal: ship not-fully-tidied shim as 4.10.1

[Hans van Kranenburg]

On 01/10/2018 06:50 AM, Juergen Gross wrote:
> On 09/01/18 23:11, Hans van Kranenburg wrote:
>> On 01/09/2018 07:22 PM, Rich Persaud wrote:
>>>
>>
>>> Since the primary audience for security fixes are production
>>> deployments of Xen where customer assets are at risk, is there an
>>> estimate for the percentage/size of Xen deployments where PVH (not
>>> only Xen 4.10) has already been deployed for production customers?
>>> That could give other customers more confidence in deploying PVH in
>>> production.
>> +1
>>
>> I have been hearing mostly-very-positive stories around, except for the
>> missing pvgrub2 support. :)
> 
> https://lists.xen.org/archives/html/xen-devel/2017-11/msg01795.html

Ah thanks. I also found the links to kernel changes and a xen change.

http://lists.gnu.org/archive/html/grub-devel/2017-12/msg2.html

I'm going to play around with it, see if I can get it running. Being
able to use pvgrub2 will mean ending up with an incredibly smooth
migration path to Xen 4.10 and PVHv2...

Hans


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Xen Project Spectre/Meltdown FAQ

[Hans van Kranenburg]

On 01/11/2018 08:22 PM, Peter wrote:
> On 2018-01-11 22:16, Lars Kurth wrote:
>> And this time with attachment
>>> On 11 Jan 2018, at 09:15, Lars Kurth 
>>> wrote:
>>>
>>> I am wondering whether something like the attached table would make
>>> understanding the FAQ easier. Page 1 is clearly what is Xen specific
>>> and we definitely should cover.
>>> Page 2 in general covers Linux and guests. The first block is
>>> relatively straightforward.
>>>
>>> The 2nd and 3rd block is based on information from Doug: as there
>>> are many gaps, I would be uneasy about publishing these somewhere
>>> prominent.
>>>
>>> Also
>>>
 As this is really guest specific this information can't be
 provided by
 Xen.
>>>
>>> which carries a risk that any analysis made by anyone might only
>>> apply to the context in which the analysis was done.
>>>
>>> But the question keeps coming up, so making this clearer is maybe
>>> sensible.
> 
> In the matrix I see "Is a user space attack on the guest kernel possible
> (when running in a Xen VM)?" For PVH (and HVM) = Yes[1] where [1]
> Impacts Intel CPUs only.
> 
> Is there any mitigation for this?  i.e. How to protect a guest VM from
> its own userspace processes.

That part is handled by the kernel inside the guest. Xen doesn't see
that happening.

It's for example the KPTI/KAISER patches that got into the linux kernels
now.

>>>
>>> Best Regards
>>> Lars
>>>
>>> On 10 Jan 2018, at 06:03, Juergen Gross  wrote:
>>> On 10/01/18 04:58, Peter wrote:
>>> On 2018-01-09 15:04, Stefano Stabellini wrote:
>>> On Sun, 7 Jan 2018, Marek Marczykowski-Górecki wrote:
>>> On Fri, Jan 05, 2018 at 07:05:56PM +, Andrew Cooper wrote:
>>> On 05/01/18 18:16, Rich Persaud wrote:
>>> On Jan 5, 2018, at 06:35, Lars Kurth >> > wrote:
>>> Linux’s KPTI series is designed to address SP3 only.  For Xen
>>  guests,
>>
>>> only 64-bit PV guests are affected by SP3. A KPTI-like approach was
>>> explored initially, but required significant ABI changes.
>>
>> Is some partial KPTI-like approach feasible? Like unmapping memory
>> owned
>> by other guests, but keeping Xen areas mapped? This will still allow
>> leaking Xen memory, but there are very few secrets there (vCPUs state,
>> anything else?), so overall impact will be much lower.
>>
>> +1
>>
>> I believe
>> https://blog.xenproject.org/2018/01/04/xen-project-spectremeltdown-faq/
>> is clear re VMs attacking/accessing the host/dom0/hypervisor and the
>> mitigations for that.
>>
>> However the page seems ambiguous about whether 64 bit VMs running as
>> PVHv2 with host provided kernels are protected or not (from each VM's
>> own processes).
>>
>> PVHv2 is using exactly the same runtime environment as HVM seen from
>> the
>> hypervisor. So a guest running as PVHv2 needs a PTI like approach like
>> HVM in its kernel.
>>
>>> Can the page be updated to be more explicit and perhaps describe how
>>> the
>>> VM kernel or how the PVHv2 virtualization provides that protection.
>>> And
>>> ideally how that could be checked from the VM itself.  e.g. grep pti
>>> /proc/cpuinfo?
>>
>> As this is really guest specific this information can't be provided by
>> Xen.
>>
>>> e.g. the page says: "Guest kernels running in 64-bit PV mode are not
>>> directly vulnerable to attack using SP3, because 64-bit PV guests
>>> already run in a KPTI-like mode." but it does not mention PVHv2 for
>>> that.  Is it protected under PVHv2?  Does it depend on the kernel?
>>> Is
>>> so what is the patchset/option/mechanism that protects the VM from
>>> its
>>> own processes?
>>
>> This question should have been answered above already.
>>
>> Juergen
>>
> 
> ___
> Xen-devel mailing list
> Xen-devel@lists.xenproject.org
> https://lists.xenproject.org/mailman/listinfo/xen-devel


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Clarification regarding Meltdown and 64-bit PV guests

[Hans van Kranenburg]

On 01/13/2018 07:42 AM, Andy Smith wrote:
> Hi,
> 
> In
> :
> 
> "On Intel processors, only 64-bit PV mode guests can attack Xen
> using Variant 3. [...]

64bit PV domU userspace can attack the hypervisor -> read "host memory"
which also contains contents of guest memory (including itself)

> Interestingly, guest kernels running in 64-bit PV mode are not
> vulnerable to attack using Variant 3, because 64-bit PV guests
> already run in a KPTI-like mode."

64-bit PV domU userspace can't attack kernel inside guest
Quoting Andrew C (on IRC): "the 64bit PV ABI already has split
guest-user / guest-kernel pagetables; (because 64bit x86 took away
working segment limits from the 32bit world); it mitigates guest
userspace SP3 attacks against the guest kernel"

This is also the reason for adding the if-statement which prevents the
new kpti code from becoming active when booting your latest kernel in a
64-bit PV domU.

> However, in multiple other places, including
>  and
> :
> 
> "Note that both of these shim-based approaches prevent attacks on
> the host, but leave the guest vulnerable to Meltdown attacks by
> its own unprivileged processes; this is true even if the guest
> OS has KPTI or similar Meltdown mitigation."

So, even while it can't attack itself inside the guest, it can make a
"detour" over the hypervisor to again read its own memory.

> These seem to contradict each other.
> 
> The FAQ seems to suggest that:
> 
> - 32-bit PV guest userland processes can use Variant 3 against their
>   own kernels and that the KPTI patch would protect against that.
> 
> - Without Comet/Vixen, 64-bit PV guests can't use Variant 3 on
>   themselves but can use it on the hypervisor, and KPTI patches in
>   the guest do not prevent that.

^^

> - Running PV guests inside Comet or Vixen prevents them making use
>   of Variant 3, they still cannot use Variant 3 against their own
>   kernels, and KPTI patches in the guest are not necessary.

Like it was for a long time already, because of the "64bit PV ABI
already has split [...]"

> The Comet and Vixen READMEs seem to suggest that:
> 
> - Use of Comet/Vixen prevents PV guests from using Variant 3 against
>   the hypervisor (and thus other guests as well).

By injecting a copy of a hypervisor between the outer level hypervisor
(that's called L0 right?) (in HVM or PVH mode) and the guest, having it
just run 1 guest, that (64-bit PV) guest cannot attack its own kernel,
but it can attack the intermediate hypervisor which results in reading
it's own memory from the fake intermediate "host memory".

> - The guest itself remains able to use Variant 3 on its own kernel
>   and KPTI patches inside the guest cannot prevent this.

This one should be incorrect.

> Which is correct, or have I misunderstood and they are somehow both
> correct?

It's complicated. At least, the above is what I understood.

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] Clarification regarding Meltdown and 64-bit PV guests

[Hans van Kranenburg]

On 01/13/2018 11:08 AM, Andy Smith wrote:
> Hi Hans,
> 
> On Sat, Jan 13, 2018 at 10:43:03AM +0100, Hans van Kranenburg wrote:
>> By injecting a copy of a hypervisor between the outer level hypervisor
>> (that's called L0 right?) (in HVM or PVH mode) and the guest, having it
>> just run 1 guest, that (64-bit PV) guest cannot attack its own kernel,
>> but it can attack the intermediate hypervisor which results in reading
>> it's own memory from the fake intermediate "host memory".
> 
> So are you saying that, considering only SP3/Variant 3/Meltdown, it
> works out like this:
> 
> == 64-bit PV mode guest ==
> 
> - Can't use SP3/Variant 3/Meltdown directly on its own kernel.
> 
> - Can use SP3/Variant 3/Meltdown on the hypervisor to read data from
>   hypervisor so effectively everything including other kernels and
>   its own kernel.
> 
> - Can't be mitigated by KPTI in the guest.
> 
> == PV-in-Comet and PV-in-Vixen ==
> 
> - Can't use SP3/Variant 3/Meltdown directly on its own kernel
> 
> - Can't use SP3/Variant 3/Meltdown on the real hypervisor.
> 
> - Can still use SP3/Variant 3/Meltdown on the shim hypervisor to
>   still gain access to data from itself.
> 
> - Can't be mitigated by KPTI in the guest.
> 
> == HVM and PVHv2 ==
> 
> - Can use SP3/Variant 3/Meltdown directly on its own kernel.
> 
> - Can't use SP3/Variant 3/Meltdown on the hypervisor.
> 
> - Can be mitigated by KPTI in the guest (becomes not a Xen issue).
> 
> ?

Exactly.

> If so, then I can see how the FAQ, README.Comet and README.Vixen
> can all be correct in this regard,

Well, what just happened is that I didn't provide any new information,
but presented the same information by just rewording it again, which
triggered a few missing connections to complete your mental image.

> but do note that this is extremely confusing

Yes it is.

A. Different types of Xen versions:
  1. <= 4.5, which is EOL and EOS, but still used (like AWS popping up
having things running based on Xen 3.4)
  2. 4.6 and 4.7, which have some security support
  3. 4.8 and 4.9, for which rumours are going that it might get PVHv2
from 4.10 backported
  4. 4.10, which can be used with PVHv2 (with guest kernel linux 4.14)

B. Different archs:
  1. x86 / 32-bit
  2. x64 / amd64 / 64-bit
  3. Add ARM things here...
  4. ...

C. Different CPU vendors:
  1. Intel
  2. AMD
  3. ...

D. Then different virtualization modes:
  1. PV
  2. HVM
  3. PVHv2

E. Different mitigation techniques for Xen (which are each only possible
with a subset of choices from other categories):
  1. Convert PV -> HVM
  2. Convert PV -> PVHv2
  3. Insert shim Vixen
  4. Insert shim Comet

F. Different kernels in the guest (only looking at linux here now):
  1. Any kernel released < Jan 3 2018
  2. Old version without any kaiser/kpti patch available
  3. Old version with kaiser backport patch (3.2, 3.16, 4.9 etc)
  4. 4.14 or 4.15 kernel with KPTI patch

G. Different mitigation techniques for inside the guest:
  1. Do nothing because 64-bit PV guest
  2. Get kernel with kaiser or kpti patch

H. And of course describing any situation as "vulnerable" has a
shortcoming, because we need to distinguish (4 combinations possible):
  1. Can the situation be used to attack?
  2. Is the situation vulnerable to attack by another guest/whatever?

ad H. Oh, and what if someone has a mix of HVM and PV guests? If there
are only HVM guests, they're safe from each other, but if you add 1
64-bit PV guest to it, suddenly you have one available to do an attack,
and all the HVM are suddenly vulnerable.

So it's a bit of an 7 or 8-dimensional space, and every situation of
different users is floating somewhere in there as a point. Every
combination of all of those can result in a wildly different outcome.

Oh wait, I forgot we're only talking about SP3. Let's add SP1 and SP2,
now we are 9-dimensional.

You know these kind of puzzles?
http://www.burre.nl/p/puzzels/logisch/logisch_1.gif

It's what we're doing here. :-) Well, instead of a boolean result, our
puzzle has multiple results, describing what's relevant for hypervisor,
for guest kernel etc...

> and a lot of people are only reading the
> comments that say that Xen PV can't make use of SP3/Variant
> 3/Meltdown.

Expressing all these things in text is really hard.

Lars made a new attempt with the tables, but you immediately see it
takes 4 or 5 tables below each other, and then it's still like "argh!"
because we can't flatten 8-dimensional to 2-dimensional that easy.

 >8 

So, I have a new idea:

Let's make an interactive html/javascript thingie where you can play
around with all the combinations above. Based on everything you choose,
there'

Re: [Xen-devel] Clarification regarding Meltdown and 64-bit PV guests

[Hans van Kranenburg]

On 14/01/2018 15:00, Dongli Zhang wrote:
> Hi Hans,
> 
> On 01/13/2018 07:12 PM, Hans van Kranenburg wrote:
>> On 01/13/2018 11:08 AM, Andy Smith wrote:
>>> Hi Hans,
>>>
>>> On Sat, Jan 13, 2018 at 10:43:03AM +0100, Hans van Kranenburg wrote:
>>>> By injecting a copy of a hypervisor between the outer level hypervisor
>>>> (that's called L0 right?) (in HVM or PVH mode) and the guest, having it
>>>> just run 1 guest, that (64-bit PV) guest cannot attack its own kernel,
>>>> but it can attack the intermediate hypervisor which results in reading
>>>> it's own memory from the fake intermediate "host memory".
>>>
>>> So are you saying that, considering only SP3/Variant 3/Meltdown, it
>>> works out like this:
>>>
>>> == 64-bit PV mode guest ==
>>>
>>> - Can't use SP3/Variant 3/Meltdown directly on its own kernel.
>>>
>>> - Can use SP3/Variant 3/Meltdown on the hypervisor to read data from
>>>   hypervisor so effectively everything including other kernels and
>>>   its own kernel.
>>>
>>> - Can't be mitigated by KPTI in the guest.
>>>
>>> == PV-in-Comet and PV-in-Vixen ==
>>>
>>> - Can't use SP3/Variant 3/Meltdown directly on its own kernel
>>>
>>> - Can't use SP3/Variant 3/Meltdown on the real hypervisor.
>>>
>>> - Can still use SP3/Variant 3/Meltdown on the shim hypervisor to
>>>   still gain access to data from itself.
>>>
>>> - Can't be mitigated by KPTI in the guest.
>>>
>>> == HVM and PVHv2 ==
>>>
>>> - Can use SP3/Variant 3/Meltdown directly on its own kernel.
>>>
>>> - Can't use SP3/Variant 3/Meltdown on the hypervisor.
>>>
>>> - Can be mitigated by KPTI in the guest (becomes not a Xen issue).
>>>
>>> ?
>>
>> Exactly.
> 
> Does this indicate that there is no solution to prevent a malicious user space
> program (running on 64-bit PV domU) from reading the memory of domU kernel 
> space
> (via meltdown), no matter whether comet/vixen is enabled or not?
> 
> Therefore, comet/vixen is only used to prevent the cross-VM meltdown attack.

Yes.

For reference, I looked back at my IRC logs. This is the related
conversation about it, #xen IRC on freenode, from Jan 6 2018, when I was
firing off questions to Andrew.

18:17 < Knorrie> with Xen 4.8+ and the pvshim method for non-pvh guest
kernels, is the xen implementation that does the same as KPTI inside the
guest for 64bit PV still present?
18:20 < andyhhp> hmm
18:20 < Knorrie> I think that's an interesting questino :)
18:20 < andyhhp> the 64bit PV ABI already has an isolation between guest
userspace and guest kernel
18:20 < andyhhp> so in effect, has had KPTI for 12 years already...
18:21 < andyhhp> however
18:21 < andyhhp> Xen's mappings are not currently isolated, so either
guest userspace or guest kernel can attack Xen using SP3/Meltdown
18:22 < Knorrie> yes, as I understood the reason of the pvshim feature
was to fix exactly that
18:23 < andyhhp> the Linux KPTI feature should turn itself off when it
detects it is running as a PV guest under Xen
18:24 < andyhhp> but this was quite late to the party, and it appears
that Redhat have published an update to RHEL 6.x which bricks PV guests :(
18:24 < Knorrie> ah right, I remember having seen that if statement one
of these days yes
18:24 < Knorrie> so no need to manually turn  it off in exactly that
scenario
18:26 < andyhhp> the PV-shim doesn't do anything clever
18:26 < andyhhp> all we do is stick a second version of Xen in there
18:26 < andyhhp> so we've got L0 Xen running a PVH guest (so in an HVM
container).
18:26 < andyhhp> that PVH guest is actually a Xen running a single
legacy PV guest
18:27 < Knorrie> well, the clever things is that you can't see outside
of your own memory any more right
18:27 < Knorrie> s/things/thing/
18:27 < andyhhp> the alteration is/should/will be transparent from the
point of view of the legacy PV guest
18:28 < andyhhp> the legacy PV guest can still attempt to pull off
SP3/Meltdown attacks, but they stop at the L1 Xen
18:29 < andyhhp> so all it can do is do is read details relating to
itself.  It can't read anything belonging to other VMs
18:29 < Knorrie> like, outgoing network traffic from other processes on
the same guest?
18:29 < Knorrie> (regardless of how practical and realistic those
attacks are)

 >8 

18:30 < andyhhp> it should be noted that PV-SHIM isn't a perfect
replacement for some kind of Xen based KPTI so

[Xen-devel] What about dom0? (was: Re: Clarification regarding Meltdown and 64-bit PV guests)

[Hans van Kranenburg]

On 13/01/2018 07:42, Andy Smith wrote:
> [...]

Related...

Dom0 is a PV guest with more privileges than a domU, right?

Currently all documentation and FAQ do not contain a section about this.
It's all about the hypervisor and guests.

Probable reason for this is that most people on this list implicitly
think about dom0 as a thing that has as only purpose to manage starting
and stopping domUs, forwarding some network traffic using bridges or
openvswitch, and with its own network connection probably on a separate
management network...

But what about small-deployment-users who use the dom0 for more things,
and have a bunch of Xen domUs running inside. Who knows what they do,
maybe they even have their graphical desktop environment running in dom0
with web browser and everything?

Just throwing some dust in the air... :)

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [PATCH v2] libxl: put RSDP for PVH guest near 4GB

[Hans van Kranenburg]

On 01/24/2018 01:06 PM, Juergen Gross wrote:
> On 24/01/18 11:54, Roger Pau Monné wrote:
>> On Wed, Jan 24, 2018 at 10:42:39AM +, George Dunlap wrote:
>>> On Wed, Jan 24, 2018 at 2:41 AM, Boris Ostrovsky
>>>  wrote:
 On 01/18/2018 05:33 AM, Wei Liu wrote:
> On Thu, Jan 18, 2018 at 11:31:32AM +0100, Juergen Gross wrote:
>> Wei,
>>
>> On 01/12/17 15:14, Juergen Gross wrote:
>>> Instead of locating the RSDP table below 1MB put it just below 4GB
>>> like the rest of the ACPI tables in case of PVH guests. This will
>>> avoid punching more holes than necessary into the memory map.
>>>
>>> Signed-off-by: Juergen Gross 
>>> Acked-by: Wei Liu 
>> Mind applying this one?
> Don't worry, it is in my queue.
>
> Will come to this and other patches I accumulated soon.
>
> Wei.

 This requires kernel changes, doesn't it?

 https://lists.xenproject.org/archives/html/xen-devel/2017-12/msg00714.html

 And this series apparently never made it to the tree.

 PVH guests are broken now on staging.
>>>
>>> And the Linux side of PVH is officially supported now, right?  Which
>>> means we need to have some way of being backwards compatible with
>>> kernels without that patch, even if it does get  backported.
>>
>> Linux had a bug that on PVH it expected to find the RSDP at the low
>> 1MiB. The PVHv2 boot protocol has _never_ mentioned that the RSDP
>> should be below 1MiB. There's just one comment in the start_info
>> header that says:
>>
>> "Xen on x86 will always try to place all the data below the 4GiB
>> boundary."
>>
>> And the RSDP address must be fetched from the rsdp_paddr field of the
>> start_info.
>>
>> I agree that the fact that Linux is broken is bad, but should not
>> force us to change the protocol. What's more, when booting a PVHv2
>> Dom0 the RSDP _must_ not be searched in the low 1MiB, or else Dom0 is
>> going to find the firmware provided RSDP and everything is going to
>> blow up.
> 
> Right.
> 
> My patches to repair this in the Linux kernel are on hold as they are
> touching the boot parameters between grub2 and the kernel, too. And
> this caused the kernel no longer booting in some cases as most distros
> are shipping a grub2 breaking this interface via a non-upstream patch.

Linking the related thread:
https://www.spinics.net/lists/linux-acpi/msg80003.html

Out of interest, which "non-upstream patch" is this by the way? I didn't
see it mentioned yet.

Is it defining an extra custom field in that stuct yolo style? Because
from my limited understanding that would of course also need the
counterpart being implemented at kernel side.

Or (and I suspect it's this) is is not really using those bytes, but
leaving garbage behind, which forces you to implement the extra
validation checks?

Thanks,

>> IMO, Linux must be fixed and the patch backported up to 4.11.
> 
> I'll try soon.

Hans


___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: Compiling Xen from source

[Hans van Kranenburg]

Hi,

On 9/16/20 12:00 AM, Andrew Cooper wrote:
> On 15/09/2020 21:16, David I wrote:
>> Hello,
>>
>> I am unable to mcompile xen-tools from source.
>>
>> here is the "stacktrace":
>>
>> ln -f xenstore xenstore-watch
>> gcc xenstore_control.o    
>>  -Wl,-rpath-link=/home/david/xen/xen/tools/xenstore/../../tools/libs/toolcore
>> /home/david/xen/xen/tools/xenstore/../../tools/xenstore/libxenstore.so
>> -ldl
>>  
>> /home/david/xen/xen/tools/xenstore/../../tools/libs/toolcore/libxentoolcore.so
>>  -o xenstore-control
>> gcc  -m64 -DBUILD_ID -fno-strict-aliasing -std=gnu99 -Wall
>> -Wstrict-prototypes -Wdeclaration-after-statement
>> -Wno-unused-but-set-variable -Wno-unused-local-typedefs   -O2
>> -fomit-frame-pointer
>> -D__XEN_INTERFACE_VERSION__=__XEN_LATEST_INTERFACE_VERSION__ -MMD -MF
>> .xs_tdb_dump.o.d -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE   -Werror
>> -I. -include /home/david/xen/xen/tools/xenstore/../../tools/config.h
>> -I./include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/libs/evtchn/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/libxc/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/libs/toollog/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/libs/foreignmemory/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/libs/devicemodel/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/include
>> -D__XEN_TOOLS__
>> -I/home/david/xen/xen/tools/xenstore/../../tools/libs/toolcore/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/include
>> -DXEN_LIB_STORED="\"/var/lib/xenstored\""
>> -DXEN_RUN_STORED="\"/var/run/xenstored\""   -c -o xs_tdb_dump.o
>> xs_tdb_dump.c
>> gcc  -m64 -DBUILD_ID -fno-strict-aliasing -std=gnu99 -Wall
>> -Wstrict-prototypes -Wdeclaration-after-statement
>> -Wno-unused-but-set-variable -Wno-unused-local-typedefs   -O2
>> -fomit-frame-pointer
>> -D__XEN_INTERFACE_VERSION__=__XEN_LATEST_INTERFACE_VERSION__ -MMD -MF
>> .utils.o.d -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE   -Werror -I.
>> -include /home/david/xen/xen/tools/xenstore/../../tools/config.h
>> -I./include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/libs/evtchn/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/libxc/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/libs/toollog/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/libs/foreignmemory/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/libs/devicemodel/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/include
>> -D__XEN_TOOLS__
>> -I/home/david/xen/xen/tools/xenstore/../../tools/libs/toolcore/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/include
>> -DXEN_LIB_STORED="\"/var/lib/xenstored\""
>> -DXEN_RUN_STORED="\"/var/run/xenstored\""  
>> -I/home/david/xen/xen/tools/xenstore/../../tools/libs/gnttab/include
>> -I/home/david/xen/xen/tools/xenstore/../../tools/include  -c -o
>> utils.o utils.c
>> gcc xs_tdb_dump.o utils.o tdb.o talloc.o     -o xs_tdb_dump
>> /usr/bin/ld : utils.o:(.data.rel.local+0x0) : définitions multiples de
>> « xprintf »; xs_tdb_dump.o:(.bss+0x0) : défini pour la première fois ici
>> collect2: error: ld returned 1 exit status
>> make[4]: *** [Makefile:97 : xs_tdb_dump] Erreur 1
>> make[4] : on quitte le répertoire « /home/david/xen/xen/tools/xenstore »
>> make[3]: *** [/home/david/xen/xen/tools/../tools/Rules.mk:249 :
>> subdir-install-xenstore] Erreur 2
>> make[3] : on quitte le répertoire « /home/david/xen/xen/tools »
>> make[2]: *** [/home/david/xen/xen/tools/../tools/Rules.mk:244 :
>> subdirs-install] Erreur 2
>> make[2] : on quitte le répertoire « /home/david/xen/xen/tools »
>> make[1]: *** [Makefile:74 : install] Erreur 2
>> make[1] : on quitte le répertoire « /home/david/xen/xen/tools »
>> make: *** [Makefile:127 : install-tools] Erreur 2
>> david@debian:~/xen/xen$ uname -a
>> Linux debian 5.7.0-2-amd64 #1 SMP Debian 5.7.10-1 (2020-07-26) x86_64
>> GNU/Linux
>> david@debian:~/xen/xen$ gcc --version
>> gcc (Debian 10.2.0-5) 10.2.0
>> Copyright (C) 2020 Free Software Foundation, Inc.
>> This is free software; see the source for copying conditions.  There is NO
>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
>> PURPOSE.
>>
>> david@debian:~/xen/xen$ /usr/bin/ld --version
>> GNU ld (GNU Binutils for Debian) 2.35
>> Copyright (C) 2020 Free Software Foundation, Inc.
>> Ce logiciel est libre; si vous le redistribuez, vous devez le faire
>> selon les termes
>> de la licence GNU General Public License version 3 ou p

Re: Compiling Xen from source

[Hans van Kranenburg]

On 9/16/20 6:12 PM, Olaf Hering wrote:
> On Wed, Sep 16, David I wrote:
> 
>> So, how did the debian package was compiled ? is this the same source code 
>> with
>> different options ?
> 
> Xen 4.11 is from 2018. Use also a compiler from that year.
> Using this years compiler will lead to errors...

In Debian, Buster with Xen 4.11 uses gcc 8.

The Xen 4.11 that is in Debian unstable now does not compile any more
with gcc 10. That's why we really need to get Xen 4.14 in there now to
un-stuck that (even with additional picked patches again already).

Hans

4.14.0 FTBFS for Debian unstable, libxlu_pci.c (╯°□°）╯︵ ┻━┻

[Hans van Kranenburg]

Hi!

News from the Debian Xen team (well, that's still only Ian and me). We
still have Xen 4.11 in Debian unstable and stable (Buster) now, but at
this point I really want to start working on the preparations for the
next Debian release which will happen in about a little bit less than a
year from now.

So, the 4.14.0 release is a good moment to kick it off. In february 2020
Ian and I already spent a day to move the Debian packaging to 4.13, and
the result has been laying around for a bit. Now I'm forwarding it to
4.14.0 and I really want to get this in Debian so users can start
playing around with it and have enough time to either contribute new
things (like cross-building for raspberry pi4!) etc.

All the yolo WIP stuff without anything cleaned up is here:
https://salsa.debian.org/xen-team/debian-xen/-/commits/knorrie/4.14

Unfortunately, it FTBFS in an unexpected way, since I cannot relate this
to any of our additional patches or anything.

This is the last part of the output with the failure:

 >8 

gcc  -m64 -DBUILD_ID -fno-strict-aliasing -std=gnu99 -Wall
-Wstrict-prototypes -Wdeclaration-after-statement
-Wno-unused-but-set-variable -Wno-unused-local-typedefs   -O2
-fomit-frame-pointer
-D__XEN_INTERFACE_VERSION__=__XEN_LATEST_INTERFACE_VERSION__ -MMD -MP
-MF .libxlu_pci.o.d -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE  -g -O2
-fdebug-prefix-map=/home/knorrie/build/xen/debian-xen=.
-fstack-protector-strong -Wformat -Werror=format-security -Wdate-time
-D_FORTIFY_SOURCE=2 -Werror -Wno-format-zero-length
-Wmissing-declarations -Wno-declaration-after-statement
-Wformat-nonliteral -I. -fPIC -pthread
-I/home/knorrie/build/xen/debian-xen/tools/libxl/../../tools/libxc/include
-I/home/knorrie/build/xen/debian-xen/tools/libxl/../../tools/libs/toollog/include
-I/home/knorrie/build/xen/debian-xen/tools/libxl/../../tools/include
-I/home/knorrie/build/xen/debian-xen/tools/libxl/../../tools/libs/foreignmemory/include
-I/home/knorrie/build/xen/debian-xen/tools/libxl/../../tools/include
-I/home/knorrie/build/xen/debian-xen/tools/libxl/../../tools/libs/devicemodel/include
-I/home/knorrie/build/xen/debian-xen/tools/libxl/../../tools/include
-I/home/knorrie/build/xen/debian-xen/tools/libxl/../../tools/include
-D__XEN_TOOLS__   -c -o libxlu_pci.o libxlu_pci.c
libxlu_pci.c: In function 'xlu_pci_parse_bdf':
libxlu_pci.c:32:18: error: 'func' may be used uninitialized in this
function [-Werror=maybe-uninitialized]
   32 | pcidev->func = func;
  | ~^~
libxlu_pci.c:51:29: note: 'func' was declared here
   51 | unsigned dom, bus, dev, func, vslot = 0;
  | ^~~~
libxlu_pci.c:31:17: error: 'dev' may be used uninitialized in this
function [-Werror=maybe-uninitialized]
   31 | pcidev->dev = dev;
  | ^
libxlu_pci.c:51:24: note: 'dev' was declared here
   51 | unsigned dom, bus, dev, func, vslot = 0;
  |^~~
libxlu_pci.c:30:17: error: 'bus' may be used uninitialized in this
function [-Werror=maybe-uninitialized]
   30 | pcidev->bus = bus;
  | ^
libxlu_pci.c:51:19: note: 'bus' was declared here
   51 | unsigned dom, bus, dev, func, vslot = 0;
  |   ^~~
libxlu_pci.c:29:20: error: 'dom' may be used uninitialized in this
function [-Werror=maybe-uninitialized]
   29 | pcidev->domain = domain;
  | ~~~^~~~
libxlu_pci.c:51:14: note: 'dom' was declared here
   51 | unsigned dom, bus, dev, func, vslot = 0;
  |  ^~~
cc1: all warnings being treated as errors
make[6]: ***
[/home/knorrie/build/xen/debian-xen/tools/libxl/../../tools/Rules.mk:218:
libxlu_pci.o] Error 1
make[6]: Leaving directory '/home/knorrie/build/xen/debian-xen/tools/libxl'
make[5]: ***
[/home/knorrie/build/xen/debian-xen/tools/../tools/Rules.mk:242:
subdir-install-libxl] Error 2
make[5]: Leaving directory '/home/knorrie/build/xen/debian-xen/tools'
make[4]: ***
[/home/knorrie/build/xen/debian-xen/tools/../tools/Rules.mk:237:
subdirs-install] Error 2
make[4]: Leaving directory '/home/knorrie/build/xen/debian-xen/tools'
make[3]: *** [Makefile:72: install] Error 2
make[3]: Leaving directory '/home/knorrie/build/xen/debian-xen/tools'
make[2]: *** [Makefile:134: install-tools] Error 2
make[2]: Leaving directory '/home/knorrie/build/xen/debian-xen'
make[1]: *** [debian/rules:205: override_dh_auto_build] Error 2
make[1]: Leaving directory '/home/knorrie/build/xen/debian-xen'
make: *** [debian/rules:153: build] Error 2
dpkg-buildpackage: error: debian/rules build subprocess returned exit
status 2

 >8 

I lost all of the other output already, but if needed, I can just cause
it again and provide a full build log with all specific package versions
of compilers etc. Anyway, it just used all the current stuff in Debian
unstable.

Knorrie

P.S. The next big thing to fix in the packaging before it can go into
Debian unstable is no usage of python 2 anywhere. I mig

Bug: toolstack allows too low values to be set for shadow_memory

[Hans van Kranenburg]

This bug report is a follow-up to the thread "Domu windows 2012 crash"
on the xen-users list. In there we found out that it is possible to set
a value for shadow_memory that is lower than a safe minimum value.

This became apparent after XSA-304, which caused using more of this type
of memory. Having a hardcoded line like shadow_memory = 8 results in
random crashes of the guest, with the following errors in xl dmesg of
the host:

(XEN) Failed to shatter gfn 105245: -12
(XEN) d75v1 EPT violation 0x19c (--x/rw-) gpa 0x0105245760 mfn
0x285245 type 0
(XEN) d75v1 Walking EPT tables for GFN 105245:
(XEN) d75v1  epte 9c04105f9007
(XEN) d75v1  epte 9c0280f3
(XEN) d75v1  --- GLA 0x7ff98b40d760
(XEN) domain_crash called from vmx.c:3497
(XEN) Domain 75 (vcpu#1) crashed on cpu#4:
(XEN) [ Xen-4.11.4-pre  x86_64  debug=n   Not tainted ]
(XEN) CPU:4
(XEN) RIP:0033:[<7ff98b40d760>]
(XEN) RFLAGS: 00010216   CONTEXT: hvm guest (d75v1)
(XEN) rax: 1212   rbx: 00c714d9da58   rcx: 02351590
(XEN) rdx: 00c70001   rsi: 00c714d9da18   rdi: 00c714d9db20
(XEN) rbp: 00c714d9d950   rsp: 00c714d9d918   r8:  02351470
(XEN) r9:  023514f0   r10: 7ff99c5c0923   r11: 00c714d9d970
(XEN) r12:    r13: 00c714d9d9d0   r14: 00c714d9da58
(XEN) r15: 0006   cr0: 80050031   cr4: 00060678
(XEN) cr3: 0001d9458002   cr2: 7ff98b6fa048
(XEN) fsb:    gsb: 00c714e9e000   gss: cd015dc4
(XEN) ds: 002b   es: 002b   fs: 0053   gs: 002b   ss: 002b   cs: 0033

Why do users have this shadow_memory line in their guest config file?
Well, I guess it's simply because there are not many proper examples of
how to run Windows under Xen, so whoever is searching ze interwobz for
it will end up on a page like...


https://www.howtoforge.com/how-to-run-fully-virtualized-guests-hvm-with-xen-3.2-on-debian-lenny-x86_64

...and then start copy pasting and adjusting until things work. Or...
already have such a config for 10+ years and just only adjust things
when it breaks after upgrading Xen.

Hans van Kranenburg

Re: Bug: toolstack allows too low values to be set for shadow_memory

[Hans van Kranenburg]

Hi,

On 5/26/20 9:41 AM, Jan Beulich wrote:
> On 25.05.2020 17:51, Hans van Kranenburg wrote:
>> This bug report is a follow-up to the thread "Domu windows 2012 crash"
>> on the xen-users list. In there we found out that it is possible to set
>> a value for shadow_memory that is lower than a safe minimum value.
>>
>> This became apparent after XSA-304, which caused using more of this type
>> of memory. Having a hardcoded line like shadow_memory = 8 results in
>> random crashes of the guest,
> 
> I don't think it is the tool stack's responsibility to override
> admin requested values, or at least not as far a affecting guest
> stability goes;

This is not primarily a technical issue, or about if software works
correct in a mathematical proven way.

It's a usability issue, so it's about what levels of unknowingly
unloading guns into feet is deemed desirable.

And, if that's happening, how difficult it should be for a user to
actually find out what's wrong.

> host stability of course needs to be guaranteed,
> but that's then still the hypervisor's job, not the tool stack's.
> 
> Compare this to e.g. setting too small a memory= for a guest to
> be able to boot at all, or setting maxmem > memory for a guest
> without balloon driver.
> 
> Furthermore - what would the suggestion be as to a "safe minimum
> value"? Assuming _all_ large pages may potentially get shattered
> is surely a waste of memory, unless the admin really knows
> guests are going to behave that way. (In your report you also
> didn't mention what memory= values the issue was observed with.
> Obviously larger memory= also require bumping shadow_memory= at
> least from some point onwards.)

Thanks,
Hans

Re: [Xen-devel] Design session report: Xen on Distros

[Hans van Kranenburg]

Hi,

On 7/15/19 4:42 PM, George Dunlap wrote:
> On 7/15/19 3:23 PM, Jan Beulich wrote:
>> On 15.07.2019 16:11, George Dunlap wrote:
>>> There was a long discussion about security patches, with the general
>>> proposal being that we should cut a point release for every security issue.
>>
>> Interesting. Looks like in politics that until a decision fits people
>> they keep re-raising the point. Iirc on a prior meeting (Budapest?)
>> we had settled on continuing with the current scheme. Were there any
>> new arguments towards this alternative model?
> 
> Well I don't know if there were any new arguments because I don't
> immediately remember the old discussion.  Do we have a summary of the
> discussion in Budapest, with its conclusions, anywhere?
> 
> The basic idea was that:
> 
> 1. Most distros / packagers are going to want to do an immediate release
> anyway.
> 
> 2. Distros generally seemed to be rebasing on top of staging as soon as
> the XSA went out anyway (and ISTR this being the recommeneded course of
> action)

FYI, In Debian, we only ship the stable branch, not the staging branch.
Better safe than sorry.

And yes, this means there's a delay, and it's not immediate, etc.

Well, at least since I have been involved. In the past security update
packages were made by applying patches manually on top of older (like,
4.4.1 instead of 4.4.x) frozen versions, but we decided that "trying to
assemble our own subset of the patches is riskier than taking upstream's
collection".

The Debian Release Team allows us to upload newer upstream versions
during a security upload for Xen, which is officially not allowed in
Debian stable. One of the things that obviously help with being able to
keep doing this is the "track record" of the quality (expecially
regression testing) of the stable-4.x branches.

Currently, our package versions mostly look like e.g.
4.11.1+92-g6c33308a8d-1, instead of a nice simple version number. For me
this is fine, but I can understand that for an end user it would just
look nicer (and feel better perception-wise) to get a 4.11.x package.

So just for that reason I would be all +1 for more tagged releases.

> So for all intents and purposes, we have something which is, in fact, a
> release; all it's missing is a signed tag and a tarball.
> 
> Obviously there are testing implications that would need to be sorted
> out before this could become a reality.
> 
> In any case, the ball is in the court of "VOLUNTEER" to write up a
> concrete proposal which could be discussed.  You'll be able to raise all
> your concerns at that point if you want (although having a sketch would
> of course be helpful for whoever is writing such a proposal).

Hans

___
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Debian Bug#1004269: Linker segfault while building src:xen

[Hans van Kranenburg]

(To both the Debian bug # and xen-devel list, reply-all is fine)
Hi Xen people,

I just filed a bug at Debian on the binutils package, because since the 
latest binutils package update (Debian 2.37.50.20220106-2), Xen (both 
4.14 and 4.16) fails to build with a segfault at the following point:


x86_64-linux-gnu-ld -mi386pep --subsystem=10 
--image-base=0x82d04000 --stack=0,0 --heap=0,0 
--section-alignment=0x20 --file-alignment=0x20 
--major-image-version=4 --minor-image-version=16 --major-os-version=2 
--minor-os-version=0 --major-subsystem-version=2 
--minor-subsystem-version=0 --no-insert-timestamp --build-id=sha1 -T 
efi.lds -N prelink.o 
/builds/xen-team/debian-xen/debian/output/source_dir/xen/common/symbols-dummy.o 
-b pe-x86-64 efi/buildid.o -o 
/builds/xen-team/debian-xen/debian/output/source_dir/xen/.xen.efi.0x82d04000.0 
&& :

Segmentation fault (core dumped)

Full message and links to build logs etc are in the initial bug message, 
to be seen at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1004269


We (Debian Xen Team) are awaiting response, but I thought to also let 
you know already.


* Does the above error 'ring a bell'?
* Can you maybe also reproduce this in a development environment with 
very latest binutils?
* Maybe someone has a useful comment for the Debian binutils maintainer 
about what's happening in this step of the build?

* Any suggestions about what we can do to help figure this out?
* We'll try to help debug, but will surely appreciate upstream help if 
things get too technical. It's simply the case that I did not have to 
look into a very similar issue before, so it's new.


Thanks!
Hans

Re: Bug#1063270: The "64bits time_t transition" in Debian/Xen

[Hans van Kranenburg]

Hi,

On 2/12/24 18:43, Andrew Cooper wrote:
> On 12/02/2024 5:27 pm, zithro wrote:
>> Hey all,
>>
>> the Debian project is focused on the "2038 time_t" switch.
>> So the maintainers of the Debian Xen package must ensure that all
>> imported Xen code conforms to the new Debian standards.
>>
>> I was asked by Andrew Cooper to post here about this, I'll quote him :
>> "So I had been idly wondering whether Xen would match up to Debian's new
>> policy, and it appears not
>> this topic really needs to be brought up on the xen-devel mailing list
>> do you have any more details as to what has gone wrong?
>> this is something we ought to arrange to happen in CI by default
>> but it sounds like there's some work needed first"
>>
>> (Not answering the question because I'm just a messenger).
> 
> xen.git/xen$ git grep -w time_t -- :/
> ../tools/console/client/main.c:106: time_t start, now;
> ../tools/console/daemon/io.c:272:   time_t now = time(NULL);
> ../tools/libs/light/libxl_qmp.c:116:    time_t timeout;
> ../tools/libs/light/libxl_qmp.c:585:   
> time_t ask_timeout)
> ../tools/libs/light/libxl_x86.c:516:    time_t t;
> ../tools/libs/toollog/xtl_logger_stdio.c:61:    time_t now = time(0);
> ../tools/tests/xenstore/test-xenstore.c:453:    time_t stop;
> ../tools/xenmon/xenbaked.c:98:time_t start_time;
> ../tools/xenstored/core.c:109:  time_t now;
> ../tools/xenstored/core.h:150:  time_t ta_start_time;
> ../tools/xenstored/domain.c:143:    time_t mem_last_msg;
> ../tools/xenstored/domain.c:188:static time_t wrl_log_last_warning; /*
> 0: no previous warning */
> ../tools/xenstored/domain.c:1584:   time_t now;
> ../tools/xenstored/lu.c:160:    time_t now = time(NULL);
> ../tools/xenstored/lu.c:185:    time_t now = time(NULL);
> ../tools/xenstored/lu.c:292:    time_t now = time(NULL);
> ../tools/xenstored/lu.h:32: time_t started_at;
> ../tools/xentop/xentop.c:947:   time_t curt;
> ../tools/xl/xl_info.c:742:static char *current_time_to_string(time_t now)
> ../tools/xl/xl_info.c:759:static void print_dom0_uptime(int short_mode,
> time_t now)
> ../tools/xl/xl_info.c:810:static void print_domU_uptime(uint32_t domuid,
> int short_mode, time_t now)
> ../tools/xl/xl_info.c:847:    time_t now;
> ../tools/xl/xl_vmcontrol.c:336:    time_t start;
> ../tools/xl/xl_vmcontrol.c:495:    time_t now;
> ../tools/xl/xl_vmcontrol.c:504:    if (now == ((time_t) -1)) {
> ../tools/xs-clients/xenstore_control.c:33:    time_t time_start;
> arch/x86/cpu/mcheck/mce.h:224:    uint64_t time; /* wall time_t when
> error was detected */
> arch/x86/time.c:1129: * machines were long is 32-bit! (However, as
> time_t is signed, we
> 
> 
> I don't see any ABI problems from using a 64bit time_t.  The only header
> file with a time_t is xenstored/lu.h which is a private header and not a
> public ABI.
> 
> I guess we fell into the "could not be analysed via
> abi-compliance-checker" case?

Thanks for also looking into this!

Maximilian mentioned in #debian-xen that doing a Debian package build
with DEB_BUILD_OPTIONS=abi=+lfs and _FILE_OFFSET_BITS=64 and
_TIME_BITS=64 resulted in the exact same binaries for shared libs.

What we also found is these reports:

1. Enabling lfs, which has no effect:
https://adrien.dcln.fr/misc/armhf-time_t/2024-02-06T16%3A48%3A00/compat_reports/libxen-dev/base_to_lfs/compat_report.html

2. Enabling the 64-bit time_t as well:
https://adrien.dcln.fr/misc/armhf-time_t/2024-02-06T16%3A48%3A00/compat_reports/libxen-dev/lfs_to_time_t/compat_report.html
In there, see "Problems with Data Types, Low Severity  2 " about
struct_timeval:

 >8 

  [+] struct timeval
Change -> Effect
1 Type of field tv_sec has been changed from __time_t to __time64_t.
-> Recompilation of a client program may be broken.
2 Type of field tv_usec has been changed from __suseconds_t to
__suseconds64_t. -> Recompilation of a client program may be broken.

  [+] affected symbols: 3 (0.2%)
* libxl_osevent_afterpoll ( libxl_ctx* ctx, int nfds, struct pollfd
const* fds, struct timeval now ) -> 4th parameter 'now' is of type
'struct timeval'.
* libxl_osevent_beforepoll ( libxl_ctx* ctx, int* nfds_io, struct
pollfd* fds, int* timeout_upd, struct timeval now ) -> 5th parameter
'now' is of type 'struct timeval'.
* libxl_osevent_register_hooks ( libxl_ctx* ctx, libxl_osevent_hooks
const* hooks, void* user ) -> Field 'hooks.timeout_modify.p2' in 2nd
parameter 'hooks' (pointer) has base type 'struct timeval'.

 >8 

So, the question is, is this correct and would it cause a problem.

If so, it also means that those functions are in a versioned lib,
libxenlight.so.4.17.0 (in binary package libxenmisc4.17).

Coincidentally, we are currently preparing the upload to switch from Xen
4.17 to Xen 4.18 in Debian unstable. So, if we just go ahead with doing
that, and make sure it's built in the new way already...

then...

tada.wav!

We just immediately have the correct libxenmisc4.1

Re: [XEN PATCH v2] docs: set date to SOURCE_DATE_EPOCH if available

[Hans van Kranenburg]

Hi,

On 12/23/20 5:56 PM, Maximilian Engelhardt wrote:
> check if a GNU date that supports the '-u -d @...' options and syntax or
> a BSD date are available. If so, use the appropriate options for the
> date command to produce a custom date if SOURCE_DATE_EPOCH is defined.
> If SOURCE_DATE_EPOCH is not defined or no suitable date command was
> found, use the current date. This enables reproducible builds.
> 
> Signed-off-by: Maximilian Engelhardt 
> 
> Changes in v2:
> - add capability detection for the 'date' command using ax_prog_date.m4
> - add information about detected date command to config/Docs.mk
> - only call a supported date command in docs/Makefile
> ---
> Please note the ax_prog_date.m4 macro is taken from the autoconf-archive
> repository [1] and it's license is GPL v3 or later with an exception for
> the generated configure script.
> 
> [1] https://www.gnu.org/software/autoconf-archive/
> ---
>  config/Docs.mk.in  |   3 +
>  docs/Makefile  |  16 +++-
>  docs/configure | 213 +
>  docs/configure.ac  |   9 ++
>  m4/ax_prog_date.m4 | 139 +
>  5 files changed, 379 insertions(+), 1 deletion(-)
>  create mode 100644 m4/ax_prog_date.m4

Wait, what. The comment about the -d option already existing since 2005
(in the previous thread) is relevant here...

I guess there would be other reasons why the whole current Xen master
branch would not compile on e.g. Debian Sarge 3.1 from 2005... Like,
amd64 did not even exist as release architecture yet, back then...

I'd prefer

  1 file changed, 7 insertions(+), 1 deletion(-)

over

  5 files changed, 379 insertions(+), 1 deletion(-)

in this case.

Hans

> diff --git a/config/Docs.mk.in b/config/Docs.mk.in
> index e76e5cd5ff..cc2abd8fcc 100644
> --- a/config/Docs.mk.in
> +++ b/config/Docs.mk.in
> @@ -7,3 +7,6 @@ POD2HTML:= @POD2HTML@
>  POD2TEXT:= @POD2TEXT@
>  PANDOC  := @PANDOC@
>  PERL:= @PERL@
> +
> +# Variables
> +DATE_TYPE_AT:= @DATE_TYPE_AT@
> diff --git a/docs/Makefile b/docs/Makefile
> index 8de1efb6f5..c79fe0b63e 100644
> --- a/docs/Makefile
> +++ b/docs/Makefile
> @@ -3,7 +3,21 @@ include $(XEN_ROOT)/Config.mk
>  -include $(XEN_ROOT)/config/Docs.mk
>  
>  VERSION  := $(shell $(MAKE) -C $(XEN_ROOT)/xen 
> --no-print-directory xenversion)
> -DATE := $(shell date +%Y-%m-%d)
> +
> +DATE_FMT := +%Y-%m-%d
> +ifdef SOURCE_DATE_EPOCH
> +ifeq ($(DATE_TYPE_AT),GNU)
> +DATE := $(shell date -u -d "@$(SOURCE_DATE_EPOCH)" "$(DATE_FMT)")
> +else
> +ifeq ($(DATE_TYPE_AT),BSD)
> +DATE := $(shell date -u -r "$(SOURCE_DATE_EPOCH)" "$(DATE_FMT)")
> +else
> +DATE := $(shell date "$(DATE_FMT)")
> +endif
> +endif
> +else
> +DATE := $(shell date "$(DATE_FMT)")
> +endif
>  
>  DOC_ARCHES  := arm x86_32 x86_64
>  MAN_SECTIONS:= 1 5 7 8
> diff --git a/docs/configure b/docs/configure
> index f55268564e..9c3218f560 100755
> --- a/docs/configure
> +++ b/docs/configure
> @@ -587,6 +587,7 @@ PACKAGE_URL='https://www.xen.org/'
>  ac_unique_file="misc/xen-command-line.pandoc"
>  ac_subst_vars='LTLIBOBJS
>  LIBOBJS
> +DATE_TYPE_AT
>  PERL
>  PANDOC
>  POD2TEXT
> @@ -1808,6 +1809,86 @@ case "$host_os" in
>  esac
>  
>  
> +# Fetched from 
> https://git.savannah.gnu.org/gitweb/?p=autoconf-archive.git;a=blob_plain;f=m4/ax_prog_date.m4
> +# Commit ID: fd1d25c14855037f6ccd7dbc7fe9ae5fc9bea8f4
> +# ===
> +#   https://www.gnu.org/software/autoconf-archive/ax_prog_date.html
> +# ===
> +#
> +# SYNOPSIS
> +#
> +#   AX_PROG_DATE()
> +#
> +# DESCRIPTION
> +#
> +#   This macro tries to determine the type of the date (1) command and some
> +#   of its non-standard capabilities.
> +#
> +#   The type is determined as follow:
> +#
> +# * If the version string contains "GNU", then:
> +#   - The variable ax_cv_prog_date_gnu is set to "yes".
> +#   - The variable ax_cv_prog_date_type is set to "gnu".
> +#
> +# * If date supports the "-v 1d" option, then:
> +#   - The variable ax_cv_prog_date_bsd is set to "yes".
> +#   - The variable ax_cv_prog_date_type is set to "bsd".
> +#
> +# * If both previous checks fail, then:
> +#   - The variable ax_cv_prog_date_type is set to "unknown".
> +#
> +#   The following capabilities of GNU date are checked:
> +#
> +# * If date supports the --date arg option, then:
> +#   - The variable ax_cv_prog_date_gnu_date is set to "yes".
> +#
> +# * If date supports the --utc arg option, then:
> +#   - The variable ax_cv_prog_date_gnu_utc is set to "yes".
> +#
> +#   The following capabilities of BSD date are checked:
> +#
> +# * If date supports the -v 1d  option, then:
> +#   - The variable ax_cv_prog_date_bsd_adjust is set to "yes".
> +#
> +# * If date supports the -r ar

Re: [PATCH] xen/iommu: smmu: Use 1UL << 31 rather than 1 << 31

[Hans van Kranenburg]

On 12/24/20 4:24 PM, Julien Grall wrote:
> From: Julien Grall 
> 
> Replace all the use of 1 << 31 with 1UL << 31 to prevent undefined
> behavior in the SMMU driver.

You're replacing it by 1U, not 1UL, in the patch below.

Hans

> Signed-off-by: Julien Grall 
> ---
>  xen/drivers/passthrough/arm/smmu.c | 8 
>  1 file changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/xen/drivers/passthrough/arm/smmu.c 
> b/xen/drivers/passthrough/arm/smmu.c
> index ed04d85e05e9..3e8aa378669b 100644
> --- a/xen/drivers/passthrough/arm/smmu.c
> +++ b/xen/drivers/passthrough/arm/smmu.c
> @@ -405,7 +405,7 @@ static struct iommu_group *iommu_group_get(struct device 
> *dev)
>  #define ID0_NUMSMRG_SHIFT0
>  #define ID0_NUMSMRG_MASK 0xff
>  
> -#define ID1_PAGESIZE (1 << 31)
> +#define ID1_PAGESIZE (1U << 31)
>  #define ID1_NUMPAGENDXB_SHIFT28
>  #define ID1_NUMPAGENDXB_MASK 7
>  #define ID1_NUMS2CB_SHIFT16
> @@ -438,7 +438,7 @@ static struct iommu_group *iommu_group_get(struct device 
> *dev)
>  
>  /* Stream mapping registers */
>  #define ARM_SMMU_GR0_SMR(n)  (0x800 + ((n) << 2))
> -#define SMR_VALID(1 << 31)
> +#define SMR_VALID(1U << 31)
>  #define SMR_MASK_SHIFT   16
>  #define SMR_MASK_MASK0x7fff
>  #define SMR_ID_SHIFT 0
> @@ -506,7 +506,7 @@ static struct iommu_group *iommu_group_get(struct device 
> *dev)
>  #define RESUME_RETRY (0 << 0)
>  #define RESUME_TERMINATE (1 << 0)
>  
> -#define TTBCR_EAE(1 << 31)
> +#define TTBCR_EAE(1U << 31)
>  
>  #define TTBCR_PASIZE_SHIFT   16
>  #define TTBCR_PASIZE_MASK0x7
> @@ -562,7 +562,7 @@ static struct iommu_group *iommu_group_get(struct device 
> *dev)
>  #define MAIR_ATTR_IDX_CACHE  1
>  #define MAIR_ATTR_IDX_DEV2
>  
> -#define FSR_MULTI(1 << 31)
> +#define FSR_MULTI(1U << 31)
>  #define FSR_SS   (1 << 30)
>  #define FSR_UUT  (1 << 8)
>  #define FSR_ASF  (1 << 7)
> 

Re: (Debian) Bug#1051862: server flooded with xen_mc_flush warnings with xen 4.17 + linux 6.1

[Hans van Kranenburg]

Hi Radoslav,

Thanks for your report...

Hi Juergen, Boris and xen-devel,

At Debian, we got the report below. (Also at
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1051862)

This hardware, with only Xen and Dom0 running is hitting the failed
multicall warning and logging in arch/x86/xen/multicalls.c. Can you help
advise what we can do to further debug this issue?

Since this looks like pretty low level Xen/hardware stuff, I'd rather
ask upstream for directions first. If needed the Debian Xen Team can
assist the end user with the debugging process.

Thanks,

More reply inline...

On 9/13/23 20:12, Radoslav Bodó wrote:
> Package: xen-system-amd64
> Version: 4.17.1+2-gb773c48e36-1
> Severity: important
> 
> Hello,
> 
> after upgrade from Bullseye to Bookworm one of our dom0's
> became unusable due to logs/system being continuously flooded
> with warnings from arch/x86/xen/multicalls.c:102 xen_mc_flush, and the 
> system become unusable.
> 
> The issue starts at some point where system services starts to come up, 
> but nothing very special is on that box (dom0, nftables, fail2ban, 
> prometheus-node-exporter, 3x domU). We have tried to disable all domU's 
> and fail2ban as the name of the process would suggest, but issue is 
> still present. We have tried also some other elaboration but none of 
> them have helped so far:
> 
> * the issue arise when xen 4.17 + linux >= 6.1 is booted
> * xen + bookworm-backports linux-image-6.4.0-0.deb12.2-amd64 have same isuue
> * without xen hypervisor, linux 6.1 runs just fine
> * systemrescue cd boot and xfs_repair rootfs did not helped
> * memtest seem to be fine running for hours

Thanks for already trying out all these combinations.

> As a workaround we have booted xen 4.17 + linux 5.10.0-25 (5.10.191-1)
> and the system is running fine as for last few months.
> 
> Hardware:
> * Dell PowerEdge R750xs
> * 2x Intel Xeon Silver 4310 2.1G
> * 256GB RAM
> * PERC H755 Adapter, 12x 18TB HDDs

I have a few quick additional questions already:

1. For clarification.. From your text, I understand that only this one
single server is showing the problem after the Debian version upgrade.
Does this mean that this is the only server you have running with
exactly this combination of hardware (and BIOS version, CPU microcode
etc etc)? Or, is there another one with same hardware which does not
show the problem?

2. Can you reply with the output of 'xl dmesg' when the problem happens?
Or, if the system gets unusable too quick, do you have a serial console
connection to capture the output?

3. To confirm... I understand that there are many of these messages.
Since you pasted only one, does that mean that all of them look exactly
the same, with "1 of 1 multicall(s) failed: cpu 10" "call  1: op=1
arg=[a1a9eb10] result=-22"? Or are there variations? If so, can
you reply with a few different ones?

Since this very much looks like an issue of Xen related code where the
Xen hypervisor, dom0 kernel and hardware has to work together correctly,
(and not a Debian packaging problem) I'm already asking upstream for
advice about what we should/could do next, instead of trying to make a
guess myself.

Thanks,
Hans

> Any help, advice or bug confirmation would be appreciated
> 
> Best regards
> bodik
> 
> 
> (log also in attachment)
> 
> ```
> kernel: [   99.762402] WARNING: CPU: 10 PID: 1301 at 
> arch/x86/xen/multicalls.c:102 xen_mc_flush+0x196/0x220
> kernel: [   99.762598] Modules linked in: nvme_fabrics nvme_core bridge 
> xen_acpi_processor xen_gntdev stp llc xen_evtchn xenfs xen_privcmd 
> binfmt_misc intel_rapl_msr ext4 intel_rapl_common crc16 
> intel_uncore_frequency_common mbcache ipmi_ssif jbd2 nfit libnvdimm 
> ghash_clmulni_intel sha512_ssse3 sha512_generic aesni_intel acpi_ipmi 
> nft_ct crypto_simd cryptd mei_me mgag200 ipmi_si iTCO_wdt intel_pmc_bxt 
> ipmi_devintf drm_shmem_helper dell_smbios nft_masq iTCO_vendor_support 
> isst_if_mbox_pci drm_kms_helper isst_if_mmio dcdbas mei intel_vsec 
> isst_if_common dell_wmi_descriptor wmi_bmof watchdog pcspkr 
> intel_pch_thermal ipmi_msghandler i2c_algo_bit acpi_power_meter button 
> nft_nat joydev evdev sg nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 
> nf_defrag_ipv4 nf_tables nfnetlink drm fuse loop efi_pstore configfs 
> ip_tables x_tables autofs4 xfs libcrc32c crc32c_generic hid_generic 
> usbhid hid dm_mod sd_mod t10_pi crc64_rocksoft crc64 crc_t10dif 
> crct10dif_generic ahci libahci xhci_pci libata xhci_hcd
> kernel: [   99.762633]  megaraid_sas tg3 crct10dif_pclmul 
> crct10dif_common crc32_pclmul crc32c_intel bnxt_en usbcore scsi_mod 
> i2c_i801 libphy i2c_smbus usb_common scsi_common wmi
> kernel: [   99.764765] CPU: 10 PID: 1301 Comm: python3 Tainted: G 
> W  6.1.0-12-amd64 #1  Debian 6.1.52-1
> kernel: [   99.764989] Hardware name: Dell Inc. PowerEdge R750xs/0441XG, 
> BIOS 1.8.2 09/14/2022
> kernel: [   99.765214] RIP: e030:xen_mc_flush+0x196/0x220
> kernel: [   99.765436] Code: e2 06 48 01 da 85 c0 0f 8