Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2022-01-27 Thread Mike
I just made the huge mistake of hibernating and resuming, I'm going trough the process of rescue and all, thankfully I had a 2016 cd in the drive. I'll read up once the sheer panic settles. -Michael On Wed, Jan 26, 2022, 21:22 John Paul Adrian Glaubitz < glaub...@physik.fu-berlin.de> wrote: > Hi

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2022-01-26 Thread John Paul Adrian Glaubitz
Hi Michael! On 1/13/22 01:17, John Paul Adrian Glaubitz wrote: > On 1/9/22 23:17, John Paul Adrian Glaubitz wrote: >> On 1/7/22 12:20, John Paul Adrian Glaubitz wrote: Can you separately test with (on the host): # echo 0 > /sys/module/kvm_hv/parameters/dynamic_mt_modes >>> >>> I'm

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2022-01-12 Thread John Paul Adrian Glaubitz
Hi Michael! On 1/9/22 23:17, John Paul Adrian Glaubitz wrote: > On 1/7/22 12:20, John Paul Adrian Glaubitz wrote: >>> Can you separately test with (on the host): >>> >>> # echo 0 > /sys/module/kvm_hv/parameters/dynamic_mt_modes >> >> I'm trying to turn off "dynamic_mt_modes" first and see if that

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2022-01-09 Thread John Paul Adrian Glaubitz
Hi Michael! On 1/7/22 12:20, John Paul Adrian Glaubitz wrote: >> Can you separately test with (on the host): >> >> # echo 0 > /sys/module/kvm_hv/parameters/dynamic_mt_modes > > I'm trying to turn off "dynamic_mt_modes" first and see if that makes any > difference. > > I will report back. So f

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2022-01-07 Thread John Paul Adrian Glaubitz
Hi Michael! On 1/6/22 11:58, Michael Ellerman wrote: >> We're currently running 5.15.x on the host system and the guests and the >> testsuite >> for gcc-9 still reproducibly kills the KVM host. > > Have you been able to try the different -smp options I suggested? > > Can you separately test wit

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2022-01-06 Thread Michael Ellerman
John Paul Adrian Glaubitz writes: > Hi Michael! > > Sorry for the long time without any responses. Shall we continue debugging > this? Yes! Sorry also that I haven't been able to fix it yet, I had to stop chasing this bug and work on other things before the end of the year. > We're currently r

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2022-01-04 Thread John Paul Adrian Glaubitz
Hi Michael! Sorry for the long time without any responses. Shall we continue debugging this? We're currently running 5.15.x on the host system and the guests and the testsuite for gcc-9 still reproducibly kills the KVM host. Adrian On 11/1/21 07:53, Michael Ellerman wrote: > Sure, will give th

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-11-01 Thread Michal Suchánek
On Fri, Oct 29, 2021 at 02:33:12PM +0200, John Paul Adrian Glaubitz wrote: > Hi Nicholas! > > On 10/29/21 02:41, Nicholas Piggin wrote: > > Soft lockup should mean it's taking timer interrupts still, just not > > scheduling. Do you have the hard lockup detector enabled as well? Is > > there anyth

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-11-01 Thread Michal Suchánek
Hello, On Thu, Oct 28, 2021 at 04:15:19PM +0200, John Paul Adrian Glaubitz wrote: > Hi! > > On 10/28/21 16:05, John Paul Adrian Glaubitz wrote: > > The following packages were being built at the same time: > > > > - guest 1: virtuoso-opensource and openturns > > - guest 2: llvm-toolchain-13 > >

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-11-01 Thread John Paul Adrian Glaubitz
Hi Michael! On 11/1/21 08:37, John Paul Adrian Glaubitz wrote: > I made another experiment and upgraded the host to 5.15-rc7 which contains > your > fixes and made the guests build gcc-10. Interestingly, this time, the gcc-10 > build crashed the guest but didn't manage to crash the host. I will u

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-11-01 Thread John Paul Adrian Glaubitz
Hi Michael! On 11/1/21 07:53, Michael Ellerman wrote: > Sure, will give that a try. > > I was able to crash my machine over the weekend, building openjdk, but I > haven't been able to reproduce it for ~24 hours now (I didn't change > anything). I made another experiment and upgraded the host to

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-11-01 Thread Michael Ellerman
John Paul Adrian Glaubitz writes: > Hi Michael! > > On 10/28/21 08:39, Michael Ellerman wrote: >> That completed fine on my BE VM here. >> >> I ran these in two tmux windows: >> $ sbuild -d sid --arch=powerpc --no-arch-all gcc-11_11.2.0-10.dsc >> $ sbuild -d sid --arch=ppc64 --no-arch-all gcc

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-30 Thread John Paul Adrian Glaubitz
Hi Michael! On 10/28/21 08:39, Michael Ellerman wrote: > That completed fine on my BE VM here. > > I ran these in two tmux windows: > $ sbuild -d sid --arch=powerpc --no-arch-all gcc-11_11.2.0-10.dsc > $ sbuild -d sid --arch=ppc64 --no-arch-all gcc-11_11.2.0-10.dsc Could you try gcc-10 inste

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-29 Thread John Paul Adrian Glaubitz
Hi Nicholas! On 10/29/21 02:41, Nicholas Piggin wrote: > Soft lockup should mean it's taking timer interrupts still, just not > scheduling. Do you have the hard lockup detector enabled as well? Is > there anything stuck spinning on another CPU? I haven't enabled it. But looking at the documentat

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-28 Thread Nicholas Piggin
Excerpts from John Paul Adrian Glaubitz's message of October 29, 2021 12:05 am: > Hi Michael! > > On 10/28/21 13:20, John Paul Adrian Glaubitz wrote: >> It seems I also can no longer reproduce the issue, even when building the >> most problematic >> packages and I think we should consider it fixe

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-28 Thread John Paul Adrian Glaubitz
Hi! On 10/28/21 16:05, John Paul Adrian Glaubitz wrote: > The following packages were being built at the same time: > > - guest 1: virtuoso-opensource and openturns > - guest 2: llvm-toolchain-13 > > I really did a lot of testing today with no issues and just after I sent my > report > to oss-s

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-28 Thread John Paul Adrian Glaubitz
Hi Michael! On 10/28/21 13:20, John Paul Adrian Glaubitz wrote: > It seems I also can no longer reproduce the issue, even when building the > most problematic > packages and I think we should consider it fixed for now. I will keep > monitoring the server, > of course, and will let you know in ca

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-28 Thread John Paul Adrian Glaubitz
Hello! On 10/28/21 15:52, John Paul Adrian Glaubitz wrote: > I am not sure what triggered my previous crash but I don't think it's related > to this > particular bug. I will keep monitoring the server in any case and open a new > bug report > in case I'm running into similar issues. This is ver

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-28 Thread John Paul Adrian Glaubitz
Hello! An update to this post with oss-security CC'ed. On 10/26/21 10:48, John Paul Adrian Glaubitz wrote: > I have tested these patches against 5.14 but it seems the problem [1] still > remains for me > for big-endian guests. I built a patched kernel yesterday, rebooted the KVM > server and le

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-28 Thread John Paul Adrian Glaubitz
Hi Michael! On 10/28/21 08:39, Michael Ellerman wrote: >>> No, I will try that now. > > That completed fine on my BE VM here. > > I ran these in two tmux windows: > $ sbuild -d sid --arch=powerpc --no-arch-all gcc-11_11.2.0-10.dsc > $ sbuild -d sid --arch=ppc64 --no-arch-all gcc-11_11.2.0-10

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-27 Thread Michael Ellerman
[ Dropping oss-security from Cc] John Paul Adrian Glaubitz writes: > On 10/27/21 13:06, Michael Ellerman wrote: >> John Paul Adrian Glaubitz writes: >>> On 10/27/21 07:30, Michael Ellerman wrote: I did test the repro case you gave me before (in the bugzilla), which was building glibc,

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-27 Thread Michael Ellerman
John Paul Adrian Glaubitz writes: > Hi Michael! > > On 10/27/21 07:30, Michael Ellerman wrote: >> I did test the repro case you gave me before (in the bugzilla), which >> was building glibc, that passes for me with a patched host. > > Did you manage to crash the unpatched host? Yes, the parallel

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-27 Thread John Paul Adrian Glaubitz
Hi Michael! On 10/27/21 13:06, Michael Ellerman wrote: > John Paul Adrian Glaubitz writes: >> Hi Michael! >> >> On 10/27/21 07:30, Michael Ellerman wrote: >>> I did test the repro case you gave me before (in the bugzilla), which >>> was building glibc, that passes for me with a patched host. >> >

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-27 Thread John Paul Adrian Glaubitz
Hi Michael! On 10/27/21 07:30, Michael Ellerman wrote: > I did test the repro case you gave me before (in the bugzilla), which > was building glibc, that passes for me with a patched host. Did you manage to crash the unpatched host? If the unpatched host crashes for you but the patched doesn't, I

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-26 Thread Michael Ellerman
John Paul Adrian Glaubitz writes: > Hi Michael! Hi Adrian, Thanks for testing ... >> The Linux kernel for powerpc since v5.2 has a bug which allows a >> malicious KVM guest to crash the host, when the host is running on >> Power8. >> >> Only machines using Linux as the hypervisor, aka. KVM, po

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-26 Thread Nicholas Piggin
Excerpts from John Paul Adrian Glaubitz's message of October 26, 2021 6:48 pm: > Hi Michael! > >> The Linux kernel for powerpc since v5.2 has a bug which allows a >> malicious KVM guest to crash the host, when the host is running on >> Power8. >> >> Only machines using Linux as the hypervisor, ak

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-26 Thread John Paul Adrian Glaubitz
Hi Michael! > The Linux kernel for powerpc since v5.2 has a bug which allows a > malicious KVM guest to crash the host, when the host is running on > Power8. > > Only machines using Linux as the hypervisor, aka. KVM, powernv or bare > metal, are affected by the bug. Machines running PowerVM are n