Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2022-01-27 Thread Mike
I just made the huge mistake of hibernating and resuming, I'm going trough the process of rescue and all, thankfully I had a 2016 cd in the drive. I'll read up once the sheer panic settles. -Michael On Wed, Jan 26, 2022, 21:22 John Paul Adrian Glaubitz < glaub...@physik.fu-berlin.de> wrote: > Hi

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2022-01-26 Thread John Paul Adrian Glaubitz
Hi Michael! On 1/13/22 01:17, John Paul Adrian Glaubitz wrote: > On 1/9/22 23:17, John Paul Adrian Glaubitz wrote: >> On 1/7/22 12:20, John Paul Adrian Glaubitz wrote: Can you separately test with (on the host): # echo 0 > /sys/module/kvm_hv/parameters/dynamic_mt_modes >>> >>> I'm

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2022-01-12 Thread John Paul Adrian Glaubitz
Hi Michael! On 1/9/22 23:17, John Paul Adrian Glaubitz wrote: > On 1/7/22 12:20, John Paul Adrian Glaubitz wrote: >>> Can you separately test with (on the host): >>> >>> # echo 0 > /sys/module/kvm_hv/parameters/dynamic_mt_modes >> >> I'm trying to turn off "dynamic_mt_modes" first and see if that

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2022-01-09 Thread John Paul Adrian Glaubitz
Hi Michael! On 1/7/22 12:20, John Paul Adrian Glaubitz wrote: >> Can you separately test with (on the host): >> >> # echo 0 > /sys/module/kvm_hv/parameters/dynamic_mt_modes > > I'm trying to turn off "dynamic_mt_modes" first and see if that makes any > difference. > > I will report back. So f

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2022-01-07 Thread John Paul Adrian Glaubitz
Hi Michael! On 1/6/22 11:58, Michael Ellerman wrote: >> We're currently running 5.15.x on the host system and the guests and the >> testsuite >> for gcc-9 still reproducibly kills the KVM host. > > Have you been able to try the different -smp options I suggested? > > Can you separately test wit

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2022-01-06 Thread Michael Ellerman
John Paul Adrian Glaubitz writes: > Hi Michael! > > Sorry for the long time without any responses. Shall we continue debugging > this? Yes! Sorry also that I haven't been able to fix it yet, I had to stop chasing this bug and work on other things before the end of the year. > We're currently r

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-11-01 Thread Michal Suchánek
On Fri, Oct 29, 2021 at 02:33:12PM +0200, John Paul Adrian Glaubitz wrote: > Hi Nicholas! > > On 10/29/21 02:41, Nicholas Piggin wrote: > > Soft lockup should mean it's taking timer interrupts still, just not > > scheduling. Do you have the hard lockup detector enabled as well? Is > > there anyth

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-11-01 Thread Michal Suchánek
Hello, On Thu, Oct 28, 2021 at 04:15:19PM +0200, John Paul Adrian Glaubitz wrote: > Hi! > > On 10/28/21 16:05, John Paul Adrian Glaubitz wrote: > > The following packages were being built at the same time: > > > > - guest 1: virtuoso-opensource and openturns > > - guest 2: llvm-toolchain-13 > >

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-11-01 Thread John Paul Adrian Glaubitz
Hi Michael! On 11/1/21 08:37, John Paul Adrian Glaubitz wrote: > I made another experiment and upgraded the host to 5.15-rc7 which contains > your > fixes and made the guests build gcc-10. Interestingly, this time, the gcc-10 > build crashed the guest but didn't manage to crash the host. I will u

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-11-01 Thread John Paul Adrian Glaubitz
Hi Michael! On 11/1/21 07:53, Michael Ellerman wrote: > Sure, will give that a try. > > I was able to crash my machine over the weekend, building openjdk, but I > haven't been able to reproduce it for ~24 hours now (I didn't change > anything). I made another experiment and upgraded the host to

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-31 Thread Michael Ellerman
John Paul Adrian Glaubitz writes: > Hi Michael! > > On 10/28/21 08:39, Michael Ellerman wrote: >> That completed fine on my BE VM here. >> >> I ran these in two tmux windows: >> $ sbuild -d sid --arch=powerpc --no-arch-all gcc-11_11.2.0-10.dsc >> $ sbuild -d sid --arch=ppc64 --no-arch-all gcc

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-30 Thread John Paul Adrian Glaubitz
Hi Michael! On 10/28/21 08:39, Michael Ellerman wrote: > That completed fine on my BE VM here. > > I ran these in two tmux windows: > $ sbuild -d sid --arch=powerpc --no-arch-all gcc-11_11.2.0-10.dsc > $ sbuild -d sid --arch=ppc64 --no-arch-all gcc-11_11.2.0-10.dsc Could you try gcc-10 inste

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-29 Thread John Paul Adrian Glaubitz
Hi Nicholas! On 10/29/21 02:41, Nicholas Piggin wrote: > Soft lockup should mean it's taking timer interrupts still, just not > scheduling. Do you have the hard lockup detector enabled as well? Is > there anything stuck spinning on another CPU? I haven't enabled it. But looking at the documentat

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-28 Thread Nicholas Piggin
Excerpts from John Paul Adrian Glaubitz's message of October 29, 2021 12:05 am: > Hi Michael! > > On 10/28/21 13:20, John Paul Adrian Glaubitz wrote: >> It seems I also can no longer reproduce the issue, even when building the >> most problematic >> packages and I think we should consider it fixe

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-28 Thread John Paul Adrian Glaubitz
Hi! On 10/28/21 16:05, John Paul Adrian Glaubitz wrote: > The following packages were being built at the same time: > > - guest 1: virtuoso-opensource and openturns > - guest 2: llvm-toolchain-13 > > I really did a lot of testing today with no issues and just after I sent my > report > to oss-s

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-28 Thread John Paul Adrian Glaubitz
Hi Michael! On 10/28/21 13:20, John Paul Adrian Glaubitz wrote: > It seems I also can no longer reproduce the issue, even when building the > most problematic > packages and I think we should consider it fixed for now. I will keep > monitoring the server, > of course, and will let you know in ca

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-28 Thread John Paul Adrian Glaubitz
Hello! On 10/28/21 15:52, John Paul Adrian Glaubitz wrote: > I am not sure what triggered my previous crash but I don't think it's related > to this > particular bug. I will keep monitoring the server in any case and open a new > bug report > in case I'm running into similar issues. This is ver

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-28 Thread John Paul Adrian Glaubitz
Hello! An update to this post with oss-security CC'ed. On 10/26/21 10:48, John Paul Adrian Glaubitz wrote: > I have tested these patches against 5.14 but it seems the problem [1] still > remains for me > for big-endian guests. I built a patched kernel yesterday, rebooted the KVM > server and le

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-28 Thread John Paul Adrian Glaubitz
Hi Michael! On 10/28/21 08:39, Michael Ellerman wrote: >>> No, I will try that now. > > That completed fine on my BE VM here. > > I ran these in two tmux windows: > $ sbuild -d sid --arch=powerpc --no-arch-all gcc-11_11.2.0-10.dsc > $ sbuild -d sid --arch=ppc64 --no-arch-all gcc-11_11.2.0-10

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-27 Thread Michael Ellerman
[ Dropping oss-security from Cc] John Paul Adrian Glaubitz writes: > On 10/27/21 13:06, Michael Ellerman wrote: >> John Paul Adrian Glaubitz writes: >>> On 10/27/21 07:30, Michael Ellerman wrote: I did test the repro case you gave me before (in the bugzilla), which was building glibc,

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-27 Thread John Paul Adrian Glaubitz
Hi Michael! On 10/27/21 13:06, Michael Ellerman wrote: > John Paul Adrian Glaubitz writes: >> Hi Michael! >> >> On 10/27/21 07:30, Michael Ellerman wrote: >>> I did test the repro case you gave me before (in the bugzilla), which >>> was building glibc, that passes for me with a patched host. >> >

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-27 Thread Michael Ellerman
John Paul Adrian Glaubitz writes: > Hi Michael! > > On 10/27/21 07:30, Michael Ellerman wrote: >> I did test the repro case you gave me before (in the bugzilla), which >> was building glibc, that passes for me with a patched host. > > Did you manage to crash the unpatched host? Yes, the parallel

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-27 Thread John Paul Adrian Glaubitz
Hi Michael! On 10/27/21 07:30, Michael Ellerman wrote: > I did test the repro case you gave me before (in the bugzilla), which > was building glibc, that passes for me with a patched host. Did you manage to crash the unpatched host? If the unpatched host crashes for you but the patched doesn't, I

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-26 Thread Michael Ellerman
John Paul Adrian Glaubitz writes: > Hi Michael! Hi Adrian, Thanks for testing ... >> The Linux kernel for powerpc since v5.2 has a bug which allows a >> malicious KVM guest to crash the host, when the host is running on >> Power8. >> >> Only machines using Linux as the hypervisor, aka. KVM, po

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-26 Thread Nicholas Piggin
Excerpts from John Paul Adrian Glaubitz's message of October 26, 2021 6:48 pm: > Hi Michael! > >> The Linux kernel for powerpc since v5.2 has a bug which allows a >> malicious KVM guest to crash the host, when the host is running on >> Power8. >> >> Only machines using Linux as the hypervisor, ak

Re: Linux kernel: powerpc: KVM guest can trigger host crash on Power8

2021-10-26 Thread John Paul Adrian Glaubitz
Hi Michael! > The Linux kernel for powerpc since v5.2 has a bug which allows a > malicious KVM guest to crash the host, when the host is running on > Power8. > > Only machines using Linux as the hypervisor, aka. KVM, powernv or bare > metal, are affected by the bug. Machines running PowerVM are n