> On 2022/Apr/26, at 01:01, Sergei Trofimovich <sly...@gmail.com> wrote:
>
> On Tue, 26 Apr 2022 02:43:00 +0000
> Pedro Miguel Justo <pm...@texair.net> wrote:
>
>>> On 2022/Apr/25, at 14:27, Pedro Miguel Justo <pm...@texair.net> wrote:
>>>
>>>
>>>
>>>> On 2022/Apr/25, at 14:09, Sergei Trofimovich <sly...@gmail.com> wrote:
>>>>
>>>> On Mon, 25 Apr 2022 15:07:58 +0000
>>>> Pedro Miguel Justo <pm...@texair.net> wrote:
>>>>
>>>>>> On 2022/Apr/25, at 01:22, Pedro Miguel Justo <pm...@texair.net> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> On 2022/Apr/25, at 01:14, Frank Scheiner <frank.schei...@web.de> wrote:
>>>>>>>
>>>>>>> Hi guys,
>>>>>>>
>>>>>>> On 25.04.22 10:09, John Paul Adrian Glaubitz wrote:
>>>>>>>>> From what I can understand by the information in the bugcheck, this
>>>>>>>>> is somewhat related to a violation
>>>>>>>>> in parameter copy from user to kernel during some boot-time, crypto,
>>>>>>>>> self-test. Does that sound right?
>>>>>>>>> If that is the case, how would this be related to FW?
>>>>>>>>
>>>>>>>> I'm not claiming that it must be related to the firmware, I'm just
>>>>>>>> saying that I don't see this problem
>>>>>>>> on my RX2660 at all and I have even reinstalled it recently with one
>>>>>>>> of the latest firmware images
>>>>>>>> without having to pass any parameter to the command line.
>>>>>>>
>>>>>>> A difference between Adrian's rx2660 and Pedro's rx2660 is Montecito
>>>>>>> left and Montvale right.
>>>>>>>
>>>>>>> But could still be multiple other reasons we haven't looked at yet in
>>>>>>> detail:
>>>>>>>
>>>>>>> * amount of memory installed
>>>>>>> * SMT enabled or not
>>>>>>> * number of processor modules installed
>>>>>>>
>>>>>>> It might be possible for me to check on my rx2660s (one with Montvale
>>>>>>> and one with Montecito(s)) tomorrow. I will then also look at my other
>>>>>>> Itanium gear and gather relevant information.
>>>>>>>
>>>>>>
>>>>>> Yes, this sounds mode likely to me too.
>>>>>>
>>>>>> The crypto self-tests seem to be an innocent bystander here. I tried
>>>>>> booting the most recent kernel with the option “cryptomgr.notests” and
>>>>>> it went much farther. Alas it still failed with another buffer copy
>>>>>> validation for a different caller altogether:
>>>>>>
>>>>>> [ 3.836466] [<a000000101353690>] usercopy_abort+0x120/0x130
>>>>>> [ 3.836466] sp=e0000001000cfdf0
>>>>>> bsp=e0000001000c9388
>>>>>> [ 3.836466] [<a0000001004c5660>] __check_object_size+0x3c0/0x420
>>>>>> [ 3.836466] sp=e0000001000cfe00
>>>>>> bsp=e0000001000c9350
>>>>>> [ 3.836466] [<a000000100570030>] sys_getcwd+0x250/0x420
>>>>>> [ 3.836466] sp=e0000001000cfe00
>>>>>> bsp=e0000001000c92c8
>>>>>> [ 3.836466] [<a00000010000c860>] ia64_ret_from_syscall+0x0/0x20
>>>>>> [ 3.836466] sp=e0000001000cfe30
>>>>>> bsp=e0000001000c92c8
>>>>>> [ 3.836466] [<a000000000040720>] ia64_ivt+0xffffffff00040720/0x400
>>>>>> [ 3.836466] sp=e0000001000d0000
>>>>>> bsp=e0000001000c92c8
>>>>>>
>>>>>> This suggests the bug might be in the logic validating these buffers
>>>>>> against the allocations (heap, span, etc).
>>>>>>
>>>>>> I don’t know why hardened_usercopy=off is not being observed by the
>>>>>> kernel. As a work-around I am copying myself a new kernel with
>>>>>> CONFIG_HARDENED_USERCOPY disabled at the source.
>>
>> So, I finished compiling my kernel with CONFIG_HARDENED_USERCOPY disabled.
>> Guess what:
>>
>> pmsjt@debian:~$ uname -a
>> Linux debian 5.17.3-rt17 #2 SMP Mon Apr 25 16:55:00 PDT 2022 ia64 GNU/Linux
>>
>> Yup, the system starts just fine with the most recent kernel. So, two things
>> we can infer from this:
>> - Yes, usercopy validation appears to be broken. The contours of how broken
>> it is are yet unknown but we’ll have to investigate to see what part of the
>> validation is failing.
>
> My vague memory tells me that the breakage mechanics is the following:
>
> 1. [ok] mm/usercopy.c:check_kernel_text_object() gets called on various
> kernel addresses
> 2. [ok] ia64 address space has at least two widely used kernel addresses:
> - 0xe0000000_00000000-0xe000..ff-ffffffff - proper linear mapping,
> 0xe0000000_00000000 + phys
> (so called, RGN_KERNEL = 7)
> - 0xa0000001_00000000-0xa0000000_00ffffff - kernel ~linear mapping,
> 0xa0000001_00000000 + phys
> (so called, RGN_GATE = 5)
>
> Both are easy to translate to proper linear address, but the translations
> are not identical. Note that RGN_GATE has an offset.
>
> ia64's stack for some reason lives in RGN_KERNEL space (probably due to
> historical reasons before kernel switched to vmapped stacks on other arches).
> You can see these addresses in backtraces as
> sp=e0000001000cfdf0 bsp=e0000001000c9388
>
> ia64's kernel code on the other hand lives in RGN_GATE.
> You can see these addresses in backtraces as
> [<a000000101353690>] usercopy_abort+0x120/0x130
>
> Now, check_kernel_text_object() compares passed kernel address for 2 ranges:
> - RGN_GATE:
> unsigned long textlow = (unsigned long)_stext;
> unsigned long texthigh = (unsigned long)_etext;
> if (overlaps(ptr, n, textlow, texthigh))
> usercopy_abort("kernel text", NULL, to_user, ptr - textlow, n);
>
> I think this check is correct. There is no address translation.
>
> - RGN_KERNEL:
> textlow_linear = (unsigned long)lm_alias(textlow);
> texthigh_linear = (unsigned long)lm_alias(texthigh);
> if (overlaps(ptr, n, textlow_linear, texthigh_linear))
> usercopy_abort("linear kernel text", NULL, to_user,...
>
> I think this one is wrong. Looking at the definition of lm_alias()
> it can ge used only on RGN_KERNEL addresses (which looks useless,
> they are already in correct form) and literal symbol names:
>
> include/linux/mm.h:#define lm_alias(x) __va(__pa_symbol(x))
> include/linux/mm.h:#define __pa_symbol(x) __pa(RELOC_HIDE((unsigned
> long)(x), 0))
> arch/ia64/include/asm/page.h:#define __pa(x) ({ia64_va _v;
> _v.l = (long) (x); _v.f.reg = 0; _v.l;})
> #define __va(x) ({ia64_va _v; _v.l = (long) (x); _v.f.reg =
> -1; _v.p;})
>
> I think that means __pa_symbol(x) against local variables like textlow
> is completely broken.
>
> If lm_alias() works at all on ia64 maybe that is enough to unbreak usercoy:
>
> --- a/mm/usercopy.c
> +++ b/mm/usercopy.c
> @@ -133,13 +133,13 @@ static inline void check_kernel_text_object(const
> unsigned long ptr,
> * __pa() is not just the reverse of __va(). This can be detected
> * and checked:
> */
> - textlow_linear = (unsigned long)lm_alias(textlow);
> + textlow_linear = (unsigned long)lm_alias(_stext);
> /* No different mapping: we're done. */
> if (textlow_linear == textlow)
> return;
>
> /* Check the secondary mapping... */
> - texthigh_linear = (unsigned long)lm_alias(texthigh);
> + texthigh_linear = (unsigned long)lm_alias(_etext);
> if (overlaps(ptr, n, textlow_linear, texthigh_linear))
> usercopy_abort("linear kernel text", NULL, to_user,
> ptr - textlow_linear, n);
>
> If not here is an ia64-specific workaround:
>
> --- a/mm/usercopy.c
> +++ b/mm/usercopy.c
> @@ -133,13 +133,13 @@ static inline void check_kernel_text_object(const
> unsigned long ptr,
> * __pa() is not just the reverse of __va(). This can be detected
> * and checked:
> */
> - textlow_linear = (unsigned long)lm_alias(textlow);
> + textlow_linear = (unsigned long)ia64_imva(textlow);
> /* No different mapping: we're done. */
> if (textlow_linear == textlow)
> return;
>
> /* Check the secondary mapping... */
> - texthigh_linear = (unsigned long)lm_alias(texthigh);
> + texthigh_linear = (unsigned long)ia64_imva(texthigh);
> if (overlaps(ptr, n, textlow_linear, texthigh_linear))
> usercopy_abort("linear kernel text", NULL, to_user,
> ptr - textlow_linear, n);
>
> CAVEAT: I did not compile- or run- test it. Might need a bit of pointer/long
> casting around.
>
I don’t mind at all giving this a try. Sounds quite plausible.
Do you think the stacks (and RSE) will eventually have to move from RGN_KERNEL
to vmapped the same as other archs to avoid issues like this and other to come?
>> - hardened_usercopy=off seems to be ignored by current kernels. When passing
>> this option the system was still failing just the same.
>
> I wonder how you are passing that flag to the kernel. Just a grub parameter
> to kernel?
> Worth specifying it first and last to make sure if does not get eaten by
> intermediate
> flags (in case there is a parser bug).
>
>>>>>>
>>>>>
>>>>> Even with kernel "Linux debian 4.19.0-5-mckinley #1 SMP Debian 4.19.37-5
>>>>> (2019-06-19) ia64 GNU/Linux"
>>>>>
>>>>> Things are still not 100%. After a few hours into building the kernel it
>>>>> started crashing also with usercopy validations but, this time, the other
>>>>> way around. And because it was the other way around, it led to process
>>>>> termination instead of full-blown bugcheck. This could be related or not.
>>>>> Coule very well be a different bug that happens to manifest itself round
>>>>> the same validation.
>>>>>
>>>>> CC [M] drivers/net/wireless/realtek/rtw88/rtw8822be.o
>>>>> LD [M] drivers/net/wireless/realtek/rtw88/rtw88_8822be.o
>>>>> CC [M] drivers/net/wireless/realtek/rtw88/rtw8822c.o
>>>>> Segmentation fault
>>>>> make[5]: *** [scripts/Makefile.build:293:
>>>>> drivers/net/wireless/realtek/rtw88/rtw8822c.o] Error 139
>>>>> make[5]: *** Deleting file 'drivers/net/wireless/realtek/rtw88/rtw8822c.o'
>>>>> make[4]: *** [scripts/Makefile.build:555:
>>>>> drivers/net/wireless/realtek/rtw88] Error 2
>>>>> make[3]: *** [scripts/Makefile.build:555: drivers/net/wireless/realtek]
>>>>> Error 2
>>>>> make[2]: *** [scripts/Makefile.build:555: drivers/net/wireless] Error 2
>>>>> make[1]: *** [scripts/Makefile.build:555: drivers/net] Error 2
>>>>> make: *** [Makefile:1855: drivers] Error 2
>>>>> pmsjt@debian:~/linux-source-5.17$ make
>>>>>
>>>>> Message from syslogd@debian at Apr 25 07:58:08 ...
>>>>> kernel:[23420.984012] usercopy: Kernel memory overwrite attempt detected
>>>>> to linear kernel text (offset 1916912, size 8)!
>>>>>
>>>>> Message from syslogd@debian at Apr 25 07:58:08 ...
>>>>> kernel:[23421.268009] usercopy: Kernel memory overwrite attempt detected
>>>>> to linear kernel text (offset 1818608, size 8)!
>>>>> HOSTCC scripts/sign-file
>>>>> CALL scripts/checksyscalls.sh
>>>>> <stdin>:1517:2: warning: #warning syscall clone3 not implemented [-Wcpp]
>>>>> CALL scripts/atomic/check-atomics.sh
>>>>> CHK include/generated/compile.h
>>>>> make[2]: *** [scripts/Makefile.build:294: arch/ia64/kernel/signal.o]
>>>>> Segmentation fault
>>>>>
>>>>> Message from syslogd@debian at Apr 25 07:58:11 ...
>>>>> kernel:[23423.626254] usercopy: Kernel memory overwrite attempt detected
>>>>> to linear kernel text (offset 1933296, size 8)!
>>>>> make[1]: *** [scripts/Makefile.build:555: arch/ia64/kernel] Error 2
>>>>> make: *** [Makefile:1855: arch/ia64] Error 2
>>>>
>>>
>>> Hi Sergei
>>>
>>>> In my understanding hardened_usercopy=on is completely broken on ia64
>>>> today. It can't run any userspace. Even init process would not survive
>>>> machine boot. At least that's what I experienced on rx3600.
>>>>
>>>> Thus I think if your system survives that much time I would guess
>>>> that you have hardened_usercopy=off in full effect at least at boot.
>>>>
>>>
>>> I want to make sure there is no confusion here. My system only ’survives’
>>> this much when I am using the 4.19 kernel (even when the
>>> hardened_usercopy=off is not present). With kernels more recent than that
>>> the system will bugcheck very early on boot even if hardened_usercopy=off
>>> is present.
>>>
>>>> I would speculate it's some kind of memory corruption around
>>>> 'bypass_usercopy_checks' key.
>>>>
>>>> Worth adding a few printk()s to mm/usercopy.c into 'usercopy_abort()'
>>>> and into 'set_hardened_usercopy()' just to make sure
>>>> 'bypass_usercopy_checks'
>>>> has expected 'true' setting at boot time and at crash time.
>>>
>>> Right - we definitively need more context about what is the root cause and
>>> characteristics of the bug. When the failure happens, is the (pointer,
>>> range) of the copy really out-of-whack, or is the validation code not
>>> making sense of the boundaries and over-actively failing.
>>>
>>>>
>>>> --
>>>>
>>>> Sergei
>>>
>>
>
>
> --
>
> Sergei
