[PATCH 0/1] Add usercopy fixup accuracy tests

these tests, and any other suggestions would also be greatly appreciated. This was used in conjunction with lib/test_user_copy.c to help verify the following usercopy fixup patches: https://lore.kernel.org/lkml/20200914150958.2200-1-oli.sw...@arm.com/ Thanks in advance, Oli Oliver Swede (1

[PATCH 1/1] kselftest: arm64: Add usercopy fixup accuracy tests

that the user buffer base address is set to be close to an invalid page so the copy intentionally faults at the specified index. The return value (number of bytes not copied due to the fault) is then compared against the number of bytes remaining in the user buffer. Signed-off-by: Oliver Swede

[PATCH v5 04/14] arm64: Import latest version of Cortex Strings' memmove

d-off-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/lib/memmove.S | 232 +-- 1 file changed, 78 insertions(+), 154 deletions(-) diff --git a/arch/arm64/lib/memmove.S b/arch/arm64/lib/memmove.S index 02cda2e33bde..d0977d0ad745 100644 --- a/arch/arm6

[PATCH v5 03/14] arm64: Import latest version of Cortex Strings' memcmp

ff-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/lib/memcmp.S | 333 ++-- 1 file changed, 117 insertions(+), 216 deletions(-) diff --git a/arch/arm64/lib/memcmp.S b/arch/arm64/lib/memcmp.S index c0671e793ea9..580dd0b12ccb 100644 --- a/arch/arm6

[PATCH v5 01/14] arm64: Allow passing fault address to fixup handlers

o separate patch, use UL(), expand commit message ] Signed-off-by: Robin Murphy [ os: move insertion to condition block for rebase onto bpf changes] Signed-off-by: Oliver Swede --- arch/arm64/include/asm/assembler.h | 9 + arch/arm64/include/asm/extable.h | 11 ++- arc

[PATCH v5 06/14] arm64: Import latest version of Cortex Strings' strlen

ff-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/lib/strlen.S | 247 +++- 1 file changed, 168 insertions(+), 79 deletions(-) diff --git a/arch/arm64/lib/strlen.S b/arch/arm64/lib/strlen.S index ee3ed882dd79..974b67dcc186 100644 --- a/arch/arm6

[PATCH v5 02/14] arm64: kprobes: Drop open-coded exception fixup

world is already very on fire. Thus we can reasonably drop the call from kprobe_fault_handler() and leave uaccess fixups to the regular flow. Signed-off-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/kernel/probes/kprobes.c | 7 --- 1 file changed, 7 deletions(-) diff --git a/

[PATCH v5 14/14] arm64: usercopy: Reduce overhead in fixup

re-evaluated when importing new optimized copy routines to determine if the property still holds, or e.g. if N needs to be increased, to ensure the fixup remains precise. Signed-off-by: Oliver Swede --- arch/arm64/lib/copy_user_fixup.S | 16 1 file changed, 16 insertions(+) diff

[PATCH v5 13/14] arm64: usercopy: Add conclusive fixup routine

cases. Signed-off-by: Oliver Swede --- arch/arm64/lib/copy_user_fixup.S | 13 +++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/arch/arm64/lib/copy_user_fixup.S b/arch/arm64/lib/copy_user_fixup.S index 6a7b2406d948..4858edd55994 100644 --- a/arch/arm64/lib

[PATCH v5 12/14] arm64: usercopy: Add intermediate fixup routine

fixup from the second instruction fault. Signed-off-by: Oliver Swede --- arch/arm64/lib/copy_from_user.S | 18 ++- arch/arm64/lib/copy_in_user.S | 16 ++ arch/arm64/lib/copy_template_user.S | 2 ++ arch/arm64/lib/copy_to_user.S | 16 ++ arch/arm64

[PATCH v5 10/14] arm64: usercopy: Store the arguments on stack

pointer is restored to its initial position either from the fixup code in the case of a fault, or at the end of the copy algorithm otherwise. The .Luaccess_finish directive is also moved to copy_template_user.S as the code is common to all usercopy functions. Signed-off-by: Oliver Swede --- arch/arm64

[PATCH v5 09/14] arm64: Tidy up _asm_extable_faultaddr usage

From: Robin Murphy To match the way the USER() shorthand wraps _asm_extable entries, introduce USER_F() to wrap _asm_extable_faultaddr and clean up a bit. Signed-off-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/include/asm/assembler.h | 4 ++ arch/arm64/lib/copy_from_user.S

[PATCH v5 07/14] arm64: Import latest version of Cortex Strings' strncmp

ff-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/lib/strncmp.S | 363 ++- 1 file changed, 163 insertions(+), 200 deletions(-) diff --git a/arch/arm64/lib/strncmp.S b/arch/arm64/lib/strncmp.S index 2a7ee949ed47..b954e0fd93be 100644 --- a/arch/arm6

[PATCH v5 11/14] arm64: usercopy: Check for overlapping buffers in fixup

for explicitly from within the routines themselves), and independent of any specific implementation, it should be suitable to return the full copy width back to the kernel code path calling the usercopy function. Signed-off-by: Oliver Swede --- arch/arm64/lib/copy_from_user.S | 24

[PATCH v5 08/14] arm64: Import latest optimization of memcpy

4c175c8be12 in https://github.com/ARM-software/optimized-routines. Signed-off-by: Sam Tebbs [ rm: add UAO fixups, streamline copy_exit paths, expand commit message ] Signed-off-by: Robin Murphy [ os: import newer memcpy algorithm, update commit message ] Signed-off-by: Oliver Swede --- arch/arm64/inclu

[PATCH v5 00/14] Optimise and update memcpy, user copy and string routines

s 9-14 for clarity and to reflect the new changes. This revision was tested on two machines (UAO & non-UAO) internally using a custom test module (planning on posting this shortly). v4: https://lore.kernel.org/linux-arm-kernel/f52401d9-787c-667b-c1ec-8b91106d6...@arm.com/ Oliver Swede (5): a

[PATCH v5 05/14] arm64: Import latest version of Cortex Strings' strcmp

ff-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/lib/strcmp.S | 272 +--- 1 file changed, 113 insertions(+), 159 deletions(-) diff --git a/arch/arm64/lib/strcmp.S b/arch/arm64/lib/strcmp.S index 4e79566726c8..e00ff46c4ffc 100644 --- a/arch/arm6

[PATCH v4 03/14] arm64: Import latest version of Cortex Strings' memcmp

ff-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/lib/memcmp.S | 333 ++-- 1 file changed, 117 insertions(+), 216 deletions(-) diff --git a/arch/arm64/lib/memcmp.S b/arch/arm64/lib/memcmp.S index c0671e793ea9..580dd0b12ccb 100644 --- a/arch/arm6

[PATCH v4 07/14] arm64: Import latest version of Cortex Strings' strncmp

ff-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/lib/strncmp.S | 363 ++- 1 file changed, 163 insertions(+), 200 deletions(-) diff --git a/arch/arm64/lib/strncmp.S b/arch/arm64/lib/strncmp.S index 2a7ee949ed47..b954e0fd93be 100644 --- a/arch/arm6

[PATCH v4 05/14] arm64: Import latest version of Cortex Strings' strcmp

ff-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/lib/strcmp.S | 272 +--- 1 file changed, 113 insertions(+), 159 deletions(-) diff --git a/arch/arm64/lib/strcmp.S b/arch/arm64/lib/strcmp.S index 4e79566726c8..e00ff46c4ffc 100644 --- a/arch/arm6

[PATCH v4 12/14] arm64: Add fixup routines for usercopy load exceptions

requirement to follow through with the copying of data that may reside in temporary registers on a fault, as this would greatly increase the fixup's complexity. Signed-off-by: Oliver Swede --- arch/arm64/lib/copy_user_fixup.S | 172 ++- 1 file changed, 168 inser

[PATCH v4 11/14] arm64: Use additional memcpy macros and fixups

from analysing the copy algorithm) enable fixups to be written that are modular and accurate for each case. The fixup logic should be straightforward to modify in the future, e.g. if there are further improvements to the memcpy routine. Signed-off-by: Oliver Swede --- arch/arm64/lib/copy_from_u

[PATCH v4 09/14] arm64: Tidy up _asm_extable_faultaddr usage

From: Robin Murphy To match the way the USER() shorthand wraps _asm_extable entries, introduce USER_F() to wrap _asm_extable_faultaddr and clean up a bit. Signed-off-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/include/asm/assembler.h | 4 ++ arch/arm64/lib/copy_from_user.S

[PATCH v4 01/14] arm64: Allow passing fault address to fixup handlers

o separate patch, use UL(), expand commit message ] Signed-off-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/include/asm/assembler.h | 9 + arch/arm64/include/asm/extable.h | 10 +- arch/arm64/mm/extable.c| 13 + arch/arm64/

[PATCH v4 14/14] arm64: Improve accuracy of fixup for UAO cases

alternatives are inserted in-line. Signed-off-by: Oliver Swede --- arch/arm64/lib/copy_user_fixup.S | 47 1 file changed, 41 insertions(+), 6 deletions(-) diff --git a/arch/arm64/lib/copy_user_fixup.S b/arch/arm64/lib/copy_user_fixup.S index 37ca3d99a02a..2d413f9ba5d3

[PATCH v4 02/14] arm64: kprobes: Drop open-coded exception fixup

world is already very on fire. Thus we can reasonably drop the call from kprobe_fault_handler() and leave uaccess fixups to the regular flow. Signed-off-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/kernel/probes/kprobes.c | 7 --- 1 file changed, 7 deletions(-) diff --git a/

[PATCH v4 08/14] arm64: Import latest optimization of memcpy

sage ] Signed-off-by: Oliver Swede --- arch/arm64/include/asm/alternative.h | 36 --- arch/arm64/lib/copy_from_user.S | 113 ++-- arch/arm64/lib/copy_in_user.S| 129 +++-- arch/arm64/lib/copy_template.S | 375 +++ arch/arm64/lib/copy_template_us

[PATCH v4 10/14] arm64: Store the arguments to copy_*_user on the stack

). Signed-off-by: Oliver Swede --- arch/arm64/lib/copy_from_user.S | 3 --- arch/arm64/lib/copy_in_user.S | 3 --- arch/arm64/lib/copy_template_user.S | 6 ++ arch/arm64/lib/copy_to_user.S | 3 --- arch/arm64/lib/copy_user_fixup.S| 1 + 5 files changed, 7 insertions(+), 9

[PATCH v4 13/14] arm64: Add fixup routines for usercopy store exceptions

is specific to the new copy template, which uses the latest memcpy implementation. Signed-off-by: Oliver Swede --- arch/arm64/lib/copy_user_fixup.S | 217 ++- 1 file changed, 215 insertions(+), 2 deletions(-) diff --git a/arch/arm64/lib/copy_user_fixup.S b/arch/arm64

[PATCH v4 06/14] arm64: Import latest version of Cortex Strings' strlen

ff-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/lib/strlen.S | 247 +++- 1 file changed, 168 insertions(+), 79 deletions(-) diff --git a/arch/arm64/lib/strlen.S b/arch/arm64/lib/strlen.S index ee3ed882dd79..974b67dcc186 100644 --- a/arch/arm6

[PATCH v4 04/14] arm64: Import latest version of Cortex Strings' memmove

d-off-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/lib/memmove.S | 232 +-- 1 file changed, 78 insertions(+), 154 deletions(-) diff --git a/arch/arm64/lib/memmove.S b/arch/arm64/lib/memmove.S index 02cda2e33bde..d0977d0ad745 100644 --- a/arch/arm6

[PATCH v4 00/14] arm64: Optimise and update memcpy, user copy and string routines

se of the fault address to return the exact number of bytes that haven't yet copied. [1] https://lore.kernel.org/linux-arm-kernel/e70f7b9de7e601b9e4a6fedad8eaf64d304b1637.1571326276.git.robin.mur...@arm.com/ Oliver Swede (5): arm64: Store the arguments to copy_*_user on the stack a

[PATCH v3 02/13] arm64: kprobes: Drop open-coded exception fixup

world is already very on fire. Thus we can reasonably drop the call from kprobe_fault_handler() and leave uaccess fixups to the regular flow. Signed-off-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/kernel/probes/kprobes.c | 7 --- 1 file changed, 7 deletions(-) diff --git a/

[PATCH v3 06/13] arm64: Import latest version of Cortex Strings' strlen

ff-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/lib/strlen.S | 247 +++- 1 file changed, 168 insertions(+), 79 deletions(-) diff --git a/arch/arm64/lib/strlen.S b/arch/arm64/lib/strlen.S index ee3ed882dd79..974b67dcc186 100644 --- a/arch/arm6

[PATCH v3 04/13] arm64: Import latest version of Cortex Strings' memmove

d-off-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/lib/memmove.S | 232 +-- 1 file changed, 78 insertions(+), 154 deletions(-) diff --git a/arch/arm64/lib/memmove.S b/arch/arm64/lib/memmove.S index 02cda2e33bde..d0977d0ad745 100644 --- a/arch/arm6

[PATCH v3 09/13] arm64: Tidy up _asm_extable_faultaddr usage

From: Robin Murphy To match the way the USER() shorthand wraps _asm_extable entries, introduce USER_F() to wrap _asm_extable_faultaddr and clean up a bit. Signed-off-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/include/asm/assembler.h | 4 ++ arch/arm64/lib/copy_from_user.S

[PATCH v3 05/13] arm64: Import latest version of Cortex Strings' strcmp

ff-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/lib/strcmp.S | 272 +--- 1 file changed, 113 insertions(+), 159 deletions(-) diff --git a/arch/arm64/lib/strcmp.S b/arch/arm64/lib/strcmp.S index 4e79566726c8..e00ff46c4ffc 100644 --- a/arch/arm6

[PATCH v3 03/13] arm64: Import latest version of Cortex Strings' memcmp

ff-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/lib/memcmp.S | 333 ++-- 1 file changed, 117 insertions(+), 216 deletions(-) diff --git a/arch/arm64/lib/memcmp.S b/arch/arm64/lib/memcmp.S index c0671e793ea9..580dd0b12ccb 100644 --- a/arch/arm6

[PATCH v3 10/13] arm64: Store the arguments to copy_*_user on the stack

). Signed-off-by: Oliver Swede --- arch/arm64/lib/copy_from_user.S | 3 --- arch/arm64/lib/copy_in_user.S | 3 --- arch/arm64/lib/copy_template_user.S | 6 ++ arch/arm64/lib/copy_to_user.S | 3 --- arch/arm64/lib/copy_user_fixup.S| 1 + 5 files changed, 7 insertions(+), 9

[PATCH v3 00/13] arm64: Optimise and update memcpy, user copy and string routines

e usercopy fixup routine in v2 with multiple longer fixups that each make use of the fault address to return the exact number of bytes that haven't yet copied. [1] https://lore.kernel.org/linux-arm-kernel/e70f7b9de7e601b9e4a6fedad8eaf64d304b1637.1571326276.git.robin.mur...@arm.com/ M

[PATCH v3 12/13] arm64: Add fixup routines for usercopy load exceptions

requirement to follow through with the copying of data that may reside in temporary registers on a fault, as this would greatly increase the fixup's complexity. Signed-off-by: Oliver Swede --- arch/arm64/lib/copy_user_fixup.S | 170 ++- 1 file changed, 165 inser

[PATCH v3 13/13] arm64: Add fixup routines for usercopy store exceptions

is specific to the new copy template, which uses the latest memcpy implementation. Signed-off-by: Oliver Swede --- arch/arm64/lib/copy_user_fixup.S | 96 1 file changed, 96 insertions(+) diff --git a/arch/arm64/lib/copy_user_fixup.S b/arch/arm64/lib

[PATCH v3 08/13] arm64: Import latest optimization of memcpy

sage ] Signed-off-by: Oliver Swede --- arch/arm64/include/asm/alternative.h | 36 --- arch/arm64/lib/copy_from_user.S | 115 ++-- arch/arm64/lib/copy_in_user.S| 130 -- arch/arm64/lib/copy_template.S | 375 +++ arch/arm64/lib/copy_template_us

[PATCH v3 07/13] arm64: Import latest version of Cortex Strings' strncmp

ff-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/lib/strncmp.S | 363 ++- 1 file changed, 163 insertions(+), 200 deletions(-) diff --git a/arch/arm64/lib/strncmp.S b/arch/arm64/lib/strncmp.S index 2a7ee949ed47..b954e0fd93be 100644 --- a/arch/arm6

[PATCH v3 11/13] arm64: Use additional memcpy macros and fixups

lysis of the copy algorithm, enable fixups to be written that are modular and accurate for each case. In this way the fixup logic should be straightforward to modify in the future, e.g. if there are further improvements to the memcpy routine. Signed-off-by: Oliver Swede --- arch/arm6

[PATCH v3 01/13] arm64: Allow passing fault address to fixup handlers

o separate patch, use UL(), expand commit message ] Signed-off-by: Robin Murphy Signed-off-by: Oliver Swede --- arch/arm64/include/asm/assembler.h | 9 + arch/arm64/include/asm/extable.h | 10 +- arch/arm64/mm/extable.c| 13 + arch/arm64/