On Wed, Apr 26, 2017 at 09:28:06PM +0300, Alexey Dobriyan wrote:
> Avoid unconditional branch at every copy_page() call by using
> alternatives and calling optimal variant directly.
>
> Rename individual versions to immediately show which one is used in
> profiles, etc.
>
> RBX and R12 aren't clobbered because generic version restores them
> and both REP versions don't touch them.
>
> Signed-off-by: Alexey Dobriyan <adobri...@gmail.com>
> ---
>
> arch/x86/include/asm/page_64.h | 16 ++++++++++++++--
> arch/x86/lib/copy_page_64.S | 17 +++++++----------
> 2 files changed, 21 insertions(+), 12 deletions(-)
>
> --- a/arch/x86/include/asm/page_64.h
> +++ b/arch/x86/include/asm/page_64.h
> @@ -49,8 +49,20 @@ static inline void clear_page(void *page)
> : "memory", "rax", "rcx");
> }
>
> -void copy_page(void *to, void *from);
> -
> +void copy_page_mov(void *to, void *from);
> +void copy_page_rep_movsq(void *to, void *from);
> +void copy_page_rep_movsb(void *to, void *from);
<---- newline here.
> +static __always_inline void copy_page(void *to, void *from)
> +{
> + alternative_call_2(
Please align at the opening brace, like clear_page() above it:
alternative_call_2(copy_page_mov,
copy_page_rep_movsq, X86_FEATURE_REP_GOOD,
...
> + copy_page_rep_movsb, X86_FEATURE_ERMS,
> + ASM_OUTPUT2("=D" (to), "=S" (from)),
> + "0" (to), "1" (from)
> + : "rax", "rcx", "rdx", "r8", "r9", "r10", "r11", "cc", "memory"
> + );
> +}
> #endif /* !__ASSEMBLY__ */
>
> #ifdef CONFIG_X86_VSYSCALL_EMULATION
...
> ENTRY(copy_page_rep_movsb)
> mov $4096, %ecx
> rep movsb
> ret
> ENDPROC(copy_page_rep_movsb)
> +EXPORT_SYMBOL(copy_page_rep_movsb)
>
> -ENTRY(copy_page_regs)
> +ENTRY(copy_page_mov)
copy_page_regs() is a better name IMO. copy_page_mov() doesn't tell me
anything - all three use "mov". copy_page_unrolled() sounds ok too.
--
Regards/Gruss,
Boris.
Good mailing practices for 400: avoid top-posting and trim the reply.
