This introduces a new capability for BPF program JIT's to be located in vmalloc space on x86_64. This can serve as a backup area for CONFIG_BPF_JIT_ALWAYS_ON in case an unprivileged app uses all of the module space allowed by bpf_jit_limit.
In order to allow for calls from the increased distance of vmalloc from kernel/module space, relative calls are emitted as full indirect calls if the maximum relative call distance is exceeded. So the resulting performance of call BPF instructions in this case is similar to the BPF interpreter. Below is the results for the BPF unit test "BPF_MAXINSNS: Call heavy transformations": Config Duration 1 Duration 2 -------------------------------------------------- JIT Modules 13304 14224 JIT Vmalloc 33711 26877 Interpreter 31931 28180 Retpoline JIT Modules 14517 13375 Retpoline JIT Vmalloc 65739 60281 Retpoline Interpreter 63097 61141 Since this benchmark is made up of only calls to the kernel text, it suggests real world performance shouldn't ever be worse than the interpreter. The logic for when to use vmalloc, is that we use module space unless it is full, or, we are not CAP_SYS_ADMIN and bpf_jit_limit is exceeded. So vmalloc is only used when either the insertion would fail, or BPF would fallback to the interpreter. So there should be no run time regression when used in these scenarios. Todo: - This passes all the unit tests, but could use further verification that the compiler will still always converge. - Fix ARM after "bpf: Charge bpf jit limit in bpf_jit_alloc_exec" change - Update documentation for bpf_jit_limit Possible future optimizations: - Look at re-mapping some text in vmalloc so that calls can be in relative jump range. For example, a BPF library program could maybe be re-mapped multiple times so that a copy is always near the caller and so we could use the faster calls. Rick Edgecombe (4): bpf, x64: Implement BPF call retpoline bpf, x64: Increase distance for bpf calls bpf: Charge bpf jit limit in bpf_jit_alloc_exec bpf, x64: Enable unprivlidged jit in vmalloc arch/x86/include/asm/nospec-branch.h | 45 +++++++- arch/x86/net/bpf_jit_comp.c | 149 ++++++++++++++++++++++----- include/linux/filter.h | 3 + kernel/bpf/core.c | 20 ++-- 4 files changed, 183 insertions(+), 34 deletions(-) -- 2.17.1
