This series changes the xdp_redirect helper to use a hidden default map. The
redirect_map() helper also uses the map structure to batch packets, which
results in a significant (around 50%) performance boost for the _map variant.
However, the xdp_redirect() API is simpler if one just wants to redirect to
another interface, which means people tend to use this interface and then wonder
why they getter worse performance than expected.
This series seeks to close this performance difference between the two APIs. It
achieves this by changing xdp_redirect() to use a hidden devmap for looking up
destination interfaces, thus gaining the batching benefit with no visible
difference from the user API point of view.
Allocation of the default map is done dynamically as programs using the
xdp_redirect helper are loaded and attached to interfaces, and the maps are
freed again when no longer needed. Because of tail calls, this requires two
levels of refcounting: One global level that keeps track of whether any XDP
programs using the xdp_redirect() helper are loaded in the system at all. And
another that keeps track of whether any programs that could potentially result
in a call to xdp_redirect (i.e., either programs using the helper, or programs
using tail calls) are loaded in a given namespace.
The default maps are dynamically sized to the nearest power-of-two size that can
contain all interfaces present in each interface. If allocation fails, the user
action that triggered the allocation (either attaching an XDP program, or moving
an interface with a program attached) is rejected. If new interfaces appear in
a namespace which causes the default map to become too small, a new one is
allocated with the correct size; this allocation is the only one that cannot
lead to a rejection of the userspace action, so if it fails a warning is emitted
instead.
The first patch in the series refactors devmap.c to prepare for the subsequent
patches. The second patch adds the default map handling using the existing
array-based devmap structure. The third patch adds a new map type (devmap_idx)
that hashes devices on ifindex.
Changelog:
v3 -> v4:
- Add two patches from Björn to re-factor XDP program tracking for netdevs
(patches 1+2). This obviates the need for the bpf_get_prog_by_id() function
added in the previous versions.
- Move freeing and flushing of maps to a workqueue instead of an RCU callback.
- Handle replacing a program using redirect with one that doesn't (and vice
versa).
- Use a mutex instead of a spinlock for the internal devmap locking, and make
sure everything is covered by it.
- Add selftest (and /proc/net interface) for checking if the maps are allocated
correctly (patch 6).
- Fix a bunch of nits from Jakub.
v2 -> v3:
- Fix compile warnings when CONFIG_BPF_SYSCALL is unset (as pointed out by the
kbuild test bot).
v1 -> v2:
- Add refcounting to only allocate default maps when needed
- Using said refcounting, also deallocate default maps
- Add dynamic sizing of default maps
- Handle moving of interfaces between namespaces
- Split out refactoring of devmap.c to separate patch
- Use hashmap semantics for update_elem of devmap_idx type maps
---
Björn Töpel (2):
net: xdp: refactor XDP attach
net: xdp: remove XDP_QUERY_PROG
Toke Høiland-Jørgensen (4):
xdp: Refactor devmap code in preparation for subsequent additions
xdp: Always use a devmap for XDP_REDIRECT to a device
xdp: Add devmap_idx map type for looking up devices by ifindex
selftests/bpf: Add test for default devmap allocation
drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c | 4
drivers/net/ethernet/cavium/thunder/nicvf_main.c | 3
drivers/net/ethernet/freescale/dpaa2/dpaa2-eth.c | 3
drivers/net/ethernet/intel/i40e/i40e_main.c | 3
drivers/net/ethernet/intel/ixgbe/ixgbe_main.c | 4
drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c | 4
drivers/net/ethernet/mellanox/mlx4/en_netdev.c | 24 -
drivers/net/ethernet/mellanox/mlx5/core/en_main.c | 18 -
.../net/ethernet/netronome/nfp/nfp_net_common.c | 2
drivers/net/ethernet/qlogic/qede/qede_filter.c | 3
drivers/net/netdevsim/bpf.c | 2
drivers/net/netdevsim/netdevsim.h | 2
drivers/net/tun.c | 15
drivers/net/veth.c | 15
drivers/net/virtio_net.c | 17 -
include/linux/bpf.h | 46 +
include/linux/bpf_types.h | 1
include/linux/filter.h | 2
include/linux/netdevice.h | 11
include/net/net_namespace.h | 2
include/net/netns/xdp.h | 11
include/trace/events/xdp.h | 3
include/uapi/linux/bpf.h | 1
kernel/bpf/devmap.c | 668 ++++++++++++++++++--
kernel/bpf/syscall.c | 3
kernel/bpf/verifier.c | 15
net/core/dev.c | 209 ++++--
net/core/filter.c | 69 --
net/core/rtnetlink.c | 14
tools/bpf/bpftool/map.c | 1
tools/include/uapi/linux/bpf.h | 1
tools/lib/bpf/libbpf_probes.c | 1
tools/testing/selftests/bpf/Makefile | 3
.../selftests/bpf/progs/test_xdp_tail_call.c | 39 +
tools/testing/selftests/bpf/test_maps.c | 16
.../testing/selftests/bpf/test_xdp_devmap_alloc.sh | 94 +++
36 files changed, 1000 insertions(+), 329 deletions(-)
create mode 100644 tools/testing/selftests/bpf/progs/test_xdp_tail_call.c
create mode 100755 tools/testing/selftests/bpf/test_xdp_devmap_alloc.sh
