Hi Maryam,
I have added a small nitpick.
Regards,
Shibin
> -----Original Message-----
> From: Maryam Tahhan <mtah...@redhat.com>
> Sent: Wednesday, February 14, 2024 5:04 PM
> To: ferruh.yi...@amd.com; step...@networkplumber.org;
> lihuis...@huawei.com; fengcheng...@huawei.com;
> liuyongl...@huawei.com; Marchand, David
> <david.march...@redhat.com>; Koikkara Reeny, Shibin
> <shibin.koikkara.re...@intel.com>; Loftus, Ciara <ciara.lof...@intel.com>
> Cc: dev@dpdk.org; Tahhan, Maryam <mtah...@redhat.com>;
> sta...@dpdk.org
> Subject: [v9 1/3] docs: AF_XDP Device Plugin
>
> Fixup the references to the AF_XDP Device Plugin in the documentation (was
> referred to as CNI previously) and document the single netdev limitation for
> deploying an AF_XDP based DPDK pod. Also renames af_xdp_cni.rst to
> af_xdp_dp.rst
>
> Fixes: 7fc6ae50369d ("net/af_xdp: support CNI Integration")
> Cc: sta...@dpdk.org
>
> Signed-off-by: Maryam Tahhan <mtah...@redhat.com>
> ---
> doc/guides/howto/af_xdp_cni.rst | 253 ---------------------------
> doc/guides/howto/af_xdp_dp.rst | 299
> ++++++++++++++++++++++++++++++++
> doc/guides/howto/index.rst | 2 +-
> doc/guides/nics/af_xdp.rst | 4 +-
> 4 files changed, 302 insertions(+), 256 deletions(-) delete mode 100644
> doc/guides/howto/af_xdp_cni.rst create mode 100644
> doc/guides/howto/af_xdp_dp.rst
>
> diff --git a/doc/guides/howto/af_xdp_cni.rst
> b/doc/guides/howto/af_xdp_cni.rst deleted file mode 100644 index
> a1a6d5b99c..0000000000
> --- a/doc/guides/howto/af_xdp_cni.rst
> +++ /dev/null
> @@ -1,253 +0,0 @@
> -.. SPDX-License-Identifier: BSD-3-Clause
> - Copyright(c) 2023 Intel Corporation.
> -
> -Using a CNI with the AF_XDP driver
> -==================================
> -
> -Introduction
> -------------
> -
> -CNI, the Container Network Interface, is a technology for configuring -
> container network interfaces -and which can be used to setup Kubernetes
> networking.
> -AF_XDP is a Linux socket Address Family that enables an XDP program -to
> redirect packets to a memory buffer in userspace.
> -
> -This document explains how to enable the `AF_XDP Plugin for Kubernetes`_
> within -a DPDK application using the :doc:`../nics/af_xdp` to connect and use
> these technologies.
> -
> -.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-
> for-kubernetes
> -
> -
> -Background
> -----------
> -
> -The standard :doc:`../nics/af_xdp` initialization process involves loading an
> eBPF program -onto the kernel netdev to be used by the PMD.
> -This operation requires root or escalated Linux privileges -and thus prevents
> the PMD from working in an unprivileged container.
> -The AF_XDP CNI plugin handles this situation -by providing a device plugin
> that performs the program loading.
> -
> -At a technical level the CNI opens a Unix Domain Socket and listens for a
> client -to make requests over that socket.
> -A DPDK application acting as a client connects and initiates a configuration
> "handshake".
> -The client then receives a file descriptor which points to the XSKMAP -
> associated with the loaded eBPF program.
> -The XSKMAP is a BPF map of AF_XDP sockets (XSK).
> -The client can then proceed with creating an AF_XDP socket -and inserting
> that socket into the XSKMAP pointed to by the descriptor.
> -
> -The EAL vdev argument ``use_cni`` is used to indicate that the user wishes -
> to run the PMD in unprivileged mode and to receive the XSKMAP file
> descriptor -from the CNI.
> -When this flag is set,
> -the ``XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD`` libbpf flag -should be
> used when creating the socket -to instruct libbpf not to load the default
> libbpf program on the netdev.
> -Instead the loading is handled by the CNI.
> -
> -.. note::
> -
> - The Unix Domain Socket file path appear in the end user is
> "/tmp/afxdp.sock".
> -
> -
> -Prerequisites
> --------------
> -
> -Docker and container prerequisites:
> -
> -* Set up the device plugin
> - as described in the instructions for `AF_XDP Plugin for Kubernetes`_.
> -
> -* The Docker image should contain the libbpf and libxdp libraries,
> - which are dependencies for AF_XDP,
> - and should include support for the ``ethtool`` command.
> -
> -* The Pod should have enabled the capabilities ``CAP_NET_RAW`` and
> ``CAP_BPF``
> - for AF_XDP along with support for hugepages.
> -
> -* Increase locked memory limit so containers have enough memory for
> packet buffers.
> - For example:
> -
> - .. code-block:: console
> -
> - cat << EOF | sudo tee
> /etc/systemd/system/containerd.service.d/limits.conf
> - [Service]
> - LimitMEMLOCK=infinity
> - EOF
> -
> -* dpdk-testpmd application should have AF_XDP feature enabled.
> -
> - For further information see the docs for the: :doc:`../../nics/af_xdp`.
> -
> -
> -Example
> --------
> -
> -Howto run dpdk-testpmd with CNI plugin:
> -
> -* Clone the CNI plugin
> -
> - .. code-block:: console
> -
> - # git clone https://github.com/intel/afxdp-plugins-for-kubernetes.git
> -
> -* Build the CNI plugin
> -
> - .. code-block:: console
> -
> - # cd afxdp-plugins-for-kubernetes/
> - # make build
> -
> - .. note::
> -
> - CNI plugin has a dependence on the config.json.
> -
> - Sample Config.json
> -
> - .. code-block:: json
> -
> - {
> - "logLevel":"debug",
> - "logFile":"afxdp-dp-e2e.log",
> - "pools":[
> - {
> - "name":"e2e",
> - "mode":"primary",
> - "timeout":30,
> - "ethtoolCmds" : ["-L -device- combined 1"],
> - "devices":[
> - {
> - "name":"ens785f0"
> - }
> - ]
> - }
> - ]
> - }
> -
> - For further reference please use the `config.json`_
> -
> - .. _config.json: https://github.com/intel/afxdp-plugins-for-
> kubernetes/blob/v0.0.2/test/e2e/config.json
> -
> -* Create the Network Attachment definition
> -
> - .. code-block:: console
> -
> - # kubectl create -f nad.yaml
> -
> - Sample nad.yml
> -
> - .. code-block:: yaml
> -
> - apiVersion: "k8s.cni.cncf.io/v1"
> - kind: NetworkAttachmentDefinition
> - metadata:
> - name: afxdp-e2e-test
> - annotations:
> - k8s.v1.cni.cncf.io/resourceName: afxdp/e2e
> - spec:
> - config: '{
> - "cniVersion": "0.3.0",
> - "type": "afxdp",
> - "mode": "cdq",
> - "logFile": "afxdp-cni-e2e.log",
> - "logLevel": "debug",
> - "ipam": {
> - "type": "host-local",
> - "subnet": "192.168.1.0/24",
> - "rangeStart": "192.168.1.200",
> - "rangeEnd": "192.168.1.216",
> - "routes": [
> - { "dst": "0.0.0.0/0" }
> - ],
> - "gateway": "192.168.1.1"
> - }
> - }'
> -
> - For further reference please use the `nad.yaml`_
> -
> - .. _nad.yaml: https://github.com/intel/afxdp-plugins-for-
> kubernetes/blob/v0.0.2/test/e2e/nad.yaml
> -
> -* Build the Docker image
> -
> - .. code-block:: console
> -
> - # docker build -t afxdp-e2e-test -f Dockerfile .
> -
> - Sample Dockerfile:
> -
> - .. code-block:: console
> -
> - FROM ubuntu:20.04
> - RUN apt-get update -y
> - RUN apt install build-essential libelf-dev -y
> - RUN apt-get install iproute2 acl -y
> - RUN apt install python3-pyelftools ethtool -y
> - RUN apt install libnuma-dev libjansson-dev libpcap-dev net-tools -y
> - RUN apt-get install clang llvm -y
> - COPY ./libbpf<version>.tar.gz /tmp
> - RUN cd /tmp && tar -xvmf libbpf<version>.tar.gz && cd libbpf/src &&
> make install
> - COPY ./libxdp<version>.tar.gz /tmp
> - RUN cd /tmp && tar -xvmf libxdp<version>.tar.gz && cd libxdp && make
> install
> -
> - .. note::
> -
> - All the files that need to COPY-ed should be in the same directory as
> the
> Dockerfile
> -
> -* Run the Pod
> -
> - .. code-block:: console
> -
> - # kubectl create -f pod.yaml
> -
> - Sample pod.yaml:
> -
> - .. code-block:: yaml
> -
> - apiVersion: v1
> - kind: Pod
> - metadata:
> - name: afxdp-e2e-test
> - annotations:
> - k8s.v1.cni.cncf.io/networks: afxdp-e2e-test
> - spec:
> - containers:
> - - name: afxdp
> - image: afxdp-e2e-test:latest
> - imagePullPolicy: Never
> - env:
> - - name: LD_LIBRARY_PATH
> - value: /usr/lib64/:/usr/local/lib/
> - command: ["tail", "-f", "/dev/null"]
> - securityContext:
> - capabilities:
> - add:
> - - CAP_NET_RAW
> - - CAP_BPF
> - resources:
> - requests:
> - hugepages-2Mi: 2Gi
> - memory: 2Gi
> - afxdp/e2e: '1'
> - limits:
> - hugepages-2Mi: 2Gi
> - memory: 2Gi
> - afxdp/e2e: '1'
> -
> - For further reference please use the `pod.yaml`_
> -
> - .. _pod.yaml: https://github.com/intel/afxdp-plugins-for-
> kubernetes/blob/v0.0.2/test/e2e/pod-1c1d.yaml
> -
> -* Run DPDK with a command like the following:
> -
> - .. code-block:: console
> -
> - kubectl exec -i <Pod name> --container <containers name> -- \
> - /<Path>/dpdk-testpmd -l 0,1 --no-pci \
> - --vdev=net_af_xdp0,use_cni=1,iface=<interface name> \
> - -- --no-mlockall --in-memory
> -
> -For further reference please use the `e2e`_ test case in `AF_XDP Plugin for
> Kubernetes`_
> -
> - .. _e2e: https://github.com/intel/afxdp-plugins-for-
> kubernetes/tree/v0.0.2/test/e2e
> diff --git a/doc/guides/howto/af_xdp_dp.rst
> b/doc/guides/howto/af_xdp_dp.rst new file mode 100644 index
> 0000000000..657fc8d52c
> --- /dev/null
> +++ b/doc/guides/howto/af_xdp_dp.rst
> @@ -0,0 +1,299 @@
> +.. SPDX-License-Identifier: BSD-3-Clause
> + Copyright(c) 2023 Intel Corporation.
> +
> +Using the AF_XDP driver in Kubernetes
> +=====================================
> +
> +Introduction
> +------------
> +
> +Two infrastructure components are needed in order to provision a pod
> +that is using the AF_XDP PMD in Kubernetes:
> +
> +1. AF_XDP Device Plugin (DP).
> +2. AF_XDP Container Network Interface (CNI) binary.
> +
> +Both of these components are available through the `AF_XDP Device
> +Plugin for Kubernetes`_ repository.
> +
> +The AF_XDP DP provisions and advertises networking interfaces to
> +Kubernetes, while the CNI configures and plumbs network interfaces for
> the Pod.
> +
> +This document explains how to use the `AF_XDP Device Plugin for
> +Kubernetes`_ with a DPDK application using the :doc:`../nics/af_xdp`.
> +
> +.. _AF_XDP Device Plugin for Kubernetes:
> +https://github.com/intel/afxdp-plugins-for-kubernetes
> +
> +Background
> +----------
> +
> +The standard :doc:`../nics/af_xdp` initialization process involves
> +loading an eBPF program onto the kernel netdev to be used by the PMD.
> +This operation requires root or escalated Linux privileges and thus
> +prevents the PMD from working in an unprivileged container.
> +The AF_XDP Device Plugin handles this situation by managing the eBPF
> +program(s) on behalf of the Pod, outside of the pod context.
> +
> +At a technical level the AF_XDP Device Plugin opens a Unix Domain
> +Socket
Small nitpick: added the abbreviation UDS as it used in the below sections
and listens for a client to make requests over that socket.
> +A DPDK application acting as a client connects and initiates a configuration
> "handshake".
> +After some validation on the Device Plugin side, the client receives a
> +file descriptor which points to the XSKMAP associated with the loaded eBPF
> program.
> +The XSKMAP is an eBPF map of AF_XDP sockets (XSK).
> +The client can then proceed with creating an AF_XDP socket and
> +inserting that socket into the XSKMAP pointed to by the descriptor.
> +
> +The EAL vdev argument ``use_cni`` is used to indicate that the user
> +wishes to run the PMD in unprivileged mode and to receive the XSKMAP
> +file descriptor from the CNI.
> +When this flag is set,
> +the ``XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD`` libbpf flag should be
> used
> +when creating the socket to instruct libbpf not to load the default
> +libbpf program on the netdev.
> +Instead the loading is handled by the AF_XDP Device Plugin.
> +
> +Limitations
> +-----------
> +
> +For DPDK versions <= v23.11 the Unix Domain Socket file path appears in
> +the pod at "/tmp/afxdp.sock". The handshake implementation in the
> +AF_XDP PMD is only compatible with the AF_XDP Device Plugin up to
> +commit id `38317c2`_ and the pod is limited to a single netdev.
> +
> +.. note::
> +
> + DPDK AF_XDP PMD <= v23.11 will not work with the latest version of the
> + AF_XDP Device Plugin.
> +
> +The issue is if a single pod requests different devices from different
> +pools it results in multiple UDS servers serving the pod with the
> +container using only a single mount point for their UDS as
> +``/tmp/afxdp.sock``. This means that at best one device might be able
> +to complete the handshake. This has been fixed in the AF_XDP Device
> +Plugin so that the mount point in the pods for the UDS appear at
> +``/tmp/afxdp_dp/<netdev>/afxdp.sock``. Later versions of DPDK fix this
> hardcoded path in the PMD alongside the `use_cni` parameter.
Small nitpick: `use_cni` should be ``use_cni``
> +
> +.. _38317c2:
> +https://github.com/intel/afxdp-plugins-for-
> kubernetes/commit/38317c256b
> +5c7dfb39e013a0f76010c2ded03669
> +
> +
> +Prerequisites
> +-------------
> +
> +Device Plugin and DPDK container prerequisites:
> +
> +* Create a DPDK container image.
> +
> +* Set up the device plugin and prepare the Pod Spec as described in
> + the instructions for `AF_XDP Device Plugin for Kubernetes`_.
> +
> +* The Docker image should contain the libbpf and libxdp libraries,
> + which are dependencies for AF_XDP,
> + and should include support for the ``ethtool`` command.
> +
> +* The Pod should have enabled the capabilities ``CAP_NET_RAW`` for
> + AF_XDP socket creation, ``IPC_LOCK`` for umem creation and
> + ``CAP_BPF`` (for Kernel < 5.19) along with support for hugepages.
> +
> + .. note::
> +
> + For Kernel versions < 5.19, all BPF sys calls required CAP_BPF, to access
> maps shared
> + between the eBFP program and the userspace program. Kernels >= 5.19,
> only requires CAP_BPF
> + for map creation (BPF_MAP_CREATE) and loading programs
> (BPF_PROG_LOAD).
> +
> +* Increase locked memory limit so containers have enough memory for
> packet buffers.
> + For example:
> +
> + .. code-block:: console
> +
> + cat << EOF | sudo tee
> /etc/systemd/system/containerd.service.d/limits.conf
> + [Service]
> + LimitMEMLOCK=infinity
> + EOF
> +
> +* dpdk-testpmd application should have AF_XDP feature enabled.
> +
> + For further information see the docs for the: :doc:`../../nics/af_xdp`.
> +
> +
> +Example
> +-------
> +
> +Build a DPDK container image (using Docker)
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +1. Create a Dockerfile (should be placed in top level DPDK directory):
> +
> + .. code-block:: console
> +
> + FROM fedora:38
> +
> + # Setup container to build DPDK applications
> + RUN dnf -y upgrade && dnf -y install \
> + libbsd-devel \
> + numactl-libs \
> + libbpf-devel \
> + libbpf \
> + meson \
> + ninja-build \
> + libxdp-devel \
> + libxdp \
> + numactl-devel \
> + python3-pyelftools \
> + python38 \
> + iproute
> + RUN dnf groupinstall -y 'Development Tools'
> +
> + # Create DPDK dir and copy over sources
> + # Create DPDK dir and copy over sources
> + COPY ./ /dpdk
> + WORKDIR /dpdk
> +
> + # Build DPDK
> + RUN meson setup build
> + RUN ninja -C build
> +
> +2. Build a DPDK container image (using Docker)
> +
> + .. code-block:: console
> +
> + # docker build -t dpdk -f Dockerfile
> +
> +Run dpdk-testpmd with the AF_XDP Device Plugin + CNI
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +* Clone the AF_XDP Device plugin and CNI
> +
> + .. code-block:: console
> +
> + # git clone
> + https://github.com/intel/afxdp-plugins-for-kubernetes.git
> +
> + .. note::
> +
> + Ensure you have the AF_XDP Device Plugin + CNI prerequisites installed.
> +
> +* Build the AF_XDP Device plugin and CNI
> +
> + .. code-block:: console
> +
> + # cd afxdp-plugins-for-kubernetes/
> + # make image
> +
> +* Make sure to modify the image used by the `daemonset.yml`_ file in
> +the deployments directory with
> + the following configuration:
> +
> + .. _daemonset.yml :
> + https://github.com/intel/afxdp-plugins-for-kubernetes/blob/main/deploy
> + ments/daemonset.yml
> +
> + .. code-block:: yaml
> +
> + image: afxdp-device-plugin:latest
> +
> + .. note::
> +
> + This will select the AF_XDP DP image that was built locally. Detailed
> configuration
> + options can be found in the AF_XDP Device Plugin `readme`_ .
> +
> + .. _readme:
> + https://github.com/intel/afxdp-plugins-for-kubernetes#readme
> +
> +* Deploy the AF_XDP Device Plugin and CNI
> +
> + .. code-block:: console
> +
> + # kubectl create -f deployments/daemonset.yml
> +
> +* Create the Network Attachment definition
> +
> + .. code-block:: console
> +
> + # kubectl create -f nad.yaml
> +
> + Sample nad.yml
> +
> + .. code-block:: yaml
> +
> + apiVersion: "k8s.cni.cncf.io/v1"
> + kind: NetworkAttachmentDefinition
> + metadata:
> + name: afxdp-network
> + annotations:
> + k8s.v1.cni.cncf.io/resourceName: afxdp/myPool
> + spec:
> + config: '{
> + "cniVersion": "0.3.0",
> + "type": "afxdp",
> + "mode": "primary",
> + "logFile": "afxdp-cni.log",
> + "logLevel": "debug",
> + "ethtoolCmds" : ["-N -device- rx-flow-hash udp4 fn",
> + "-N -device- flow-type udp4 dst-port 2152 action
> 22"
> + ],
> + "ipam": {
> + "type": "host-local",
> + "subnet": "192.168.1.0/24",
> + "rangeStart": "192.168.1.200",
> + "rangeEnd": "192.168.1.220",
> + "routes": [
> + { "dst": "0.0.0.0/0" }
> + ],
> + "gateway": "192.168.1.1"
> + }
> + }'
> +
> + For further reference please use the example provided by the AF_XDP
> + DP `nad.yaml`_
> +
> + .. _nad.yaml:
> + https://github.com/intel/afxdp-plugins-for-kubernetes/blob/main/exampl
> + es/network-attachment-definition.yaml
> +
> +* Run the Pod
> +
> + .. code-block:: console
> +
> + # kubectl create -f pod.yaml
> +
> + Sample pod.yaml:
> +
> + .. code-block:: yaml
> +
> + apiVersion: v1
> + kind: Pod
> + metadata:
> + name: dpdk
> + annotations:
> + k8s.v1.cni.cncf.io/networks: afxdp-network
> + spec:
> + containers:
> + - name: testpmd
> + image: dpdk:latest
> + command: ["tail", "-f", "/dev/null"]
> + securityContext:
> + capabilities:
> + add:
> + - NET_RAW
> + - IPC_LOCK
> + resources:
> + requests:
> + afxdp/myPool: '1'
> + limits:
> + hugepages-1Gi: 2Gi
> + cpu: 2
> + memory: 256Mi
> + afxdp/myPool: '1'
> + volumeMounts:
> + - name: hugepages
> + mountPath: /dev/hugepages
> + volumes:
> + - name: hugepages
> + emptyDir:
> + medium: HugePages
> +
> + For further reference please use the `pod.yaml`_
> +
> + .. _pod.yaml:
> + https://github.com/intel/afxdp-plugins-for-kubernetes/blob/main/exampl
> + es/pod-spec.yaml
> +
> +* Run DPDK with a command like the following:
> +
> + .. code-block:: console
> +
> + kubectl exec -i <Pod name> --container <containers name> -- \
> + /<Path>/dpdk-testpmd -l 0,1 --no-pci \
> + --vdev=net_af_xdp0,use_cni=1,iface=<interface name> \
> + --no-mlockall --in-memory \
> + -- -i --a --nb-cores=2 --rxq=1 --txq=1
> + --forward-mode=macswap;
> diff --git a/doc/guides/howto/index.rst b/doc/guides/howto/index.rst index
> 71a3381c36..a7692e8a97 100644
> --- a/doc/guides/howto/index.rst
> +++ b/doc/guides/howto/index.rst
> @@ -8,7 +8,7 @@ HowTo Guides
> :maxdepth: 2
> :numbered:
>
> - af_xdp_cni
> + af_xdp_dp
> lm_bond_virtio_sriov
> lm_virtio_vhost_user
> flow_bifurcation
> diff --git a/doc/guides/nics/af_xdp.rst b/doc/guides/nics/af_xdp.rst index
> 1932525d4d..4dd9c73742 100644
> --- a/doc/guides/nics/af_xdp.rst
> +++ b/doc/guides/nics/af_xdp.rst
> @@ -155,9 +155,9 @@ use_cni
> ~~~~~~~
>
> The EAL vdev argument ``use_cni`` is used to indicate that the user wishes to
> -enable the `AF_XDP Plugin for Kubernetes`_ within a DPDK application.
> +enable the `AF_XDP Device Plugin for Kubernetes`_ with a DPDK
> application/pod.
>
> -.. _AF_XDP Plugin for Kubernetes: https://github.com/intel/afxdp-plugins-
> for-kubernetes
> +.. _AF_XDP Device Plugin for Kubernetes:
> +https://github.com/intel/afxdp-plugins-for-kubernetes
>
> .. code-block:: console
>
> --
> 2.41.0
