Hey Stephen, it's been a while...
> On May 16, 2023, at 2:43 PM, Stephen Hemminger <step...@networkplumber.org> > wrote: > > On Tue, 16 May 2023 13:18:40 -0600 > Philip Prindeville <philipp_s...@redfish-solutions.com> wrote: > >> Hi, >> >> I'm a packaging maintainer for some of the OpenWrt ancillary packages, and >> I'm looking at upstreaming DPDK support into OpenWrt. Apologies in advance >> if this is a bit dense (a brain dump) or hops between too many topics. >> >> I was looking at what's been done to date, and it seems there are a few >> shortcomings which I hope to address. >> >> Amongst them are: >> >> * getting DPDK support into OpenWrt's main repo for the kmod's and into the >> packages repo for the user-space support; > > DPDK kernel modules are deprecated, creating more usage of them is > problematic. Well, some kernel modules are still required, aren't they? What's been deprecated? > >> >> * making DPDK supported on all available architectures (i.e. agnostic, not >> just x86_64 specific); >> >> * integrating into the OpenWrt "make menuconfig" system, so that editing >> packages directly isn't required; > > Does openwrt build system support meson? > >> * supporting cross-building and avoiding the [flawed] assumption that the >> micro-architecture of the build server is in any way related to the >> processor type of the target machine(s); >> >> * integration into the OpenWrt CI/CD tests; >> >> * making the kernel support as secure/robust as possible (i.e. avoiding an >> ill-behaved application taking down the kernel, since this is a firewall >> after all); > > Not a problem with vfio > >> >> * avoiding conflict with other existing module or package functionality; >> >> * avoiding, to the extent possible, introducing one-off toolchain >> dependencies that unnecessarily complicate the build ecosystem; >> >> To this end, I'm asking the mailing list for guidance on the best packaging >> practices that satisfy these goals. I'm willing to do whatever heavy >> lifting that's within my ability. >> >> I have some related questions. >> >> 1. OpenWrt already bakes "vfio.ko" and "vfio-pci.ko" here: >> >> https://github.com/openwrt/openwrt/blob/master/package/kernel/linux/modules/virt.mk#L77-L117 >> >> but does so with "CONFIG_VFIO_PCI_IGD=y", which seems to be specifically >> for VGA acceleration of guests in a virtualized environment. That seems to >> be an odd corner case, and unlikely given that OpenWrt almost always runs on >> headless hardware. Should this be reverted? > > Yes. Okay, thanks. > >> >> 2. "vfio.ko" is built with "CONFIG_VFIO_NOIOMMU=y", which seems insecure. >> Can this be reverted? > > No. Most of OpenWrt's systems do not have an IOMMU. And most of OpenWrt isn't going to need DPDK, either. We're thinking of Xeon, Xeon-D, and Atom64 (C26xx) based Intel hardware, or high-end multicore ARM64 designs like the Traverse Technologies ten64. In other words, Enterprise-class firewalls and data center appliances. > >> 3. Is "uio_pci_generic.ko" worth the potential insecurity/instability of a >> misbehaving application? My understanding is that it's only needed on >> SR-IOV hardware where MSI/MSI-X interrupts aren't supported: is there even >> any current hardware that doesn't support MSI/MSI-X? Or am I >> misunderstanding the use case? > > Use VFIO noiommu, it is more supported and tested upstream. With PCI > generic, no interrupts work. What is the risk/reward of using CONFIG_NOIOMMU=n? It's a non-trivial bit of logic to include in a processor design: it must have had some scenario where it would be useful otherwise that's a lot of wasted gates... > >> 4. Can most functionality be achieved with VFIO + IOMMU support? > > Yes. > >> 5. I've seen packaging for the "iommu_v2.ko" module done here: >> >> https://github.com/k13132/openwrt-dpdk/blob/main/packages/kmod-iommu_v2/Makefile#L22-L42 >> >> Is this potentially useful? What platforms/architectures use this driver? >> >> 6. Hand editing a wrapper for dpdk.tar.gz is a non-starter. I'd rather add >> Kconfig adornments to OpenWrt packaging for the wrapper so that options for >> "-Denable_drivers=" and "-Dcpu_instruction_set=" can be passed in once >> global build options for OpenWrt have been selected. Defaulting the >> instruction set to the build host is going to be wrong at least some of the >> time, if not most of the time. For x86_64, what is a decent compromise for >> a micro-architecture that will build and run on most AMD and Intel hardware? >> What's a decent baseline for an ARM64 micro-architecture that will build >> and run on most ARM hardware? >> >> 7. Many embedded systems don't build with glibc because it's too bloated >> (and because critical fixes sometimes take too long to roll out), and >> instead use MUSL, eglibc, or uClibc (although the last one seems to be >> waning). Only glibc supports <execinfo.h> from what I can tell. Can >> broader support for other C runtimes be added? Can RTE_BACKTRACE be made a >> parameter or conditionally defined based on the runtime headers? (autotools >> would be and HAVE_EXECINFO_H would be really handy here, but I'm not sure >> how to make this play well with meson/ninja, and truth be told I'm an old >> school Makefile + autotools knuckle-dragger). > > You could do without backtrace, but then if DPDK applications you are flying > blind. Yeah, that occurred to me too, but it seems that libbacktrace couldn't be shimmed in because it does require heap integrity if I remember... > >> 8. How do I validate that DPDK is properly being built with the cross-tools >> and not native tools? Even when building x86_64 targets on an x86_64 build >> host, we end up using a custom toolchain and not the "native" compiler that >> comes with the distro. >> >> 9. What is the parity between AMD64 and ARM64? Do both platforms offer >> equal functionality and security, if not performance? > > Apples/Oranges. > >> 10. Who else is using DPDK with OpenWrt that is open to collaboration? >> >> 11. What is the user-space TCP/IP stack of choice (or reference) for use >> with DPDK? > > No user-space TCP/IP stack is really robust and that great. > VPP has one but it is likely to be specific to using VPP and not sure if you > want to go there. > I don't think Fedora/Suse/Debian/Ubuntu have packaged any userspace TCP yet. Not robust how? In terms of performance, security, handling error conditions, having complete feature handlings... ? -Philip
