Thanks, Christian, this is great information. Just for fun, before I read this, I wanted to test some of my kernel building skills. So I just wish to build modules in the drivers/vfio directory. I want to do what might be called "building an external module" or "building only one kernel module" as in http://askubuntu.com/questions/515407/how-recipe-to-build-only-one-kernel-module or "building a third party module". But the Google hits like that one assume that you have no interest in changing the .config. And one Ubuntu package that may be useful, linux-source, has info showing up in Synaptic saying "you don't want this package." Here is my procedure and I hope it works -- it is relatively painless. I'll describe it for a particular level. I am using the desktop version of Ubuntu.
1. Install linux-headers-4.8.0-34-generic (I think it may be installed automatically.) 2. Install linux-headers-4.8.0-34 (again, may be automatically installed.) $ cd $HOME $ cp -a /usr/src/linux-headers-4.8.0-34 . $ cp -a /usr/src/linux-headers-4.8.0-34-generic . $ cd linux-headers-4.8.0-34-generic Now "make menuconfig" and go into drivers and then go into vfio and then go inside there and find the interesting option to enable. Now you need the source code for drivers/vfio. I guess the safest way is to install linux-source-4.8.0 and untar its linux-source-4.8.0-tar.bz into your home directory so you end up with linux-source-4.8.0 under $HOME. When I did my experimenting, I worked mostly from the torvalds Linux repo, at tag v4.8. $ cd $HOME/linux-source-4.8.0 #assuming you untarred the source from Ubuntu here. $ make -C $HOME/linux-headers-4.8.0-34-generic M=$(pwd)/drivers/vfio modules #in addition to modules, you can use "clean" as appropriate This is tricky stuff because there is plenty of non-information and misinformation out there AND M=<ABSOLUTE-DIRECTORY_PATH> #better not try relative path O=<DO NOT ATTEMPT TO USE THIS OPTION WITH THE M OPTION> #kernel Makefile does not warn you clearly but it bombs Normally I like the O option to put the built stuff away from the source. Well, this stuff may be slightly off topic, but if I ever have to retrieve my instructions, I'll know where to search, chuckle. And I hope they work like they did for me. I was able to insmod a module that I built with source from torvalds repo. I built form Ubuntu source and I assume it will also insmod but haven't checked. Burt On Thu, Jan 26, 2017 at 9:00 AM, Christian Ehrhardt < christian.ehrha...@canonical.com> wrote: > On Wed, Jan 25, 2017 at 5:24 PM, Ed Warnicke <hagb...@gmail.com> wrote: > >> Looping in the deb_dpdk and rpm_dpdk guys to see if they have thoughts on >> how these things play out in deployment. >> > > Thanks Ed. > Adding my 2 cents inline below just to provide another POV ... > > > >> On Tue, Jan 24, 2017 at 9:05 PM, Burt Silverman <bur...@gmail.com> wrote: >> >>> Hi Damjan, >>> >>> My understanding is that CONFIG_VFIO_NOIOMMU will never be set in a >>> stock kernel, and you will need to build a custom kernel for that. >>> >> > Well, to Enable the config itself isn't bad until enable_unsafe_noiommu_mo > deis set (which is default off). > See https://patchwork.kernel.org/patch/7906971/ for a few more Details. > > IMHO that means it can be safely in a default kernel with support, and it > is on the administrator to enable that mode. > But that is like giving wrong people root, or missetting something else - > not as bad as building an own kernel. > > In fact there was a guy with my name who got that enabled in the next > Ubuntu Release Zesty :-) > https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1634050 > > For vfio (as the default) there is another thing much more in the way. > People will likely want iommu support if possible, but in 9/10 cases I've > seen I'd say that some oddity forced them to be unable to use it. > Certain Firmware rendered iommu useless, Unable to set up the iommu group > assignment correctly, ... > I happened to get the feeling that it might only work in just one special > Lab setup somebody has out there. > > The noiommu config might make that simpler, but then it throws away a lot > that vfio should provide. > So I'd only use it for test/devel but would not like it too much for > production use. > > That said IMHO I would not suggest vfio as a default these days even with > the kernel option enabled. > > > >> I understand that with this option, the kernel cannot guarantee that >>> applications are prevented from creating bugs that normally the kernel can >>> guarantee will not occur (outside of a kernel bug.) It therefore violates >>> the fundamental Linux system design. That being said, you may wish to >>> accept the risk for performance reasons and build a custom kernel. The >>> other strange thing would be that MSI or MSI-X style interrupts are not >>> needed for performance. The people who developed them have made a lot of >>> noise about how they came about for performance reasons. I have no direct >>> experience, but to learn that they are not important is a shock. >>> >> > > I don't have good insight on the related performance impact for the > question "igb_uio -> uio_pci_generic". > But then (whatever it is worth) I usually go with uio_pci_generic and most > people I talked to did so as well (on the wider DPDK and not just VPP > stage). > > But here the circle closes referring to some former statements that > "compiling the kernel on your own is a no-go" > If I'm not overlooking something - at least in all environments I touched > - igb_uio is a dpdk provided kernel module and uio_pci_generic is in the > kernel. > At least http://dpdk.org/doc/guides/linux_gsg/build_dpdk. > html#loading-modules-to-enable-userspace-io-for-dpdk backs my memory up. > > That said - loading a custom module is just as bad as building your own > kernel in terms of applicability in a real case. > > So to add my unimportant opinion to this thread about what could/should be > the default: > 1. uio_pci_generic (default) > 2. vfio (as a better option for those who want/can set it up correctly) > 3. igb_uio (fallback for special cases where it is needed) > > As I said I can't participate in the performance fight between > uip_pci_generic and igb_uio > > > >>> It seems to me that the Ubuntu 14.04 issue is really a separate one from >>> all of this, although I would imagine that the conclusion to stop >>> supporting it does not change. >>> >>> Burt >>> >>> On Tue, Jan 24, 2017 at 1:07 PM, Damjan Marion (damarion) < >>> damar...@cisco.com> wrote: >>> >>>> >>>> > On 24 Jan 2017, at 18:40, Damjan Marion (damarion) < >>>> damar...@cisco.com> wrote: >>>> > >>>> >> >>>> >> On 24 Jan 2017, at 18:26, Stephen Hemminger < >>>> step...@networkplumber.org> wrote: >>>> >> >>>> >> On Tue, 24 Jan 2017 17:14:42 +0000 >>>> >> "Damjan Marion (damarion)" <damar...@cisco.com> wrote: >>>> >> >>>> >>> Is anybody aware of any valid reason why we cannot switch to >>>> uio_pci_generic >>>> >>> as default PCI uio driver in ubuntu packages? >>>> >>> >>>> >>> I think generally people don’t like out-of-tree modules, so as long >>>> as we are getting >>>> >>> the same service from uio_pci_generic we should use it… >>>> >>> >>>> >>> Thanks, >>>> >>> >>>> >>> Damjan >>>> > >>>> >> uio_pci_generic does not support MSI or MSI-X interrupts, only >>>> legacy INTX. >>>> > >>>> > I know but do we really care? >>>> > >>>> >> >>>> >> The preference should always be to use VFIO. Even on systems without >>>> IOMMU. >>>> > >>>> > What is the perf impact? >>>> > >>>> > Also, I just tried with kernel 4.8 on rangeley ATOM, and i got: >>>> > >>>> > [536030.250072] vfio-pci: probe of 0000:00:14.0 failed with error -22 >>>> > [536030.253271] vfio-pci: probe of 0000:00:14.0 failed with error -22 >>>> > >>>> > I guess I’m doing something wrong…. >>>> >>>> This explains: >>>> >>>> grep VFIO_NOIO /boot/config-4.8.0-34-generic >>>> # CONFIG_VFIO_NOIOMMU is not set >>>> >>>> So vfio is out of the game of being default choice, people can still >>>> switch simply with one line change in /etc/vpp/startup.conf. >>>> >>> > > > > -- > Christian Ehrhardt > Software Engineer, Ubuntu Server > Canonical Ltd >
_______________________________________________ vpp-dev mailing list vpp-dev@lists.fd.io https://lists.fd.io/mailman/listinfo/vpp-dev
