Hi Bruce From: Bruce Richardson, Monday, April 16, 2018 11:32 AM > On Sat, Apr 14, 2018 at 08:10:28PM +0000, Matan Azrad wrote: > > Hi all > > > > From: Burakov, Anatoly, Friday, April 13, 2018 8:41 PM > > > To: Bruce Richardson <bruce.richard...@intel.com>; Thomas Monjalon > > > <tho...@monjalon.net> > > > Cc: dev@dpdk.org; pmati...@redhat.com; david.march...@6wind.com; > > > jia....@intel.com; Matan Azrad <ma...@mellanox.com>; > > > konstantin.anan...@intel.com; step...@networkplumber.org; > > > f...@redhat.com > > > Subject: Re: kernel binding of devices + hotplug > > > > > > On 13-Apr-18 5:40 PM, Bruce Richardson wrote: > > > > On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wrote: > > > >> It's time to think (again) how we bind devices with kernel modules. > > > >> We need to decide how we want to manage hotplugged devices with > > > DPDK. > > > >> > > > >> A bit of history first. > > > >> There was some code in DPDK for bind/unbind, but it has been > > > >> removed in DPDK 1.7 - > > > >> > > > > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdp > > > d > > > >> > > > > k.org%2Fcommit%2F5d8751b83&data=02%7C01%7Cmatan%40mellanox.com > > > %7C6ea5 > > > >> > > > > 5ce994ff4bb0d65208d5a165b417%7Ca652971c7d2e4d9ba6a4d149256f461b%7 > > > C0%7 > > > >> > > > > C0%7C636592380565078675&sdata=uLRDAk65hYtJYxjIvY20de377yayCN5DrjCZ > > > x8H > > > >> p61o%3D&reserved=0 Copy of the commit message (in 2014): > > > >> " > > > >> The bind/unbind operations should not be handled by the eal. > > > >> These operations should be either done outside of dpdk or > > > >> inside the PMDs themselves as these are their problems. > > > >> " > > > >> > > > >> The question raised at this time (4 years ago) is still under > > > >> discussion. > > > >> Should we manage binding inside or outside DPDK? > > > >> Should it be controlled in the application or in the OS base? > > > >> > > > >> As you know, we use dpdk-devbind.py. > > > >> This tool lacks two major features: > > > >> - persistent configuration > > > >> - hotplug > > > >> > > > >> If we consider that the DPDK applications should be able to apply > > > >> its own policy to choose the devices to bind, then we need to > > > >> implement binding in the PMD (with EAL helpers). > > > >> > > > >> On the other hand, if we consider that it is the system > > > >> responsibility, then we could choose systemd/udev and driverctl. > > > >> > > > >> The debate is launched! > > > >> > > > > > > > > Allow me to nail my colours to the mast early! :-) > > > > > > > > I believe it's system not application responsibility. > > > > I also believe I have previously explained my reasons for that > > > > choice in some of the previous email threads. > > > > > > For what it's worth, I tend to agree, if only because writing code > > > for what is essentially a bunch of read/write/filesystem enumeration > > > in C is extremely fiddly and error prone :) IMO things like this are > > > better handled either by scripts, or by tools whose sole purpose is doing > exactly that (or both). > > > > > > I like having scripts like devbind in DPDK because we can tailor > > > them to our use cases better, and having them is amenable to > > > automation, but while I wouldn't be opposed to removing them > > > altogether in favor of some external tool > > > (systemd/udev/driverctl/whatever), in my humble opinion moving them > back into EAL or even PMD's would be a mistake. > > > > > > > Since the application runs in the system by a command of the system user I > think the responsibility is for the user. > > The DPDK user forwards the control of some devices to the DPDK > > application using the EAL whitelist\blacklist mode to specify the > > devices, Any DPDK PMD should know which binding it needs to >> probe\control the device and can apply it, So, if the user asks to control >> on a >> device by DPDK application it makes sense that the application will do the >> correct binding to the device since the user wants to use it(no need to ask >> more operation of pre binding from the user). > > Completely agree that it is ultimately up to the user. However, what I don't > want to see is the case where the user always has to specify a big long list > of > device whitelist and blacklist options to each run of an application. > Instead, if > device management is done at the system level via udev (for example) > configured via devicectl, then the application commandline can be vastly > simplified.
Actually you say that the whitelist\blacklist mechanism is not good enough and the binding workarounds it. The user need to specify somehow the devices it want to run, I think that specifying the device you want by -w option (no need to specify what you don't want in -w case) is really simpler and more descriptive than binding each device you want by prior process to its correct driver. > It also allows better usability across systems, since the same > commandline can be used on multiple systems with different hardware, with > the actual device management rules having been already configured at > system install/setup time in udev. But the user still needs to configure the udev per device for each system, I think that command line is better. > > Regarding the conflict of system rules for a device, it is again the user > responsibility, whatever we will decide for the binding procedure of DPDK > application the user needs to take it into account and to solve such like > conflicts. > > One option is to remove any binding rules of a DPDK device in the DPDK > application initialization and adjust the new rules by the PMDs, then any > conflict should not disturb the user. > > If the device management is only managed in one place, i.e. not in DPDK, > then there is no conflict to manage. I can't agree with this statement, The essence of DPDK is to give a good alternative to managing network devices, DPDK actually takes a lot of management area to manage by itself to do the user life better :) Moreover, Instead of bind script usage and dpdk running , just run dpdk (do all the job in one place). > > In current hot-plug case the application will need to do a lot of work to > bind\remap devices in plug-in\plug-out events while the PMD could have all > the knowledge to do it. > > At the cost of duplicating a lot of code between PMDs. Why a lot of code? 1 helper in EAL to be used for each relevant PMD. Each PMD just call to the EAL helper with the bind driver type. Really simple and immediate. I think it is better than duplication of user operations. > > One more issue with the script is that the user should do different bind per > device, in case of PMD responsibility the user can forget it: > > Think about that, any time the user wants to switch\add new supported nic > it should update the script usage and to do per nic operation contrary to the > DPDK principles. > > > > The udev rules syntax should provide adequate capabilities here for us to > match the correct binding behaviour. No need to have it in DPDK too. We can use it from DPDK. > /Bruce
