Re: [dpdk-dev] [RFC] igb_uio: change ISR to be threaded when using with RT kernel
Hi DPDK experts,
This patch is especially used for PREEMPT_RT kernel.
Whether it is correct, please share your opinion.
Thanks,
Yongxin
> -Original Message-
> From: dev [mailto:dev-boun...@dpdk.org] On Behalf Of Yongxin Liu
> Sent: Friday, September 20, 2019 08:45
> To: dev@dpdk.org
> Cc: david.w...@intel.com; step...@networkplumber.org;
> ferruh.yi...@intel.com; anuj.mit...@intel.com; Hao, Kexin; Yang, Liezhi
> Subject: [dpdk-dev] [RFC] igb_uio: change ISR to be threaded when using
> with RT kernel
>
> When igb_uio driver is used with RT kernel, the interrupt handler
> should not use IRQF_NO_THREAD as a flag, since uio_event_notify will
> call rt_spin_lock which might sleep.
>
> Here is the calltrace.
>
> [ 2034.457805] Call Trace:
> [ 2034.460818]
> [ 2034.463397] dump_stack+0x70/0x9a
> [ 2034.467281] ___might_sleep.cold+0xe1/0xf2
> [ 2034.471943] rt_spin_lock+0x55/0x70
> [ 2034.475994] ? __wake_up_common_lock+0x61/0xb0
> [ 2034.481004] __wake_up_common_lock+0x61/0xb0
> [ 2034.485842] __wake_up+0x13/0x20
> [ 2034.489635] uio_event_notify+0x2c/0x50 [uio]
> [ 2034.494560] igbuio_pci_irqhandler+0x1f/0x40 [igb_uio]
> [ 2034.500262] __handle_irq_event_percpu+0x5f/0x3f0
> [ 2034.505532] ? cpuidle_enter_state+0xd3/0x500
> [ 2034.510454] handle_irq_event_percpu+0x4b/0x90
> [ 2034.515462] handle_irq_event+0x3c/0x5b
> [ 2034.519864] handle_edge_irq+0xbb/0x210
> [ 2034.524264] handle_irq+0x23/0x30
> [ 2034.528143] do_IRQ+0x7e/0x150
> [ 2034.531764] common_interrupt+0xf/0xf
> [ 2034.535990]
>
> Signed-off-by: Yongxin Liu
> ---
> kernel/linux/igb_uio/igb_uio.c | 6 +-
> 1 file changed, 1 insertion(+), 5 deletions(-)
>
> diff --git a/kernel/linux/igb_uio/igb_uio.c
> b/kernel/linux/igb_uio/igb_uio.c
> index 039f5a5f6..9d7380fb7 100644
> --- a/kernel/linux/igb_uio/igb_uio.c
> +++ b/kernel/linux/igb_uio/igb_uio.c
> @@ -221,7 +221,6 @@ igbuio_pci_enable_interrupts(struct rte_uio_pci_dev
> *udev)
> msix_entry.entry = 0;
> if (pci_enable_msix(udev->pdev, &msix_entry, 1) == 0) {
> dev_dbg(&udev->pdev->dev, "using MSI-X");
> - udev->info.irq_flags = IRQF_NO_THREAD;
> udev->info.irq = msix_entry.vector;
> udev->mode = RTE_INTR_MODE_MSIX;
> break;
> @@ -229,7 +228,6 @@ igbuio_pci_enable_interrupts(struct rte_uio_pci_dev
> *udev)
> #else
> if (pci_alloc_irq_vectors(udev->pdev, 1, 1, PCI_IRQ_MSIX) ==
> 1) {
> dev_dbg(&udev->pdev->dev, "using MSI-X");
> - udev->info.irq_flags = IRQF_NO_THREAD;
> udev->info.irq = pci_irq_vector(udev->pdev, 0);
> udev->mode = RTE_INTR_MODE_MSIX;
> break;
> @@ -241,7 +239,6 @@ igbuio_pci_enable_interrupts(struct rte_uio_pci_dev
> *udev)
> #ifndef HAVE_ALLOC_IRQ_VECTORS
> if (pci_enable_msi(udev->pdev) == 0) {
> dev_dbg(&udev->pdev->dev, "using MSI");
> - udev->info.irq_flags = IRQF_NO_THREAD;
> udev->info.irq = udev->pdev->irq;
> udev->mode = RTE_INTR_MODE_MSI;
> break;
> @@ -249,7 +246,6 @@ igbuio_pci_enable_interrupts(struct rte_uio_pci_dev
> *udev)
> #else
> if (pci_alloc_irq_vectors(udev->pdev, 1, 1, PCI_IRQ_MSI) == 1)
> {
> dev_dbg(&udev->pdev->dev, "using MSI");
> - udev->info.irq_flags = IRQF_NO_THREAD;
> udev->info.irq = pci_irq_vector(udev->pdev, 0);
> udev->mode = RTE_INTR_MODE_MSI;
> break;
> @@ -259,7 +255,7 @@ igbuio_pci_enable_interrupts(struct rte_uio_pci_dev
> *udev)
> case RTE_INTR_MODE_LEGACY:
> if (pci_intx_mask_supported(udev->pdev)) {
> dev_dbg(&udev->pdev->dev, "using INTX");
> - udev->info.irq_flags = IRQF_SHARED | IRQF_NO_THREAD;
> + udev->info.irq_flags = IRQF_SHARED;
> udev->info.irq = udev->pdev->irq;
> udev->mode = RTE_INTR_MODE_LEGACY;
> break;
> --
> 2.14.4
Re: [dpdk-dev] [PATCH v2] usertools/devbind: fix binding for built-in 1kernel drivers
> -Original Message-
> From: Burakov, Anatoly
> Sent: Thursday, November 19, 2020 00:28
> To: Liu, Yongxin ; dev@dpdk.org;
> tho...@monjalon.net
> Subject: Re: [dpdk-dev] [PATCH v2] usertools/devbind: fix binding for
> built-in 1kernel drivers
>
>
> On 18-Nov-20 2:58 AM, Yongxin Liu wrote:
> > In commit 681a67288655 ("usertools: check if module is loaded before
> > binding"), script will exit if no driver is found in /sys/module/.
> >
> > However, for built-in kernel driver, /sys/module/MODULENAME only shows
> > up if it has a version or at least one parameter. Take ixgbe for
> > example, after kernel commit 34a2a3b83e2c ("net/intel: remove driver
> > versions from Intel drivers"), and if ixgbe is built directly into
> > kernel, there is no ixgbe folder in /sys/module. So the devbind script
> > should not exit.
> >
> > Signed-off-by: Yongxin Liu
> > ---
> >
> > v2:
> > - fix git commit description style in commit log
> > - fix typo spelling
> >
> > ---
> > usertools/dpdk-devbind.py | 4
> > 1 file changed, 4 deletions(-)
> >
> > diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py
> > index 99112b7ab..f3c0d9814 100755
> > --- a/usertools/dpdk-devbind.py
> > +++ b/usertools/dpdk-devbind.py
> > @@ -530,10 +530,6 @@ def bind_all(dev_list, driver, force=False):
> > # driver generated error - it's not a valid device ID, so all
> is well
> > pass
> >
> > -# check if we're attempting to bind to a driver that isn't loaded
> > -if not module_is_loaded(driver.replace('-','_')):
> > -sys.exit("Error: Driver '%s' is not loaded." % driver)
> > -
>
> I believe there is a way to check if module is built-in, can't we use that?
> We could keep a list of built-in modules of interest that we can get from:
>
> cat /lib/modules/$(uname -r)/modules.builtin
>
> It's a bit more changes, but this is better than just removing the error
> check.
Thanks Anatoly for your advice.
I have sent v3. Please review.
/Yongxin
>
> > try:
> > dev_list = map(dev_id_from_dev_name, dev_list)
> > except ValueError as ex:
> >
>
>
> --
> Thanks,
> Anatoly
Re: [dpdk-dev] [PATCH v3] usertools/devbind: fix binding for built-in kernel drivers
> -Original Message-
> From: Burakov, Anatoly
> Sent: Thursday, November 19, 2020 20:19
> To: Liu, Yongxin ; dev@dpdk.org;
> tho...@monjalon.net
> Subject: Re: [dpdk-dev] [PATCH v3] usertools/devbind: fix binding for
> built-in kernel drivers
>
>
> On 19-Nov-20 7:16 AM, Yongxin Liu wrote:
> > A driver can be loaded as a dynamic module or a built-in module.
> > In commit 681a67288655 ("usertools: check if module is loaded before
> > binding"), script only checks modules in /sys/module/.
> >
> > However, for built-in kernel driver, it only shows up in /sys/module/,
> > if it has a version or at least one parameter. So add check for
> > modules in /lib/modules/$(uname -r)/modules.builtin.
> >
> > Thanks for Anatoly Burakov's advice.
> >
> > Signed-off-by: Yongxin Liu
> > ---
> >
> > v3:
> > - Add built-in module list in loaded_modules for checking
> > instead of removing error check.
> >
> > v2:
> > - fix git commit description style in commit log
> > - fix typo spelling
> >
> > ---
> > usertools/dpdk-devbind.py | 8 +++-
> > 1 file changed, 7 insertions(+), 1 deletion(-)
> >
> > diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py
> > index 99112b7ab..5b34ccd2a 100755
> > --- a/usertools/dpdk-devbind.py
> > +++ b/usertools/dpdk-devbind.py
> > @@ -181,7 +181,13 @@ def module_is_loaded(module):
> >
> > loaded_modules = sysfs_mods
> >
> > -return module in sysfs_mods
> > +# add built-in modules as loaded
> > +builtin_mods = subprocess.check_output(["cat /lib/modules/$(uname
> > + -r)/modules.builtin"], shell=True).splitlines()
>
> I'd rather not call shell directly, it's bad practice and is potentially
> unsafe. The $(uname -r) can be replaced with:
>
> import platform
> release = platform.uname().release # equivalent to $(uname -r)
>
> The rest can follow:
>
> filename = os.path.join("/lib/modules/", release,
> "modules.builtin")
> # read the file contents, split lines, etc.
>
> Also, please check if file exists to ensure we don't crash the script.
>
> > +for mod in builtin_mods:
> > +mod_name = os.path.basename(mod.decode("utf8")).split(".ko",
> > + 1)
>
> os.path.splitext(os.path.basename(...))?
Thanks Anatoly for your detailed instruction.
v4 has been sent.
/Yongxin
>
> > +loaded_modules.append(mod_name[0])
> > +
> > +return module in loaded_modules
> >
> >
> > def check_modules():
> >
>
>
> --
> Thanks,
> Anatoly
Re: [dpdk-dev] [PATCH v5] usertools/devbind: fix binding for built-in kernel drivers
Hi Anatoly,
Do you have any further comments on this v5?
Or you can submit your own patch directly.
I am really expecting this issue to be fixed.
Thank you very much.
Yongxin
> -Original Message-
> From: dev On Behalf Of Yongxin Liu
> Sent: Monday, November 23, 2020 11:06
> To: dev@dpdk.org; anatoly.bura...@intel.com; tho...@monjalon.net
> Subject: [dpdk-dev] [PATCH v5] usertools/devbind: fix binding for built-in
> kernel drivers
>
> A driver can be loaded as a dynamic module or a built-in module.
> In commit 681a67288655 ("usertools: check if module is loaded before
> binding"), script only checks modules in /sys/module/.
>
> However, for built-in kernel driver, it only shows up in /sys/module/, if
> it has a version or at least one parameter. So add check for modules in
> /lib/modules/$(uname -r)/modules.builtin.
>
> Signed-off-by: Yongxin Liu
> ---
>
> v5:
> - Make code robust and more memory efficient.
>
> v4:
> - Replace shell call with platform.uname(). Check file existence
>before reading.
>
> v3:
> - Add built-in module list in loaded_modules for checking
>instead of removing error check.
>
> v2:
> - fix git commit description style in commit log
> - fix typo spelling
>
> ---
> usertools/dpdk-devbind.py | 13 -
> 1 file changed, 12 insertions(+), 1 deletion(-)
>
> diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py index
> 054ad2e1c..4bc0b6207 100755
> --- a/usertools/dpdk-devbind.py
> +++ b/usertools/dpdk-devbind.py
> @@ -7,6 +7,7 @@
> import os
> import subprocess
> import argparse
> +import platform
>
> from glob import glob
> from os.path import exists, basename
> @@ -107,7 +108,17 @@ def module_is_loaded(module):
>
> loaded_modules = sysfs_mods
>
> -return module in sysfs_mods
> +# add built-in modules as loaded
> +release = platform.uname().release
> +filename = os.path.join("/lib/modules/", release, "modules.builtin")
> +if os.path.exists(filename):
> +try:
> +with open(filename) as f:
> +loaded_modules +=
> [os.path.splitext(os.path.basename(mod))[0] for mod in f]
> +except IOError:
> +print("Warning: cannot read list of built-in kernel
> + modules")
> +
> +return module in loaded_modules
>
>
> def check_modules():
> --
> 2.14.4
