Thanks Rick we were going to go that route but ran across this (ironic being that they are now Nvidia) section of the Bright Computing (now Nvidia Base Manager) admin manual <https://support.brightcomputing.com/manuals/9.2/admin-manual.pdf#page.124>: “*Drivers To Use For NFS over RDMA Must Be From The Parent Distribution* The use of the RDMA protocol to provide NFS, by installing updated cluster manager OFED drivers is currently *not* supported. This is because these drivers are packaged by Bright Computing from the vendor (Mellanox or Qlogic) releases, and the vendor releases themselves do not support NFS over RDMA.
The option can be selected, but NFS will fall back to using the default NFS TCP/IP protocol. When using NFS over RDMA, ibnet, the IP network used for InfiniBand, should be set." So it MAY have worked to do a new build, but now we know we'd likly to use MOFED from Nvidia. On Fri, Aug 9, 2024 at 3:15 AM Mohr, Rick <moh...@ornl.gov> wrote: > > Rob, > > Those errors appear to be IB/OFED related and not ZFS related. This can > happen when you install a version of OFED that is different from the > version that the Lustre packages were compiled against. > > -Rick > > > > On 8/8/24, 9:28 AM, "lustre-discuss on behalf of Rob Kudyba via > lustre-discuss" <lustre-discuss-boun...@lists.lustre.org <mailto: > lustre-discuss-boun...@lists.lustre.org> on behalf of > lustre-discuss@lists.lustre.org <mailto:lustre-discuss@lists.lustre.org>> > wrote: > > > Thanks Peter but we're using ZFS for pools with the NVME's and for ZFS > snapshots. We also use Bright Computing 9.2 now called Nvidia Base Manager. > In their manual for RDMA they mention this:"Drivers To Use For NFS over > RDMA Must Be From The Parent Distribution > The use of the RDMA protocol (section 3.6) to provide NFS, by installing > updated cluster manager OFED > drivers (section 7.6 of the Installation Manual) is currently not > supported. This is because these drivers are > packaged by Bright Computing from the vendor (Mellanox or Qlogic) > releases, and the vendor releases > themselves do not support NFS over RDMA. > The option can be selected, but NFS will fall back to using the default > NFS TCP/IP protocol. > When using NFS over RDMA, ibnet, the IP network used for InfiniBand, > should be set. Section 3.6.3 > explains how that can be done." > > > > > > > Well we used the Bright-provided OFED when using the already-installer > version 2.14, specifically lustre-client-2.14.0_ddn136. Could that be the > cause of the ksym errors? > > > > > On Wed, Aug 7, 2024 at 4:38 PM Peter Jones <pjo...@whamcloud.com <mailto: > pjo...@whamcloud.com> <mailto:pjo...@whamcloud.com <mailto: > pjo...@whamcloud.com>>> wrote: > > > Rob > > > If you’re wanting to run Lustre on ZFS I would recommend that you use the > latest community LTS release – 2.15.5. > > > Peter > > > From: lustre-discuss <lustre-discuss-boun...@lists.lustre.org <mailto: > lustre-discuss-boun...@lists.lustre.org> <_blank>> on behalf of Rob > Kudyba via lustre-discuss <lustre-discuss@lists.lustre.org <mailto: > lustre-discuss@lists.lustre.org> <_blank>> > Reply-To: Rob Kudyba <rk3...@columbia.edu <mailto:rk3...@columbia.edu> > <_blank>> > Date: Wednesday, August 7, 2024 at 12:56 PM > To: "lustre-discuss@lists.lustre.org <mailto: > lustre-discuss@lists.lustre.org> <_blank>" < > lustre-discuss@lists.lustre.org <mailto:lustre-discuss@lists.lustre.org> > <_blank>> > Subject: [lustre-discuss] nothing provides ksym with ZFS and Lustre 2.14 > from DDN > > > > > > > Do we need to rebuild the Lustre packages with options for ZFS? Or is > there a missing package(s)? > > > yum install ./kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64.rpm > ./lustre-client-2.14.0_ddn136-1.el9.x86_64.rpm > ./lustre-client-devel-2.14.0_ddn136-1.el9.x86_64.rpm --nobest > Updating Subscription Management repositories. > CUIT_EL9_RPMS 32 kB/s | 2.1 kB 00:00 > Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) 34 kB/s | 2.4 kB > 00:00 > Red Hat Satellite Client 6 for RHEL 9 x86_64 (RPMs) 29 kB/s | 2.1 kB 00:00 > Red Hat Enterprise Linux 9 for x86_64 - Supplementary (RPMs) 31 kB/s | 2.1 > kB 00:00 > Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) 42 kB/s | 2.8 kB > 00:00 > Red Hat CodeReady Linux Builder for RHEL 9 x86_64 (RPMs) 44 kB/s | 2.8 kB > 00:00 > Error: > Problem 1: conflicting requests > - nothing provides ksym(__ib_alloc_pd) = 0x285eafea needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(__ib_create_cq) = 0x71ea9d65 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(__rdma_create_kernel_id) = 0xf0cf20b7 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_alloc_mr) = 0xd8a14b92 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_dealloc_pd_user) = 0x7d473955 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_dereg_mr_user) = 0x929617d8 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_destroy_cq_user) = 0xeb72649a needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_dma_virt_map_sg) = 0x88834d04 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_get_dma_mr) = 0xea30b011 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_map_mr_sg) = 0x295149cb needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_modify_qp) = 0x5bab09bf needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_query_port) = 0x85125bb5 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_register_event_handler) = 0x61177ade needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_unregister_event_handler) = 0xfa752e5a needed > by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_accept) = 0x5e7390d0 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_bind_addr) = 0x198fbac9 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_connect_locked) = 0x028ebe82 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_create_qp) = 0x9b885678 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_destroy_id) = 0x5ef64a91 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_destroy_qp) = 0xacbc5324 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_disconnect) = 0xdeb65da4 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_listen) = 0x1d60b437 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_notify) = 0xa75102da needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_reject) = 0x10d4f713 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_resolve_addr) = 0x4b1edf4f needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_resolve_route) = 0x85ab76dd needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_set_reuseaddr) = 0xcf82f45b needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > Problem 2: package lustre-client-devel-2.14.0_ddn136-1.el9.x86_64 from > @commandline requires kmod-lustre-client = 2.14.0_ddn136, but none of the > providers can be installed > - conflicting requests > - nothing provides ksym(__ib_alloc_pd) = 0x285eafea needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(__ib_create_cq) = 0x71ea9d65 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(__rdma_create_kernel_id) = 0xf0cf20b7 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_alloc_mr) = 0xd8a14b92 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_dealloc_pd_user) = 0x7d473955 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_dereg_mr_user) = 0x929617d8 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_destroy_cq_user) = 0xeb72649a needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_dma_virt_map_sg) = 0x88834d04 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_get_dma_mr) = 0xea30b011 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_map_mr_sg) = 0x295149cb needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_modify_qp) = 0x5bab09bf needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_query_port) = 0x85125bb5 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_register_event_handler) = 0x61177ade needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_unregister_event_handler) = 0xfa752e5a needed > by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_accept) = 0x5e7390d0 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_bind_addr) = 0x198fbac9 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_connect_locked) = 0x028ebe82 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_create_qp) = 0x9b885678 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_destroy_id) = 0x5ef64a91 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_destroy_qp) = 0xacbc5324 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_disconnect) = 0xdeb65da4 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_listen) = 0x1d60b437 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_notify) = 0xa75102da needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_reject) = 0x10d4f713 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_resolve_addr) = 0x4b1edf4f needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_resolve_route) = 0x85ab76dd needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_set_reuseaddr) = 0xcf82f45b needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > Problem 3: package lustre-client-2.14.0_ddn136-1.el9.x86_64 from > @commandline requires kmod-lustre-client = 2.14.0_ddn136, but none of the > providers can be installed > - conflicting requests > - nothing provides ksym(__ib_alloc_pd) = 0x285eafea needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(__ib_create_cq) = 0x71ea9d65 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(__rdma_create_kernel_id) = 0xf0cf20b7 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_alloc_mr) = 0xd8a14b92 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_dealloc_pd_user) = 0x7d473955 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_dereg_mr_user) = 0x929617d8 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_destroy_cq_user) = 0xeb72649a needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_dma_virt_map_sg) = 0x88834d04 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_get_dma_mr) = 0xea30b011 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_map_mr_sg) = 0x295149cb needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_modify_qp) = 0x5bab09bf needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_query_port) = 0x85125bb5 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_register_event_handler) = 0x61177ade needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(ib_unregister_event_handler) = 0xfa752e5a needed > by kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_accept) = 0x5e7390d0 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_bind_addr) = 0x198fbac9 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_connect_locked) = 0x028ebe82 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_create_qp) = 0x9b885678 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_destroy_id) = 0x5ef64a91 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_destroy_qp) = 0xacbc5324 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_disconnect) = 0xdeb65da4 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_listen) = 0x1d60b437 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_notify) = 0xa75102da needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_reject) = 0x10d4f713 needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_resolve_addr) = 0x4b1edf4f needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_resolve_route) = 0x85ab76dd needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > - nothing provides ksym(rdma_set_reuseaddr) = 0xcf82f45b needed by > kmod-lustre-client-2.14.0_ddn136-1.el9.x86_64 from @commandline > (try to add '--skip-broken' to skip uninstallable packages) > > > > > > > > > > > > > > > > > > > >
_______________________________________________ lustre-discuss mailing list lustre-discuss@lists.lustre.org http://lists.lustre.org/listinfo.cgi/lustre-discuss-lustre.org
