It is worth trying a Mellanox 10 GbE NIC. In our recent internal testing with UHD + DPDK + N320/N321, we saw similar issues as Berkay notes, and eventually tracked the issue to the Intel driver. We approached Intel and while they acknowledged the probable issue, since they have moved development work to the 800-series NICs, they declined to fix the issue in their 500/700 series NIC driver. We switched to a Mellanox NIC and that resolved the issues. Please note that UHD currently supports DPDK 18.11 (exact version), which does not have support for Intel 800-series NICs -- the oldest DPDK for that support is 19.04, and 19.11 is recommended as the minimum version. We are actively working on updating DPDK support to 19.11 / 20.11 / 21.11 -- looks like the same API, which is different from that for 18.11. - MLD
On Fri, Nov 19, 2021 at 1:54 AM Berkay SAYGILI <zuhasd...@gmail.com> wrote: > Hello Guillermo, > > I have the same problem with DPDK. I can receive data with 200 MSPS in > both channels with a single N320 without DPDK, but when I use DPDK, it > starts to drop samples even at much lower sample rates. I have an X520-DA2 > NIC. When I wrote this problem to Ettus, they suggested me to try the DPDK > with Mellanox NIC. Therefore, I am considering buying a Mellanox NIC at the > moment, but I am not certain that a new NIC will solve the problem. > > > Best regards > > Berkay S. > > On Thu, 18 Nov 2021 at 18:01, Rob Kossler <rkoss...@nd.edu> wrote: > >> Hi Guillermo, >> My experience with DPDK was the following: >> >> - very difficult to configure >> - once configured properly, streaming was exceptional (significant >> improvement compared to no DPDK). I was using Intel cards (likely >> X520-DA2 >> and XL710-QDA1). >> - lots of bad side effects related to DPDK taking over a dedicated >> CPU. I'm not sure why but at times even the mouse and keyboard would get >> so >> slow that the system became unusable - it occurred to me that maybe the >> mouse/keyboard were mistakenly being serviced by the CPU that was >> dedicated >> to DPDK - very strange. And, it wasn't just the mouse - other applications >> could be affected. The side effects were severe enough for me to abandon >> using DPDK on a regular basis. That said, I would consider it in your >> situation where you needed the performance. >> >> I never experienced the troubles you are having where DPDK is working but >> performing poorly. In my case, if it worked at all, the performance was >> good. Sorry that I don't have any good suggestions for you. >> >> Whenever I am trying to capture high data rate receive streams, I save >> the data to files in a RAM file system. If I need these files permanently >> stored, I have had best success with running a separate utility that >> moves the saved files from RAM to storage. It doesn't seem like this >> approach should be necessary (as compared to multi threads within one >> process and/or keeping the data in shared memory rather than buffering in >> 'files') but perhaps my attempts at other approaches were faulty. >> >> Rob >> >> On Thu, Nov 18, 2021 at 5:03 AM Guillermo Ortas Delgado <g.or...@gmv.com> >> wrote: >> >>> Hi Marcus, >>> >>> >>> >>> Wow, good to know that I’m doing good. I thought I wasn’t the only one >>> working at these high rates. >>> >>> I guess there is the possibility to spread the load on another computer, >>> but I was interested in DPDK as it showed potential to accomplish my goal >>> more elegantly instead of just throwing more money at the problem. >>> >>> >>> >>> I’m still puzzled by DPDK dropping samples at even 25Msps on 2 channels. >>> >>> *@Rob Kossler: do you have any more input on this matter?* >>> >>> >>> >>> Best, >>> >>> Guillermo >>> >>> >>> >>> *De:* Marcus D. Leech [mailto:patchvonbr...@gmail.com] >>> *Enviado el:* 17 November 2021 17:43 >>> *Para:* Guillermo Ortas Delgado <g.or...@gmv.com>; Rob Kossler < >>> rkoss...@nd.edu> >>> *CC:* usrp-users@lists.ettus.com >>> *Asunto:* Re: [USRP-users] Re: DPDK drops samples at low rates >>> >>> >>> >>> On 2021-11-17 11:27, Guillermo Ortas Delgado wrote: >>> >>> Hi Marcus, thank you for your message. >>> >>> >>> >>> I do think the network layer is indeed the most significant part of this >>> challenging setup, let me illustrate: >>> >>> The platform I’m using is a server form-factor computer with a Xeon >>> Silver 4215R, 32GB of memory and no GPU. For storage, I have 4 SSDs of >>> 3.2TB mounted in RAID 0 providing a write capacity of 4GB/s, so that’s no >>> issue. I’m using all the optimizations mentioned here >>> <https://urldefense.com/v3/__https:/kb.ettus.com/Getting_Started_with_DPDK_and_UHD__;!!MvyJQugb!UJ8FfsPbUrq0LfNpjFLHfw9YyY1nkelByXNeHL_5zu_b0NDpEm1rMWVt0ZYW$> >>> and here >>> <https://urldefense.com/v3/__https:/kb.ettus.com/USRP_Host_Performance_Tuning_Tips_and_Tricks__;!!MvyJQugb!UJ8FfsPbUrq0LfNpjFLHfw9YyY1nkelByXNeHL_5zu_b0NDpEm1rMVnRj1fb$>, >>> so raw CPU power I think should be enough. >>> >>> Each of the four x310 USRPs is attached to a dedicated network card with >>> dual 10GbE ports, so network capacity is theoretically more than enough >>> (6.4Gb out of 10Gb maximum usage per interface). >>> >>> The scheme I’m trying to run is as follows: >>> >>> 1. USRP 1: 200 Msps on both channels >>> >>> 2. USRP 2: 200 Msps on both channels >>> >>> 3. USRP 3: 50 Msps on both channels >>> >>> 4. USRP 4: 50 Msps on both channels >>> >>> I’m streaming samples directly into shared memory, from which a separate >>> thread converts the samples to selectable 1/2/4/8/16 bits per sample and >>> stores them to the RAID 0 disk volume. Bit depth conversion is fast and >>> doesn’t seem to be the bottleneck. In fact, converting to 4 bits per sample >>> achieves better results than no conversion at all, forcing me to write the >>> full incoming 16 bits per sample. I launch a separate instance of my >>> program to store samples for each USRP, as I have observed this delivers >>> the best performance. >>> >>> With this, I’m able to run 4 channels at 200Msps and 2 channels at >>> 50Msps. But when I launch the last two channels at 50Msps the system can’t >>> keep up and the recording starts losing/dropping samples. >>> >>> I was able to run 4 channels at 184.32Msps and 4 channels at 46.08Msps >>> for a few seconds, but this is also not sustainable and samples are dropped >>> periodically. >>> >>> The application is very sensitive, so even a single sample lost or >>> dropped completely invalidates the recording. >>> >>> >>> >>> At these rates, the sheer amount of kernel systems calls seems to be the >>> most significant performance hit, that’s why I was looking at DPDK as a >>> potential solution. That being said, I’m able to sustain a solid stream >>> using the benchmark_rate program (discarding the samples) with 4 >>> channels at 200Msps and 4 channels at 50Msps without any drops/overflows. >>> >>> Do you have a sample program for high-performance/high-rate sample >>> streaming? The provided rx_samples_to_file is not nearly enough. >>> >>> What’s the preferred way to approach storing samples for maximum >>> performance? >>> >>> >>> >>> Thank you a lot and best regards, >>> >>> Guillermo >>> >>> 1Gsps is a totally *eye-watering* sample-rate for an ordinary computer >>> to "swallow" and write to disk. You are very likely at the very bleeding >>> edge with this, and >>> I'm not aware of anyone else doing work at these aggregate rates. The >>> fact that you are able to *both* "do stuff with the samples" AND write them >>> to a RAID >>> array at ~1Gsps is amazing. >>> >>> Do all USRPs have to be on the same computer for your application? Are >>> there opportunities to use a more distributed approach? >>> >>> >>> >>> *De:* Marcus D. Leech [mailto:patchvonbr...@gmail.com >>> <patchvonbr...@gmail.com>] >>> *Enviado el:* 17 November 2021 16:51 >>> *Para:* Guillermo Ortas Delgado <g.or...@gmv.com> <g.or...@gmv.com>; >>> Rob Kossler <rkoss...@nd.edu> <rkoss...@nd.edu> >>> *CC:* usrp-users@lists.ettus.com >>> *Asunto:* Re: [USRP-users] Re: DPDK drops samples at low rates >>> >>> >>> >>> On 2021-11-17 04:50, Guillermo Ortas Delgado wrote: >>> >>> Thanks for your message, I already have the mbuf size maxed out to 512 >>> (that’s the maximum value it will take). >>> >>> >>> >>> I have noticed that DPDPK v19.11 made great improvements to the BNXT >>> driver. Is there any chance to get UHD running with DPDK 19.11? Or even >>> better 20.11.3? >>> >>> Both are long-term support releases which are more mature and support >>> vector mode, which offers must better performance. >>> >>> Quote from DPDK 19.11: >>> “The BNXT PMD includes support for SSE vector mode on x86 platforms. >>> Vector provides *significantly improved performance* over the base >>> implementation” >>> >>> >>> >>> I already tried building UHD 4.1.0.4 with DPDK 19.11 by modifying the >>> makefile to accept this version, but the build fails. >>> >>> I would really appreciate it if you could add support for newer versions >>> of DPDK. >>> >>> >>> >>> Best, >>> >>> Guillermo >>> >>> >>> >>> >>> >>> P Please consider the environment before printing this e-mail. >>> >>> >>> >>> P Please consider the environment before printing this e-mail. >>> >> _______________________________________________ >> USRP-users mailing list -- usrp-users@lists.ettus.com >> To unsubscribe send an email to usrp-users-le...@lists.ettus.com >> > _______________________________________________ > USRP-users mailing list -- usrp-users@lists.ettus.com > To unsubscribe send an email to usrp-users-le...@lists.ettus.com >
_______________________________________________ USRP-users mailing list -- usrp-users@lists.ettus.com To unsubscribe send an email to usrp-users-le...@lists.ettus.com
