Hi Pat - Thanks for your sleuthing and info from Intel on how the XL710 Intel NIC allocates lanes when in 2x2x10 mode. This is certainly an issue with what the N321 expects. It would be desirable if the Intel NIC configuration utility allowed one to select which lanes to use, and/or if we could provide an FPGA image for different lane usage. I see no "fault" here on either side; we can all do better though! I'll bring this up with relevant Ettus folks to see what we can do from our end; obviously we can't control what Intel does. I like Rob's suggestion of using fiber & couplers (etc) ... I've never tried that either, but sounds like a possibility for overcoming this issue. I will look into acquiring the right parts to try out this suggestion. - MLD
On Thu, Feb 4, 2021 at 9:33 AM Rob Kossler <rkoss...@nd.edu> wrote: > Hi Pat, > Are you using fiber or copper to connect? If you are using fiber, it > seems to me it could work with a three QSFP optical transceivers and three > MTP/LC breakout fibers that you would connect via couplers such that Host > Lanes 0 and 1 go to the first N321 Lanes 0 and 1 while the Host Lanes 2 and > 3 go to the second N321 Lanes 0 and 1. The host NIC would be configured > 1x4. Note that I have not done this, but I am planning to purchase N321s > soon and so I am interested in this topic. > Rob > > On Wed, Feb 3, 2021 at 5:34 PM Patrick Kane via USRP-users < > usrp-users@lists.ettus.com> wrote: > >> I still cannot use a single 2-port QSFP+ NIC to connect to 2x N321s. >> Using the Intel tools to set the NIC to 2x2x10, the NIC doesn't recognize >> the second physical port as a valid connection to a USRP, but works as a >> loopback or a connection to another NIC. I asked Intel regarding the issues >> to see if we were configuring the NIC incorrectly, and this is the response >> we got: >> >> *Response from Intel on XL710-QDA2* >> >> “When configured for 2x2x10 the lanes are split between the ports. The A >> configuration utilizes lanes 1 and 2 on the top port and lanes 3 and 4 on >> the bottom port. These correlate to the physical connections on the QSFP+ >> module. If you are able to configure the port of the other device to >> utilize lanes 3 and 4, then it should be able to connect to the bottom port >> in the A configuration.” >> >> >> >> *Problem:* XQ FPGA image doesn’t accept connections from lanes 2 and 3, >> just 0 and 1. This prevents the bottom port on the Intel NIC from working >> in the 2x2x10 configuration: >> >> >> InterfaceHGXGWXXQAQ >> SFP+ 0 1 GbE 10 GbE White Rabbit White Rabbit 10 GbE >> SFP+ 1 10 GbE 10 GbE 10 GbE Unused 10 GbE >> QSFP+ lane 0 Unused Unused Unused 10 GbE Aurora >> QSFP+ lane 1 Unused Unused Unused 10 GbE Aurora >> QSFP+ lane 2 Unused Unused Unused *Unused* Aurora >> QSFP+ lane 3 Unused Unused Unused *Unused* Aurora >> In the 2 N321 configuration, lane 2 and 3 are valid, but point to a >> different USRP. I'm hoping theres a UHD or FW change that can prevent the >> QSFP->2x10G breakout cable because that defeats the purpose of using a >> QSFP+ NIC instead of a 4-port 10G NIC (Intel X710-DA4) >> >> Thanks, >> Pat >> >> On Tue, Nov 24, 2020 at 1:01 PM Michael Dickens < >> michael.dick...@ettus.com> wrote: >> >>> Hi Pat - I'm glad that info helped! >>> >>> Yes, I plan on adding this information into the N32x Getting Started >>> Guide, once I have a better handle on it. Right now I have just a few data >>> points & those are not consistent! and I don't know why! Thus ... >>> >>> Which Intel QSFP+ utility did you end up using? There are 2 that I can >>> find: >>> >>> 1) EPCT: >>> https://downloadcenter.intel.com/download/28933/Intel-Ethernet-Port-Configuration-Tool-Linux- >>> >>> This is the newer version that seems to work. >>> >>> 2) QCU : >>> https://downloadcenter.intel.com/download/25851/Intel-QSFP-Configuration-Utility-Linux-Final-Release?product=46828 >>> >>> This one is deprecated, though it still works to some extent. >>> >>> === >>> >>> When I execute (A), I get the following options: "4x10" and "2x2x10". I >>> do not get an "A" or "B" or "LOM" or whatever. Just literally those 2 >>> options. >>> >>> I think the first one means "1x(4x10)", meaning that just port0 is >>> active & provides 4 data lates. I was hoping this option would work with a >>> 1:4 SFP+ breakout cable from FS.com, but to the best of my testing I can >>> get just 1 of those 4 SFP+ links to work. Supposedly if one uses the Intel >>> 1:4 breakout cable this will work ... but that's paying $350 for an >>> otherwise $50 cable! I'm still investigating here. Ideally this NIC would >>> provide "2x(4x10)" with 2 1:4 breakout cables, to get double the SFP+ >>> density of current NICs (e.g., the X710-DA4 ort X722-DA4). >>> >>> The second one implies to me that both ports are available & providing 2 >>> data lanes each. The best I've been able to do is use "2x2x10" with port0; >>> port1 doesn't seem to be working in this setting. >>> >>> Admittedly, I might need to update to the current Intel Linux drivers >>> for the XL710 NIC. I usually let the OS handle this for me -- in this case, >>> Ubuntu 20.04 latest. There are new Intel drivers from early November 2020, >>> but I don't think the XL710 had any updates from the prior version. >>> >>> I'm curious what driver version & OS / version you're using ... maybe >>> let's catch up off-list for a bit & see what we can figure out here. >>> Cheers! - MLD >>> >>> On Tue, Nov 24, 2020 at 9:06 AM Patrick Kane <prkan...@gmail.com> wrote: >>> >>>> Hi Mike, >>>> >>>> That seemed to do the trick, thanks for info! At some point, can we >>>> make these steps part of the N32x getting started docs? >>>> >>>> Also- the config utilty makes me choose 2x2x10 A, B, or LOM. Choosing A >>>> disables the second port on the QDA2, and B disables the first port. LOM >>>> disables both ports (expected because it's not a MB NIC). My ideal case is >>>> using 2x N321s over QSFP on the same XL710-QDA2 NIC. Have you had any luck >>>> in this configuration? >>>> >>>> Thanks, >>>> Pat >>>> >>>> On Mon, Nov 23, 2020 at 9:23 PM Michael Dickens < >>>> michael.dick...@ettus.com> wrote: >>>> >>>>> Hi Pat - I recently verified that the N321 QAFP+ interface works with >>>>> UHD 4.0 release. I am also using an Intel XL710 (QDA2, but that probably >>>>> doesn’t matter too much). The trick for me was using the Intel QSFP+ NIC >>>>> configuration tool to set the NIC to 2x(2x10 Gb) mode. This is the setting >>>>> that the N321 requires, and one that the NIC provides. Once that was set >>>>> then you need to configures the host and USRP network interfaces as >>>>> you normally would. After all of that, the link worked very nicely! I hope >>>>> this is useful! - MLD >>>>> >>>>> On Nov 23, 2020, at 4:44 PM, Patrick Kane via USRP-users < >>>>> usrp-users@lists.ettus.com> wrote: >>>>> >>>>> >>>>> >>>>> I have an N321 connected to serial console and QSFP+ through a XL710 >>>>> Intel NIC. With the default HG image, I can connect through 1G and serial >>>>> as expected. I updated the filesystem to UHD 4.0.0.0 using mender, and the >>>>> build artifact reflects that this was successful. Then, after loading the >>>>> XQ image (using 2x 10Gb lanes through QSFP+ port), I lose all ethernet >>>>> connectivity through the 1G port SFP0 (expected), but I get the following >>>>> output in the console window: >>>>> >>>>> >>>>> [ 451.560674] nixge 40000000.ethernet sfp0: Link is Up - 10Gbps/Full >>>>> - flow control off >>>>> >>>>> [ 453.800673] nixge 40000000.ethernet sfp0: Link is Down >>>>> >>>>> [ 454.920676] nixge 40000000.ethernet sfp0: Link is Up - 10Gbps/Full >>>>> - flow control off >>>>> >>>>> [ 458.280672] nixge 40000000.ethernet sfp0: Link is Down >>>>> >>>>> [ 459.400677] nixge 40000000.ethernet sfp0: Link is Up - 10Gbps/Full >>>>> - flow control off >>>>> >>>>> [ 462.760705] nixge 40000000.ethernet sfp0: Link is Down >>>>> >>>>> [ 463.880678] nixge 40000000.ethernet sfp0: Link is Up - 10Gbps/Full >>>>> - flow control off >>>>> >>>>> [ 466.120673] nixge 40000000.ethernet sfp0: Link is Down >>>>> >>>>> >>>>> uhd_usrp_probe: >>>>> >>>>> _____________________________________________________ >>>>> >>>>> / >>>>> >>>>> | Device: N300-Series Device >>>>> >>>>> | _____________________________________________________ >>>>> >>>>> | / >>>>> >>>>> | | Mboard: ni-n3xx-31E00AC >>>>> >>>>> | | dboard_0_pid: 338 >>>>> >>>>> | | dboard_0_serial: 31DB406 >>>>> >>>>> | | dboard_1_pid: 338 >>>>> >>>>> | | dboard_1_serial: 31DB407 >>>>> >>>>> | | eeprom_version: 3 >>>>> >>>>> | | fs_version: 20200914000806 >>>>> >>>>> | | mender_artifact: v4.0.0.0_n3xx >>>>> >>>>> | | mpm_sw_version: 4.0.0.0-g90ce6062 >>>>> >>>>> | | pid: 16962 >>>>> >>>>> | | product: n320 >>>>> >>>>> | | rev: 7 >>>>> >>>>> | | rpc_connection: local >>>>> >>>>> | | serial: 31E00AC >>>>> >>>>> | | type: n3xx >>>>> >>>>> | | MPM Version: 3.0 >>>>> >>>>> | | FPGA Version: 8.0 >>>>> >>>>> | | FPGA git hash: be53058.clean >>>>> >>>>> | | >>>>> >>>>> | | Time sources: internal, external, gpsdo, sfp0 >>>>> >>>>> | | Clock sources: external, internal, gpsdo >>>>> >>>>> | | Sensors: ref_locked, gps_locked, temp, fan, gps_gpgga, >>>>> gps_sky, gps_time, gps_tpv >>>>> >>>>> >>>>> Are there any configuration items needed to connect to the N321 >>>>> through the QSFP+ port. Note that I only see eth0, sfp0, sfp1, and int0 in >>>>> /etc/network/interfaces. >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Pat >>>>> >>>>> >>>>> _______________________________________________ >>>>> USRP-users mailing list >>>>> USRP-users@lists.ettus.com >>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >>>>> >>>>> _______________________________________________ >> USRP-users mailing list >> USRP-users@lists.ettus.com >> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >> >
_______________________________________________ USRP-users mailing list USRP-users@lists.ettus.com http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
