I should add that when I said "I have not done this", I meant with N321s in this proposed configuration. However, I am presently using an XL710 (QDA1) with a QSFP+ optical transceiver and MTP/LC breakout fiber to connect to two N310s (using SFP+ optical transceivers). I have done this both with single-mode and multimode fiber. Rob
On Thu, Feb 4, 2021 at 9:32 AM Rob Kossler <rkoss...@nd.edu> wrote: > Hi Pat, > Are you using fiber or copper to connect? If you are using fiber, it > seems to me it could work with a three QSFP optical transceivers and three > MTP/LC breakout fibers that you would connect via couplers such that Host > Lanes 0 and 1 go to the first N321 Lanes 0 and 1 while the Host Lanes 2 and > 3 go to the second N321 Lanes 0 and 1. The host NIC would be configured > 1x4. Note that I have not done this, but I am planning to purchase N321s > soon and so I am interested in this topic. > Rob > > On Wed, Feb 3, 2021 at 5:34 PM Patrick Kane via USRP-users < > usrp-users@lists.ettus.com> wrote: > >> I still cannot use a single 2-port QSFP+ NIC to connect to 2x N321s. >> Using the Intel tools to set the NIC to 2x2x10, the NIC doesn't recognize >> the second physical port as a valid connection to a USRP, but works as a >> loopback or a connection to another NIC. I asked Intel regarding the issues >> to see if we were configuring the NIC incorrectly, and this is the response >> we got: >> >> *Response from Intel on XL710-QDA2* >> >> “When configured for 2x2x10 the lanes are split between the ports. The A >> configuration utilizes lanes 1 and 2 on the top port and lanes 3 and 4 on >> the bottom port. These correlate to the physical connections on the QSFP+ >> module. If you are able to configure the port of the other device to >> utilize lanes 3 and 4, then it should be able to connect to the bottom port >> in the A configuration.” >> >> >> >> *Problem:* XQ FPGA image doesn’t accept connections from lanes 2 and 3, >> just 0 and 1. This prevents the bottom port on the Intel NIC from working >> in the 2x2x10 configuration: >> >> >> InterfaceHGXGWXXQAQ >> SFP+ 0 1 GbE 10 GbE White Rabbit White Rabbit 10 GbE >> SFP+ 1 10 GbE 10 GbE 10 GbE Unused 10 GbE >> QSFP+ lane 0 Unused Unused Unused 10 GbE Aurora >> QSFP+ lane 1 Unused Unused Unused 10 GbE Aurora >> QSFP+ lane 2 Unused Unused Unused *Unused* Aurora >> QSFP+ lane 3 Unused Unused Unused *Unused* Aurora >> In the 2 N321 configuration, lane 2 and 3 are valid, but point to a >> different USRP. I'm hoping theres a UHD or FW change that can prevent the >> QSFP->2x10G breakout cable because that defeats the purpose of using a >> QSFP+ NIC instead of a 4-port 10G NIC (Intel X710-DA4) >> >> Thanks, >> Pat >> >> On Tue, Nov 24, 2020 at 1:01 PM Michael Dickens < >> michael.dick...@ettus.com> wrote: >> >>> Hi Pat - I'm glad that info helped! >>> >>> Yes, I plan on adding this information into the N32x Getting Started >>> Guide, once I have a better handle on it. Right now I have just a few data >>> points & those are not consistent! and I don't know why! Thus ... >>> >>> Which Intel QSFP+ utility did you end up using? There are 2 that I can >>> find: >>> >>> 1) EPCT: >>> https://downloadcenter.intel.com/download/28933/Intel-Ethernet-Port-Configuration-Tool-Linux- >>> >>> This is the newer version that seems to work. >>> >>> 2) QCU : >>> https://downloadcenter.intel.com/download/25851/Intel-QSFP-Configuration-Utility-Linux-Final-Release?product=46828 >>> >>> This one is deprecated, though it still works to some extent. >>> >>> === >>> >>> When I execute (A), I get the following options: "4x10" and "2x2x10". I >>> do not get an "A" or "B" or "LOM" or whatever. Just literally those 2 >>> options. >>> >>> I think the first one means "1x(4x10)", meaning that just port0 is >>> active & provides 4 data lates. I was hoping this option would work with a >>> 1:4 SFP+ breakout cable from FS.com, but to the best of my testing I can >>> get just 1 of those 4 SFP+ links to work. Supposedly if one uses the Intel >>> 1:4 breakout cable this will work ... but that's paying $350 for an >>> otherwise $50 cable! I'm still investigating here. Ideally this NIC would >>> provide "2x(4x10)" with 2 1:4 breakout cables, to get double the SFP+ >>> density of current NICs (e.g., the X710-DA4 ort X722-DA4). >>> >>> The second one implies to me that both ports are available & providing 2 >>> data lanes each. The best I've been able to do is use "2x2x10" with port0; >>> port1 doesn't seem to be working in this setting. >>> >>> Admittedly, I might need to update to the current Intel Linux drivers >>> for the XL710 NIC. I usually let the OS handle this for me -- in this case, >>> Ubuntu 20.04 latest. There are new Intel drivers from early November 2020, >>> but I don't think the XL710 had any updates from the prior version. >>> >>> I'm curious what driver version & OS / version you're using ... maybe >>> let's catch up off-list for a bit & see what we can figure out here. >>> Cheers! - MLD >>> >>> On Tue, Nov 24, 2020 at 9:06 AM Patrick Kane <prkan...@gmail.com> wrote: >>> >>>> Hi Mike, >>>> >>>> That seemed to do the trick, thanks for info! At some point, can we >>>> make these steps part of the N32x getting started docs? >>>> >>>> Also- the config utilty makes me choose 2x2x10 A, B, or LOM. Choosing A >>>> disables the second port on the QDA2, and B disables the first port. LOM >>>> disables both ports (expected because it's not a MB NIC). My ideal case is >>>> using 2x N321s over QSFP on the same XL710-QDA2 NIC. Have you had any luck >>>> in this configuration? >>>> >>>> Thanks, >>>> Pat >>>> >>>> On Mon, Nov 23, 2020 at 9:23 PM Michael Dickens < >>>> michael.dick...@ettus.com> wrote: >>>> >>>>> Hi Pat - I recently verified that the N321 QAFP+ interface works with >>>>> UHD 4.0 release. I am also using an Intel XL710 (QDA2, but that probably >>>>> doesn’t matter too much). The trick for me was using the Intel QSFP+ NIC >>>>> configuration tool to set the NIC to 2x(2x10 Gb) mode. This is the setting >>>>> that the N321 requires, and one that the NIC provides. Once that was set >>>>> then you need to configures the host and USRP network interfaces as >>>>> you normally would. After all of that, the link worked very nicely! I hope >>>>> this is useful! - MLD >>>>> >>>>> On Nov 23, 2020, at 4:44 PM, Patrick Kane via USRP-users < >>>>> usrp-users@lists.ettus.com> wrote: >>>>> >>>>> >>>>> >>>>> I have an N321 connected to serial console and QSFP+ through a XL710 >>>>> Intel NIC. With the default HG image, I can connect through 1G and serial >>>>> as expected. I updated the filesystem to UHD 4.0.0.0 using mender, and the >>>>> build artifact reflects that this was successful. Then, after loading the >>>>> XQ image (using 2x 10Gb lanes through QSFP+ port), I lose all ethernet >>>>> connectivity through the 1G port SFP0 (expected), but I get the following >>>>> output in the console window: >>>>> >>>>> >>>>> [ 451.560674] nixge 40000000.ethernet sfp0: Link is Up - 10Gbps/Full >>>>> - flow control off >>>>> >>>>> [ 453.800673] nixge 40000000.ethernet sfp0: Link is Down >>>>> >>>>> [ 454.920676] nixge 40000000.ethernet sfp0: Link is Up - 10Gbps/Full >>>>> - flow control off >>>>> >>>>> [ 458.280672] nixge 40000000.ethernet sfp0: Link is Down >>>>> >>>>> [ 459.400677] nixge 40000000.ethernet sfp0: Link is Up - 10Gbps/Full >>>>> - flow control off >>>>> >>>>> [ 462.760705] nixge 40000000.ethernet sfp0: Link is Down >>>>> >>>>> [ 463.880678] nixge 40000000.ethernet sfp0: Link is Up - 10Gbps/Full >>>>> - flow control off >>>>> >>>>> [ 466.120673] nixge 40000000.ethernet sfp0: Link is Down >>>>> >>>>> >>>>> uhd_usrp_probe: >>>>> >>>>> _____________________________________________________ >>>>> >>>>> / >>>>> >>>>> | Device: N300-Series Device >>>>> >>>>> | _____________________________________________________ >>>>> >>>>> | / >>>>> >>>>> | | Mboard: ni-n3xx-31E00AC >>>>> >>>>> | | dboard_0_pid: 338 >>>>> >>>>> | | dboard_0_serial: 31DB406 >>>>> >>>>> | | dboard_1_pid: 338 >>>>> >>>>> | | dboard_1_serial: 31DB407 >>>>> >>>>> | | eeprom_version: 3 >>>>> >>>>> | | fs_version: 20200914000806 >>>>> >>>>> | | mender_artifact: v4.0.0.0_n3xx >>>>> >>>>> | | mpm_sw_version: 4.0.0.0-g90ce6062 >>>>> >>>>> | | pid: 16962 >>>>> >>>>> | | product: n320 >>>>> >>>>> | | rev: 7 >>>>> >>>>> | | rpc_connection: local >>>>> >>>>> | | serial: 31E00AC >>>>> >>>>> | | type: n3xx >>>>> >>>>> | | MPM Version: 3.0 >>>>> >>>>> | | FPGA Version: 8.0 >>>>> >>>>> | | FPGA git hash: be53058.clean >>>>> >>>>> | | >>>>> >>>>> | | Time sources: internal, external, gpsdo, sfp0 >>>>> >>>>> | | Clock sources: external, internal, gpsdo >>>>> >>>>> | | Sensors: ref_locked, gps_locked, temp, fan, gps_gpgga, >>>>> gps_sky, gps_time, gps_tpv >>>>> >>>>> >>>>> Are there any configuration items needed to connect to the N321 >>>>> through the QSFP+ port. Note that I only see eth0, sfp0, sfp1, and int0 in >>>>> /etc/network/interfaces. >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Pat >>>>> >>>>> >>>>> _______________________________________________ >>>>> USRP-users mailing list >>>>> USRP-users@lists.ettus.com >>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >>>>> >>>>> _______________________________________________ >> USRP-users mailing list >> USRP-users@lists.ettus.com >> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com >> >
_______________________________________________ USRP-users mailing list USRP-users@lists.ettus.com http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
