Thanks for the ideas Marcus. I did check ifconfig/ethtool, and my desktop debugging setup shows no RX errors on the corresponding 10Gbps interface. I'll keep monitoring just in case it's a matter of time until something shows up. I've been testing for a day or so now . . .
I forgot to mention in the first post that we have a number of devices other than Ettus radios connected to the 100Gbps switch, and it's only the N320's that are exhibiting this problem. Thanks, Jim ________________________________ From: Marcus D. Leech <patchvonbr...@gmail.com> Sent: Thursday, April 20, 2023 10:30 AM To: usrp-users@lists.ettus.com <usrp-users@lists.ettus.com> Subject: [USRP-users] Re: N320 SFP MAC issue On 20/04/2023 10:17, Jim Palladino wrote: Hello, We have multiple N320's connected to a managed 100Gbps switch. Each switch port has a QSFP -- so 4 10Gbps connections per port. We're connecting and using both 10Gbps SFP interfaces on each N320 radio to the switch. Basically, one 100Gbps port on the switch will handle 2 N320's via the QSFP. This has been working reliably for over a year using UHD 4.1.0.5. Recently, we updated to UHD 4.4.0.0. Now, everything seems to work fine for a finite amount of time (hours/days), but then port security on our switch gets tripped for one of the ports. That radio's corresponding SFP interface is then unusable until that port gets reset on our managed switch. What seems to be happening is that our switch is configured to only allow 1 MAC address per "interface". If a different MAC address ever shows up, port security trips. Due to security constaints, we're required to run with this MAC limitation. To debug this issue, our network administrators temporarily increased the number of allowed MAC addresses per 10Gbps interface to be set to 3 instead of 1. They were able to see in switch logs that eventually a MAC other than that programmed in the N320s EEPROM showed up on that port. The offending MAC (not the proper MAC for that SFP N320 SFP interface) was "00:00:b8:ce:f6:22". We can't figure out where this is coming from, and haven't been able to determine if this happens when rebooting a radio, loading the FPGA, bringing up the SFP interfaces, randomly during streaming, or what. A thought is that at some point (when the FPGA is programmed and or the SFP interface comes up) some garbage bits come out of the interface -- maybe the switch interprets this as some sort of malformed packet? I've been running tests at my desk with an N320 connected directly to a 10Gbps interface on my desk PC trying to somewhat reproduce the issue. I've been running loops that reboot the N320, stream samples from the N320, reboot while streaming, try to start streaming before the SFP interface is up, etc, etc. With wireshark I've been watching and have not seen any packets with MACs other than the proper MAC that the N320 should assign to that interface. However we're wondering if maybe we wouldn't see a malformed packet on wireshark (might get blocked by the interface and not get passed up the stack?). However, maybe in our normal setup the CISCO switch might see it? We haven't been able to get logs from the switch that show anything beyond the fact that another MAC showed up on that port at some point. Sorry for all the words, but this has been a tough one to debug/reproduce. We've had these issues with all 5 N320s we have connected to the switch. Again, we never saw this before updating to UHD 4.4.0.0. So either this is related to N320 behavior that changed when updating UHD 4.4.0.0 or something else coincidentally happened at/around the same time as the update. If anyone has any ideas it would be appreciated. Thanks, Jim That MAC prefix is assigned to Seikosha -- part of the Seiko group of companies. I don't believe that anything in the N320 uses parts made by them -- and certainly not the MAC bits and pieces, which live inside the FPGA. Normally, if a device "babbles" a bunch of noise bits, that's an invalid frame, since the CRC won't pass, and tools like WireShark will never see those frames because they'll get dropped at a much lower layer in the stack. You might want to look at "ifconfig" on the relevant interface to see if RX errors are being logged. If the CISCO switch is showing "invalid MAC received" for "babble frames", I don't think that's correct behavior, but not having played in that space for many years, I'm not sure.
_______________________________________________ USRP-users mailing list -- usrp-users@lists.ettus.com To unsubscribe send an email to usrp-users-le...@lists.ettus.com
