Hi Ian,
OK - is there code for the dissector available to build a new version?
Or do you think it's worth it?
Also - our host sending the samples was actually only 1Gbps ethernet (we
weren't using the 10Gbps card for that), so I'm not sure that the
samples would be going into the switch faster than this. We were using
the 10G card for wireshark.
Thanks for the info about the N210 ethernet port, maybe we can try
forcing the switch ports to 1G and see if that helps.
Also adding a 1G switch in between the USRP and 10G will be a good test
if we can get our hands on a 1G switch.
We discussed just getting four 4port 1G cards, we would need to upgrade
our motherboard though.
Thanks, If we figure anything out we'll post back here.
Regards,
Kevin
On 11/09/2017 09:14 AM, Ian Buckley wrote:
Kevin,
I glanced through Captures 0,1,2 and could see the general gist of
where it all goes wrong, but it’s not really clear why. The dissector
seems out of date, at least it isn’t decoding correctly but I can
still follow it.
Capture 0:
Packets 2551 through 2606 are the actual sample data packets.
Then there is an approx 1.2second pause before UHD, having timed out
waiting for ACK’s begins some basic control interaction initiated by
the host.
Contrast that with Capture 2, where you will see a nicely paced stream
of ACK’s (starts at packet 2609) flowing back from the USRP at exact
intervals pacing the real time TX operation as the DSP consumes the
samples.
Capture 1:
This goes to the weeds early, streaming never gets close to starting.
At around packet 1324 you suddenly see a series of pauses for exactly
0.5 secs each.
So a couple of thoughts bordering on guesses:
Rate conversion in switches is always a little bit of a minefield when
you have a single stream that is faster than the skinny pipe. In this
case, operations like the sample burst are streaming from the host
faster than 1Gbs into the switch forcing the switch to buffer samples.
It’s surprising sometimes how quick these lower cost single chip
switches can get buffer starved. Try using 'ethtool' on the Linux host
to force the host port to 1Gbps to eliminate rate conversion in the
switch as a cause.
N210 in general is the most reliable of all USRP’s but it has one
quirk, which is that the ethernet port is forced to be 1Gbps and it
does no auto negotiation. This often trips up people who connect it to
a 100mbit switch or host. I have never actually tried connecting it to
10GBase-T switches…there may be some (switch implementation specific)
gotcha’s.
Mirroring multiple ports to a single port can affect how the problem
manifests because you create new bottle necks in the switch …if the
10G switch is suspect then I suggest interposing a cheap 1G switch
that supports mirroring between USRP and 10G switch, that way you’ll
have higher confidence exactly what and when went onto the wire. This
is also an interesting experiment because it isolates the N210’s
ethernet port from the 10G switch…curious to see if it now starts to
work OK. I have had this happen when debugging other products.
The ICMP port unreachable is likely a red herring, some spurious Linux
service rather than UHD originating those packets I suspect.
Might be worth looking at the switch internal counters to see if any
physical layer error counters are increasing.
No smoking gun I’m afraid; your fall back plan might be to stuff quad
port 1G cards in your host, they’re cheap if you have the PCIe slots free.
-Ian
On Nov 6, 2017, at 3:13 PM, Kevin Krieger via USRP-users
<usrp-users@lists.ettus.com <mailto:usrp-users@lists.ettus.com>> wrote:
Hi,
Any chance to look at these wireshark captures?
Thanks,
Kevin
On 10/30/2017 05:30 PM, Kevin Krieger via USRP-users wrote:
Hi Marcus,
I finally got around to this.
I've attached 9 captures.
The setup for the captures was:
XS728T with latest firmware running, factory default
Computer running tx_bursts has IP of 192.168.10.60 or .67
N200 has IP of 192.168.10.107
./tx_bursts --args addr0=192.168.10.107 --freq 10e6 --rate 1e6
Captures 0,1 and 2 are with both the ports (n200 side, and computer
side) mirrored to the wireshark capture port, so you essentially see
duplicate traffic.
Captures 3,4 and 5 are with only the computer side traffic captured
Captures 6,7 and 8 are with only the N200 side traffic captured.
Captures 0, 4, and 8 are with the messages "Sent packet: 363
samples" appearing, but then the "Waiting for async burst ACK ...
fail" message also appears.
Captures 1, 3 and 7 are with the message :Error: Runtimeerror: fifo
ctrl timed out looking for acks" appearing.
Captures 2, 5 and 6 are with successful runs of the tx bursts
program, so instead of 'fail' at the end there is 'success'.
There are txt files with the output of the program for each capture,
as well as csv files showing the dissected output (only found uhd in
the dissectors).
Can you see anything wrong with these runs? Any help is appreciated.
Thanks,
Kevin
On 06/15/2017 02:28 PM, Kevin Krieger via USRP-users wrote:
Hi Marcus,
Thank you - I was sure that the same subnet was correct but it's
good to have confirmation.
I have done the dissecting previously, and I recall getting a bunch
of "huh what's that?"' messages and then a 'wassup bro' or
something but I'll do it again and let you know what I see.
(Hilarious descriptions btw!)
Kevin
On 05/24/2017 08:07 PM, Marcus Müller via USRP-users wrote:
Hi Kevin, Hi Robin,
the same-subnet configuration is the right one, here. There's no
routing ambiguity if you have one card that serves the whole switch.
Kevin, this might be much to ask, but: I'd ask you to both install
wireshark and its development headers (typically, wireshark-dev or
wireshark-devel) from your operating system's package manager.
Then, get the UHD source code, and
cd /path/to/uhd/tools/dissectors
mkdir build
cd build
cmake -DETTUS_DISSECTOR_NAME=zpu ..
make -j4 && make install
Ideally, add your user to the group that has access to the raw
network hardware (in case of my OS, that was the "wireshark"
group), log out and back in.
If that doesn't work, you have to record as root and analyze as
your user (the USRP protocol dissectors got installed to your home
directory).
Then, record the traffic on your 10G interface, and analyze the
packets in question as UHD (some/many will be sample packets, i.e.
CVITA).
Best regards,
Marcus
On 24.05.2017 20:15, ROBIN TORTORA via USRP-users wrote:
The only thing that jumps out to me is they are all on the same
subnet...
That is something I have typically avoided, but if you say it
works in another configuration, that may not be the issue...
On May 24, 2017 at 12:12 PM Kevin Krieger via USRP-users
<usrp-users@lists.ettus.com> wrote:
Hi all,
I've got a predicament. Here's some *background* before I
describe my problem:
We purchased a Netgear xs728T 10GB switch in order to network 16
N200 devices, as well as a processing computer (which has a 10G
ethernet card). Our 16 N200 devices are going to be used in a
radar configuration where they are all simultaneously either
transmitting or receiving. They are all receiving coincident
10MHz and 1PPS signals from 2 octoclocks, which are both fed
from a third clock to ensure coincident clocks. They are
connected to the switch via cat6 cables ~7feet in length (this
item:
https://www.newegg.ca/Product/Product.aspx?Item=N82E16812119168&_ga=2.227605844.817843194.1495640587-1295759170.1493911136)
*The problem:*
During testing we've found that the example program tx_bursts,
when run on one USRP at a time with a modest bandwidth, the
"Waiting for async burst ack....failed" message appears about
75% of the time. If I try to use two or more USRPs at a time,
then it appears every time.
The line used for tx_bursts was typically something like
this:./tx_bursts --args="addr0=192.168.10.104" --freq 10e6
--rate=1e6 --channels="0"
for a one usrp test. For a multiple usrp test, I used lines like
this:./tx_bursts
--args="addr0=192.168.10.101,addr1=192.168.10.102,addr2=192.168.10.103,addr3=192.168.10.100,addr4=192.168.10.112,addr5=192.168.10.113,addr6=192.168.10.114"
--freq 10e6 --rate=1e6 --channels="0,1,2,3,4,5,6"
*Some evidence & information:
*We also have a 5 port 1G DLink DGS-1005G. We've tested 4 N200s
with tx_bursts on this switch (same cabling) and it works
flawlessly.
We also have tested a second 10G switch, the 8 port XS708E with
7 N200s with tx_bursts and it works flawlessly.
I have wireshark dumps taken via a third machine that is
connected to mirrored ports on the xs728T switch. I have
attached them in case anyone can tell me if they see something
wrong besides the missing acks.
The wireshark filter to use is : ip.src == 192.168.10.104 and
ip.dst == 192.168.10.104 or ip.src == 192.168.10.104 and ip.dst
== 192.168.10.99
The xs728t switch information is: Boot version is 1.0.0.0, SW
version is 6.5.1.26 (latest as of testing).
I contacted Netgear support, and they sent us a brand new
switch, which exhibited the same problem.
I'm wondering if there's anyone out there who has had similar
issues? Is there anything I can do to get more information or
get around this problem?
Can anyone see what the root cause might be in the wireshark
captures?
Any help or pointers are appreciated. Thank you,
Kevin
_______________________________________________
USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
_______________________________________________
USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
_______________________________________________
USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
_______________________________________________
USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
_______________________________________________
USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
_______________________________________________
USRP-users mailing list
USRP-users@lists.ettus.com <mailto:USRP-users@lists.ettus.com>
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
_______________________________________________
USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
