Thanks Wade,
That fixes it for my test case with 2 switchboards in series. I'll let you
know if I see anything strange when testing with my custom blocks.
By the way, does this bug only affect N3xx devices or all RFNoC devices?
Rob
On Wed, Dec 16, 2020 at 5:33 PM Wade Fife <wade.f...@ettus.com> wrote:
> Rob,
>
> I think I found the source of the IQ swap issue you observed. This fixes
> it for me:
>
> diff --git a/host/lib/rfnoc/mgmt_portal.cpp
> b/host/lib/rfnoc/mgmt_portal.cpp
> index a54efaaf7..7b09c540b 100644
> --- a/host/lib/rfnoc/mgmt_portal.cpp
> +++ b/host/lib/rfnoc/mgmt_portal.cpp
> @@ -617,6 +617,24 @@ public:
> _send_recv_mgmt_transaction(xport, cfg_xact);
> }
>
> + // Build a management transaction to configure the destination
> node
> + {
> + mgmt_payload cfg_xact;
> + cfg_xact.set_header(my_epid, _protover, _chdr_w);
> + _traverse_to_node(cfg_xact, dst_node_addr);
> + mgmt_hop_t cfg_hop;
> + // Configure buffer types
> + cfg_hop.add_op(mgmt_op_t(mgmt_op_t::MGMT_OP_CFG_WR_REQ,
> + mgmt_op_t::cfg_payload(REG_ISTRM_CTRL_STATUS,
> + BUILD_CTRL_STATUS_WORD(false, false, BUFF_U64,
> BUFF_U64, false))));
> + // Return the packet back to us
> + cfg_hop.add_op(mgmt_op_t(mgmt_op_t::MGMT_OP_RETURN));
> + // Send the transaction and receive a response.
> + // We don't care about the contents of the response.
> + cfg_xact.add_hop(cfg_hop);
> + _send_recv_mgmt_transaction(xport, cfg_xact);
> + }
> +
> // Wait for stream configuration to finish on the HW side
> _validate_stream_setup(xport, src_node_addr, timeout);
>
> Thanks,
>
> Wade
>
> On Mon, Dec 14, 2020 at 8:32 PM Wade Fife <wade.f...@ettus.com> wrote:
>
>> Hi Rob,
>>
>> I was able to reproduce the issue, so I'll see if I can get to the bottom
>> of it. I'm glad you're able to work around it for now. Thanks for letting
>> us know about it!
>>
>> Wade
>>
>> On Mon, Dec 14, 2020 at 8:59 AM Rob Kossler <rkoss...@nd.edu> wrote:
>>
>>> Hi Wade,
>>> I tried the command to re-load the FPGA, but I couldn't communicate with
>>> UHD nicely until I also added the command "systemctl restart ursp-uhd". I
>>> am now able to reset the N310 to proper behavior when it gets in this state.
>>>
>>> Regarding this issue of real/imag swapping, do you want me to create a
>>> support request through supp...@ettus.com? Also, do you need me to
>>> provide any more info or an example program / FPGA image? Note that now
>>> that I know how to duplicate the issue, I believe I also know how to avoid
>>> it. So, it is probably not a critical issue for me presently.
>>>
>>> Rob
>>>
>>> On Sat, Dec 12, 2020 at 9:47 AM Wade Fife <wade.f...@ettus.com> wrote:
>>>
>>>> Thanks for the updates. If you just want to reload the FPGA, try
>>>> running "overlay rm n310 && overlay add n310" on the N310. This should
>>>> simply reload the FPGA using the bistream already in the flash.
>>>>
>>>> Wade
>>>>
>>>> On Fri, Dec 11, 2020 at 6:45 PM Rob Kossler <rkoss...@nd.edu> wrote:
>>>>
>>>>> Wade,
>>>>> The following also fails using just 2 blocks and 2 attempts:
>>>>> host_tx => Switchboard#0 => Switchboard#1 => host_rx // SUCCESS
>>>>> host_tx => Switchboard#1 => Switchboard#0 => host_rx // FAILURE (RX
>>>>> samples equal swapped I/Q of TX samples)
>>>>>
>>>>> In addition to wanting to get this issue fixed, I also have a question
>>>>> about the best way to "reboot" the N310 which is what I need to do to fix
>>>>> the issue after it occurs. One way is to give the "reboot now" linux
>>>>> command. But, this takes a long time. Another way is to reprogram the
>>>>> FPGA image via uhd_image_loader. This is appreciably faster, but I'm
>>>>> thinking that this is not a great idea if the flash memory has a limited
>>>>> number of write cycles before failure. Is there another way to achieve a
>>>>> reboot other than these two?
>>>>> Rob
>>>>>
>>>>> On Fri, Dec 11, 2020 at 7:34 PM Rob Kossler <rkoss...@nd.edu> wrote:
>>>>>
>>>>>> AHA! I duplicated the issue with the Switchboard image. The way to
>>>>>> duplicate the issue is the run the following series of graphs:
>>>>>> host_tx => Switchboard#0 => Switchboard#1 => host_rx // SUCCESS
>>>>>> host_tx => Switchboard#2 => Switchboard#3 => host_rx // SUCCESS
>>>>>> host_tx => Switchboard#0 => Switchboard#2 => host_rx // FAILURE
>>>>>> (RX samples equal swapped I/Q of TX samples)
>>>>>> Each of these 3 runs consists of running my application (similar to
>>>>>> one of the examples such as rfnoc_rx_to_file) such that the UHD object is
>>>>>> re-created each time. My guess is that you could use gnuradio to do it
>>>>>> instead.
>>>>>>
>>>>>> My working theory is that the problem is caused by the fact that the
>>>>>> Switchboard#2 input port was changed from being connected to the host in
>>>>>> test case 2 to then be connected to another RFNoC block in test case 3.
>>>>>> Or, I suppose it could be the output port on this block (same logic).
>>>>>>
>>>>>> If you want me to send you my FPGA image with the 4 Switchboard
>>>>>> blocks, let me know.
>>>>>> Rob
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Dec 11, 2020 at 7:09 PM Rob Kossler <rkoss...@nd.edu> wrote:
>>>>>>
>>>>>>> Hi Wade,
>>>>>>> After thinking about it a bit, I would ignore the "negation" issue
>>>>>>> for now. This may be a byproduct of I/Q swapping related to my custom
>>>>>>> block.
>>>>>>> Rob
>>>>>>>
>>>>>>> On Fri, Dec 11, 2020 at 6:34 PM Rob Kossler <rkoss...@nd.edu> wrote:
>>>>>>>
>>>>>>>> Hi Wade,
>>>>>>>> Thanks for the info. I still do not know what's going on, but here
>>>>>>>> are a few updates:
>>>>>>>>
>>>>>>>> - I built a new N310 image with 4 switchboards (1x1) and 1
>>>>>>>> splitstream (1x2) blocks in addition to the default blocks. All of
>>>>>>>> the
>>>>>>>> additional blocks are connected to SEPs for dynamic linking. I
>>>>>>>> tried my
>>>>>>>> best to get bad behavior using a chain of 1 to 4 switchboards, but
>>>>>>>> I could
>>>>>>>> not duplicate any I/Q swapping or negation issues.
>>>>>>>> - I went back to my custom image (that I described below) and
>>>>>>>> found different behavior today (sometimes things worked OK). So,
>>>>>>>> this got
>>>>>>>> me to thinking that I am somehow getting the FPGA in a bad state
>>>>>>>> such that
>>>>>>>> a reboot (or FPGA re-flashing) fixes things. I have had some
>>>>>>>> success with
>>>>>>>> re-booting and things working properly. But, I still do not have a
>>>>>>>> true
>>>>>>>> handle on things and cannot adequately predict when things are
>>>>>>>> going to
>>>>>>>> succeed or fail.
>>>>>>>> - One thing that has been constant is that I have never seen
>>>>>>>> bad behavior when I only have 1 block in my graph (no matter which
>>>>>>>> block I
>>>>>>>> choose). Note that for all of my tests, the graph looks like this:
>>>>>>>> host_tx
>>>>>>>> => block_chain => host_rx, where block_chain is a sequential chain
>>>>>>>> of 1 or
>>>>>>>> more rfnoc blocks.
>>>>>>>>
>>>>>>>> I will send a follow up once I figure things out.
>>>>>>>> Rob
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Dec 11, 2020 at 6:22 PM Wade Fife <wade.f...@ettus.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Rob,
>>>>>>>>>
>>>>>>>>> The SEPs do have the ability to swap I and Q. This is because on
>>>>>>>>> the host computer, I is usually in the lower bits and Q is in the
>>>>>>>>> upper
>>>>>>>>> bits of each 32-bit word, but in RFNoC, for historical reasons, I
>>>>>>>>> goes in
>>>>>>>>> the upper bits and Q in the lower bits. The software programs the IQ
>>>>>>>>> swapping when it sets up the graph.
>>>>>>>>>
>>>>>>>>> I assume you're using dynamic connections (through the crossbar)
>>>>>>>>> to control the number of FFTs the data is passed through, and not
>>>>>>>>> static
>>>>>>>>> connections? If that's the case then I wonder if software configures
>>>>>>>>> IQ
>>>>>>>>> swapping incorrectly in some configurations. I'll see if I can do some
>>>>>>>>> testing next week to confirm if it's working correctly.
>>>>>>>>>
>>>>>>>>> As for negation, I'm not aware of anywhere we do that off the top
>>>>>>>>> of my head. Is that behavior block dependent? I'll see if I can find
>>>>>>>>> anywhere this happens.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Wade
>>>>>>>>>
>>>>>>>>> On Thu, Dec 10, 2020 at 3:54 PM Rob Kossler via USRP-users <
>>>>>>>>> usrp-users@lists.ettus.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> I am encountering very strange behavior with a custom FPGA image
>>>>>>>>>> (N310). It appears that data streaming through SEPs can get swapped
>>>>>>>>>> (I/Q)
>>>>>>>>>> and/or negated. Is anyone at Ettus aware of anything that could
>>>>>>>>>> cause
>>>>>>>>>> this? Of course, the issue might be on my end, but I can't think of
>>>>>>>>>> what
>>>>>>>>>> it might be given that all of my custom blocks work as expected in
>>>>>>>>>> isolation (if the block is the only block in graph).
>>>>>>>>>>
>>>>>>>>>> My custom image is the following:
>>>>>>>>>>
>>>>>>>>>> - default blocks of Radios, DDCs, DUCs (each 2x2 and
>>>>>>>>>> statically connected as in default image)
>>>>>>>>>> - custom blocks of two 1x1 windowed-fft blocks, two 1x1
>>>>>>>>>> vector-avg blocks, and one 2x2 custom block. Note: each of these
>>>>>>>>>> blocks is
>>>>>>>>>> connected to its own SEP, so I can connect dynamically in any
>>>>>>>>>> fashion.
>>>>>>>>>>
>>>>>>>>>> My test case is transmitting 8192 random samples from host to FFT
>>>>>>>>>> block and then optionally through a 2nd FFT block before back to
>>>>>>>>>> host. In
>>>>>>>>>> the test case, the radios/DDCs/DUCs are not used.
>>>>>>>>>>
>>>>>>>>>> Here is what I observed:
>>>>>>>>>>
>>>>>>>>>> - If I only include 1 FFT block in my RFNoC graph, I get the
>>>>>>>>>> expected results (the output from the FPGA matches what I
>>>>>>>>>> calculate in
>>>>>>>>>> Matlab for the FFT). This is true for either of the two FFT
>>>>>>>>>> blocks.
>>>>>>>>>> - If I include both FFT blocks in series, I can only match
>>>>>>>>>> the FPGA output if I swap the I/Q values in between my Matlab
>>>>>>>>>> FFTs.
>>>>>>>>>> - Note: that this issue is not FFT-related as I can also
>>>>>>>>>> duplicate this issue with the other blocks.
>>>>>>>>>> - If I use 3 blocks in series (each through SEP), I need to
>>>>>>>>>> negate certain data in order to get it to match the FPGA output
>>>>>>>>>>
>>>>>>>>>> My next step is likely to build a new image with Ettus-developed
>>>>>>>>>> FIFOs to prove that the data is getting swapped/negated when 2 or
>>>>>>>>>> more are
>>>>>>>>>> used in series through SEPs.
>>>>>>>>>>
>>>>>>>>>> Let me know if you have any suggestions for other things to try.
>>>>>>>>>>
>>>>>>>>>> Rob
>>>>>>>>>> _______________________________________________
>>>>>>>>>> USRP-users mailing list
>>>>>>>>>> USRP-users@lists.ettus.com
>>>>>>>>>> http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
>>>>>>>>>>
>>>>>>>>>
_______________________________________________
USRP-users mailing list
USRP-users@lists.ettus.com
http://lists.ettus.com/mailman/listinfo/usrp-users_lists.ettus.com
