Hi Jim, Thanks for your prompt reply, I really appreciate it. I understand that ZMQ in a way defines a protocol and it is on me to decode it by providing the proper interfaces. However, Gnuradio ZMQ SUB Source for instance can read whatever arbitrary tags I throw at it if I simply turn on the "Pass Tags" flag to yes. I tried to understand how it is done by investigating the source code but I don't really know c++ or zmq that well so I was a little lost. I don't know whether it would be possible to reproduce such behavior in pure Python, but if you have any suggestions or pointers there I would love to hear them.
I also realized that when I am receiving say 10000 sample long IQ vector from ZMQ, sometimes the messages arrive in 2 parts (say 8000, 2000) and sometimes in many parts where some parts can be as small as 20 long. Is this behavior controlled by the OS kernel or is this something that can also be controlled by me. Also when cutting the incoming messages manually I usually chop the pieces guided by intuition and if things don't align, I just re-cut the piece to the right or to the left until it decodes properly. Is there any rule of thumb regarding how these messages are constructed? For instance, how long is the header for both the first piece and the following pieces. If I have a gr.tag_t() where the tag value is defined through pmt.from_double, pmt.from_long or pmt.to_pmt is there a simple rule to determine how long the corresponding bytes should be to properly decode via pmt.deserialize_str? Thanks once again for your reply. Best, Temir On Fri, Oct 29, 2021 at 12:31 AM Jim Melton <jim.mel...@sncorp.com> wrote: > Your ZMQ messaging is intrinsically defining a protocol. If you change the > protocol, then you necessarily have to re-code the interfaces that use it. > > > > If you are sending your data as a ZMQ multi-part message, be aware that > ZMQ delivers multi-part messages atomically. That is, although you have to > code to receive all the parts (Python is easier, with *recv_multipart*) > then deserialize the parts into your data structure(s), there is no delay > between “receiving” the first part and receiving the last part. The > receipt/assembly of multi-part messages is handled in the ZMQ core. > > > > You have to have control of at least one side of this interface. If you > control the sender, you could choose to encode the data using PMT (see > *gr::zeromq::pub_msg_sink*(*_impl*) for example) and send a single > serialized message vs. multipart messages. > > > > If you only control the receiver, you are stuck conforming to whatever the > sender produces. > > > > --- > > *Jim Melton* > > > > *From:* Discuss-gnuradio <discuss-gnuradio-bounces+jim.melton= > sncorp....@gnu.org> *On Behalf Of *Temir Karakurum > *Sent:* Thursday, October 28, 2021 14:14 > *To:* discuss-gnuradio@gnu.org > *Subject:* [EXTERNAL] Reading tags from ZMQ in Python > > > > Hi, > > > > I am looking for a general solution to separate IQ data and tags coming > from ZMQ using python. I send RPC requests to receive finite samples via > ZMQ. ZMQ messages arrive in many pieces where the first piece has a header > and the tags whereas the rest of the pieces include a short header. > > > > Currently I can separate the incoming zmq messages by manually separating > the received message into header_magic, header_version, number of received > tags, and so on by manually cutting the received message into properly > demarcated pieces. Then I can decode the pieces using either > pmt.deserialize_str or np.frombuffer. > > > > However, a slight change in the tag or message architecture usually leads > to a rewrite of the code. I was wondering whether there is a more general > and robust solution regarding reading tagged messages arriving via ZMQ. If > anyone can point me to some existing examples available within the codebase > or share their snippet for similar purposes I would be grateful. > > > > Best, > > Temir > CONFIDENTIALITY NOTICE - SNC EMAIL: This email and any attachments are > confidential, may contain proprietary, protected, or export controlled > information, and are intended for the use of the intended recipients only. > Any review, reliance, distribution, disclosure, or forwarding of this email > and/or attachments outside of Sierra Nevada Corporation (SNC) without > express written approval of the sender, except to the extent required to > further properly approved SNC business purposes, is strictly prohibited. If > you are not the intended recipient of this email, please notify the sender > immediately, and delete all copies without reading, printing, or saving in > any manner. --- Thank You. >
