Thanks, I'll try the StringOutputStream and see what happens. Double-copying isn't a big concern for us because our protocol is not high-frequency.
On Sat, Jul 7, 2018 at 2:41 PM Ilia Mirkin <[email protected]> wrote: > It's been nearly a decade since I've looked at those APIs closely, > hopefully someone else can elaborate. If what you want is a data > buffer you can then copy somewhere else, you either need something > dynamically sizeable, or just decree what the max size is, and > allocate that. Even better would be to avoid the additional copy, but > perhaps your message broker won't allow that. > > On Sat, Jul 7, 2018 at 4:34 PM, John Lilley <[email protected]> wrote: > > If I want to write header+body to an array of bytes (in C++), is the > easiest > > thing to use StringOutputStream, then copy its buffer when finished? > > I also looked at ArrayOutputStream, but at first read it appears to > require > > knowledge of the output size before constructing the stream. True? > > Thanks > > john > > > > > > On Sat, Jul 7, 2018 at 2:18 PM John Lilley <[email protected]> wrote: > >> > >> Got it, thanks! > >> john > >> > >> On Sat, Jul 7, 2018 at 2:13 PM Ilia Mirkin <[email protected]> > wrote: > >>> > >>> CodingInputStream/OutputStream have all that. readInt32/etc. > >>> > >>> There's no strict advantage... but presumably you're using protobuf to > >>> make your life easier, and this will make your life easier. (With a > >>> string, you have to include the length, etc. And if the header ever > >>> changes and you want to have back/forward compat, it can be > >>> convenient.) > >>> > >>> On Sat, Jul 7, 2018 at 4:03 PM, John Lilley <[email protected]> wrote: > >>> > Does protobuf include utility methods for direct ser/deser on varint, > >>> > string, etc? > >>> > Thanks > >>> > john > >>> > > >>> > On Sat, Jul 7, 2018 at 2:02 PM John Lilley <[email protected]> > wrote: > >>> >> > >>> >> Thanks! > >>> >> Given that, is there any advantage to a "header message" as opposed > to > >>> >> just hand-serializing everything in the header? > >>> >> > >>> >> > >>> >> On Sat, Jul 7, 2018 at 12:45 PM Ilia Mirkin <[email protected]> > >>> >> wrote: > >>> >>> > >>> >>> You need explicit lengths. Usually this is done as <header length > >>> >>> varint><header><body>. And the header contains the body length in > it. > >>> >>> In Java, there's a CodedInputStream/OutputStream which makes it > easy > >>> >>> to consume fixed lengths (push/popLimit) as well as raw varints (as > >>> >>> for the initial header length). Other languages have similar > >>> >>> abstractions. > >>> >>> > >>> >>> On Sat, Jul 7, 2018 at 2:26 PM, John Lilley <[email protected]> > >>> >>> wrote: > >>> >>> > I am posting protobuf messages to a message broker, and in order > to > >>> >>> > identify > >>> >>> > them, I prefix the message bytes with the serialized result of a > >>> >>> > "header" > >>> >>> > message: > >>> >>> > > >>> >>> > message Header { > >>> >>> > int version = 1; > >>> >>> > string message_type = 2; > >>> >>> > } > >>> >>> > > >>> >>> > It is easy, to concatenate the header+actual message bytes and > post > >>> >>> > the > >>> >>> > resulting block to a queue. But how do I take these apart on the > >>> >>> > receiving > >>> >>> > end? Suppose I get a byte-buffer consisting of: > >>> >>> > > >>> >>> > --------------- > >>> >>> > | header | > >>> >>> > --------------- > >>> >>> > | body | > >>> >>> > --------------- > >>> >>> > > >>> >>> > Is it OK to throw this oversized buffer at the Header > >>> >>> > deserialization? > >>> >>> > Will > >>> >>> > the extra bytes hurt anything? > >>> >>> > > >>> >>> > Then, once I extract the Header message, how do I know where the > >>> >>> > body > >>> >>> > begins? I could turn around and ask the Header object "how big > >>> >>> > would > >>> >>> > you be > >>> >>> > if serialized?". Is that reliable? Is there a better way? > >>> >>> > > >>> >>> > Thanks > >>> >>> > john > >>> >>> > > >>> >>> > -- > >>> >>> > You received this message because you are subscribed to the > Google > >>> >>> > Groups > >>> >>> > "Protocol Buffers" group. > >>> >>> > To unsubscribe from this group and stop receiving emails from it, > >>> >>> > send > >>> >>> > an > >>> >>> > email to [email protected]. > >>> >>> > To post to this group, send email to [email protected]. > >>> >>> > Visit this group at https://groups.google.com/group/protobuf. > >>> >>> > For more options, visit https://groups.google.com/d/optout. > >>> > > >>> > -- > >>> > You received this message because you are subscribed to the Google > >>> > Groups > >>> > "Protocol Buffers" group. > >>> > To unsubscribe from this group and stop receiving emails from it, > send > >>> > an > >>> > email to [email protected]. > >>> > To post to this group, send email to [email protected]. > >>> > Visit this group at https://groups.google.com/group/protobuf. > >>> > For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "Protocol Buffers" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [email protected]. > > To post to this group, send email to [email protected]. > > Visit this group at https://groups.google.com/group/protobuf. > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
