On Wed, Mar 30, 2016 at 8:06 AM, 'Feng Xiao' via Protocol Buffers < [email protected]> wrote:
> > > On Mon, Mar 28, 2016 at 10:53 PM, Yoav H <[email protected]> wrote: > >> They say on their website: "When evaluating new features, we look for >> additions that are very widely useful or very simple". >> What I'm suggesting here is both very useful (speeding up serialization >> and eliminating memory duplication) and very simple (simple additions to >> the encoding, no need to change the language). >> So far, no response from the Google guys... >> > Actually there are already a "start embedding" tag and a "end embedding" > tag in protobuf: > https://developers.google.com/protocol-buffers/docs/encoding#structure > > 3 Start group groups (deprecated) > 4 End group groups (deprecated) > > They are deprecated though. > > You mentioned it will be a performance gain, but what we experienced in > google says otherwise. For example, in a lot places we are only interested > in a few fields and want to skip through all other fields (if we are > building a proxy, or the field is simply an unknown field). The start > group/end group tag pair forces the parser to decode every single field in > the a whole group even the whole group is to be ignored after parsing, and > that's a very significant drawback. > This is definitely the use-case where delimiting makes perfect sense (proxy/middleware service that reads part of a message). The name 'protocol buffers' does kinda makes that use-case obvious. If using protobuf to simply serialize/deserialize, then start/end group would definitely benefit the streaming use-case. Shameless plug: https://github.com/protostuff/protostuff optimizes for the latter use-case and was mostly the reason it was created (java only though) > > And adding a new wire tag type to protobuf is not a simple thing. Actually > I don't think we have added any new wire type to protobuf before. There are > a lot issues to consider. For example, isn't all code that switch on > protobuf wire types now suddenly broken? if a new serializer uses the new > wire type in its output, what will happen if the parsers can't understand > it? > > Proto3 is already finalized and we will not add new wire types in proto3. > Whether to add it in proto4 depends on whether we have a good use for it > and whether we can mitigate the risks of rolling out a new wire type. > > >> >> >> On Monday, March 28, 2016 at 10:24:17 AM UTC-7, Peter Hultqvist wrote: >>> >>> This exact suggestion has been up for discussion long time ago(years?, >>> before proto2?) >>> >>> When it comes to taking suggestions I'm only a 3rd party implementer but >>> my understanding is that the design process of protocol buffers and its >>> goals are internal to Google and they usually publish new versions of their >>> code implementing new features before you can read about them in the >>> documents. >>> On Mar 27, 2016 5:31 AM, "Yoav H" <[email protected]> wrote: >>> >>>> Any comment on this? >>>> Will you consider this for proto3? >>>> >>>> On Wednesday, March 23, 2016 at 11:50:36 AM UTC-7, Yoav H wrote: >>>>> >>>>> Hi, >>>>> >>>>> I have a suggestion fr improving the protobuf encoding. >>>>> Is proto3 final? >>>>> >>>>> I like the simplicity of the encoding of protobuf. >>>>> But I think it has one issue with serialization, using streams. >>>>> The problem is with length delimited fields and the fact that they >>>>> require knowing the length ahead of time. >>>>> If we have a very long string, we need to encode the entire string >>>>> before we know its length, so we basically duplicate the data in memory. >>>>> Same is true for embedded messages, where we need to encode the entire >>>>> embedded message before we can append it to the stream. >>>>> >>>>> I think there is a simple solution for both issues. >>>>> >>>>> For strings and byte arrays, a simple solution is to use "chunked >>>>> encoding". >>>>> Which means that the byte array is split into chunks and every chunk >>>>> starts with the chunk length. End of array is indicated by length zero. >>>>> >>>>> For embedded messages, the solution is to have an "start embedding" >>>>> tag and an "end embedding tag". >>>>> Everything in between is the embedded message. >>>>> >>>>> By adding these two new features, serialization can be fully >>>>> streamable and there is no need to pre-serialize big chunks in memory >>>>> before writing them to the stream. >>>>> >>>>> Hope you'll find this suggestion useful and incorporate it into the >>>>> protocol. >>>>> >>>>> Thanks, >>>>> Yoav. >>>>> >>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Protocol Buffers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/protobuf. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/protobuf. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "Protocol Buffers" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/protobuf. > For more options, visit https://groups.google.com/d/optout. > -- When the cat is away, the mouse is alone. - David Yu -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
