This is not Java specific. I just used Java as an example. For most protobuf implementations, there is a contract between the runtime and generated code which is not meant to be public. e.g. In go-protobuf, runtime expects the generated classes define some internal fields, like XXX_unrecognized, and XXX_sizecache.
If we are moving from table-driven to codegen, then I guess there will be more contracts between the runtime and generated code, and potentially introduce contracts also between the containing message and embedded message fields - you probably want to extract duplicate methods into runtime to save code size, and the generate code needs to deal with submessages itself. This is my primary concern. In a single repo development model, someone can change such contract as long as they change the protoc-gen and the runtime library at the same time. However, in open source, such change can cause dependency-hell <https://en.wikipedia.org/wiki/Dependency_hell>issues as the different versions of generated protobufs are no-longer compatible with each other and/or with the runtime. With a more tightly coupled model, we are more prone to such issue. One way to solve this is to make the runtime support all the previous versions of the contract indefinitely and be always backward compatible. Then users can switch to the newest runtime to support all the old libraries. We are following this approach in protobuf-java 3.x, which causes lots of dev overhead. I'm a bit worried that we would need to do the same for Go, if we go with the code-gen approach. On Wed, Jan 31, 2018 at 10:25 PM Walter Schulze <awalterschu...@gmail.com> wrote: > Could we perhaps get a code snippet example that non java programmers can > follow. > > PS > > When I previously referred to java programmers as real software > developers, I didn't add some much needed context. > > Sometimes in this java dominated world I personally don't feel like a real > software developer. > > On Wed, 31 Jan 2018, 21:44 liujisi via golang-nuts, < > golang-nuts@googlegroups.com> wrote: > >> >> >> On Wednesday, January 31, 2018 at 11:20:31 AM UTC-8, Walter Schulze wrote: >> >>> Hi JT please see my inline replies. >>> >>> On Wed, 31 Jan 2018 at 19:05 <thebroke...@gmail.com> wrote: >>> >>>> Thank you, Walter, for your support. >>>> >>>> > gogo/protobuf is disappointed that golang/protobuf still thinks that >>>> runtime reflection is an efficient way of serializing structures. >>>> >>>> The table-driven implementation avoids reflect in the fast and common >>>> path. Instead, are you referring to the fact that we don't perform >>>> full-code generation of Marshal/Unmarshal like what gogo/protobuf does? We >>>> are aware that full-code generation will often out-perform the table-driven >>>> approach we took. However, full code-generation drastically bloats the >>>> binary size when you have many proto messages linked in. Keeping the binary >>>> size smaller was an important design decision for us and seemed to be a >>>> better default. >>>> >>> >>> Yes, I was referring to the speed of code generation over runtime >>> reflection. >>> What I struggle to understand is why the optimize_for file option that >>> is part of proto 2 and 3 is not considered by golang/protobuf as a way to >>> specify when code generation should be used over runtime reflection. >>> This seems to work for most other languages, including Java, which I >>> heard is quite popular among real software developers. >>> >> >> Note that the code-generation approach used by other languages (mostly >> C++/Java) has its own problem. Mostly because of the tight coupling among >> generated code, runtime and embedded sub messages (generated by a different >> party using a different version of protoc). These problem don't exist >> inside of google as we use a single repo build system, but it cause >> significant issues in opensource. For instance, Hadoop is still shipping >> protobuf v2.5 generated code which is incompatible with later version of >> protobufs. All the projects using Hadoop are then version locked to v2.5, >> as an upgrade in any project (including Hadoop itself) will break the >> build. Version upgrade can only happen when all the transitive dependency >> closure upgrade together atomically. >> >> We solve this problem in protobuf-java v3.0+ by introducing ABI >> backward/forward compatibility guarantees on generated code and runtime. >> However, this introduced lots of overhead on code maintenance, reduced >> development velocity and limited the change we could do. We are now >> solving the issue by introduce table driven to Java. The recent benchmark >> result showed performance on par for android platforms, and hopefully we >> can release the new implementation in a few months. >> >> For Go, if we are going to introduce full generated code, I'd strongly >> recommend considering those complications. Major version bump is also >> expensive for protobufs as all the dependency libraries would have to bump >> their major version too. >> >> >>> >>> >>>> >>>> We are open to considering an option that allows user to specify >>>> full-code generation for select messages. >>>> >>> >>> This is exactly what gogo/protobuf allows users to do. >>> Using protobuf extensions gogo/protobuf allows the user to specify per >>> message or file whether they want to generate marshalers, unmarshalers, etc. >>> A user can also create a vanity binary to generate these methods if you >>> do not wish to use extensions and want to enforce a specific style across >>> and organization. >>> >>> >>>> >>>> > gogo/protobuf is still open to being merged back into golang/protobuf >>>> and has been since its inception 5 years ago. >>>> >>>> That is good to hear. I have not yet gone through all of gogo/protobuf >>>> to determine what it would to merge, or what should be merged. This will be >>>> future work. >>>> >>> >>> gogo/protobuf is also be open to only being partly merged. >>> >>> One other major advantage of gogo/protobuf is generating the structures >>> you want to use, by allowing you to modify the generated structure using >>> protobuf extensions like customtype. >>> This way you can avoid copying between the protobuf generated structure >>> and a user defined go structure that you actually want to use. >>> This is a huge speed and safety gain and probably the most important >>> feature of gogo/protobuf. >>> proto3 has addressed the biggest concern by allowing the generation of >>> fields without pointers, but there are other cases as well, including >>> casttype, customname for generating more lintable code and even not >>> generating the structure at all, for ultimate customization. >>> I would hope that merging some of these ideas will also be on the table. >>> >>> Looking forward to working together for a change >>> Please let me know how I can help >>> >>> Skeptically hopeful about a new era for protobufs in Go >>> Walter Schulze >>> >>> >> >>>> JT >>>> >>>> On Wednesday, January 31, 2018 at 7:13:38 AM UTC-8, Walter Schulze >>>> wrote: >>>>> >>>>> gogo/protobuf is happy to be acknowledged by Google as an entity in >>>>> the golang protobuf space. >>>>> gogo/protobuf welcomes golang/protobuf to the community and is >>>>> extremely happy to see this kind of transparency. >>>>> >>>>> gogo/protobuf will also merge these changes and as usual try to stay >>>>> as close as possible to golang/protobuf, >>>>> including also following the same version tagging. >>>>> >>>>> gogo/protobuf is disappointed that golang/protobuf still thinks that >>>>> runtime reflection is an efficient way of serializing structures. >>>>> >>>>> go Green go GoGoProtobuf >>>>> >>>>> PS >>>>> >>>>> gogo/protobuf is still open to being merged back into golang/protobuf >>>>> and has been since its inception 5 years ago. >>>>> gogo/protobuf feels for its users, especially those that are not >>>>> acknowledged by grpc-gateway and grpc-go, >>>>> and forced to employ work arounds, to preserve their missions of >>>>> safety and efficiency. >>>>> It knows that its existence is not something that anyone prefers, and >>>>> it welcomes death, >>>>> but only if it can preserve its legacy of fast serailization and >>>>> generating the structures you want to use. >>>>> >>>>> >>>>> On Tuesday, 30 January 2018 23:44:37 UTC+1, joe...@google.com wrote: >>>>>> >>>>>> Done. I tagged v1.0.0. When we perform the merge in the future, it >>>>>> will be tagged as v1.1.0. >>>>>> >>>>>> On Tuesday, January 30, 2018 at 9:37:23 AM UTC-8, Alexey Palazhchenko >>>>>> wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Can you please add tags to the repository before that? SemVer or >>>>>>> even tags with _any_ semantic would greatly help to rollback to the >>>>>>> latest >>>>>>> working version when things break. >>>>>>> >>>>>>> –-– >>>>>>> Alexey «AlekSi» Palazhchenko >>>>>>> >>>>>>> -- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "golang-nuts" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/golang-nuts/F5xFHTfwRnY/unsubscribe. >>>> >>> To unsubscribe from this group and all its topics, send an email to >>>> golang-nuts...@googlegroups.com. >>> >>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "golang-nuts" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/golang-nuts/F5xFHTfwRnY/unsubscribe. >> > To unsubscribe from this group and all its topics, send an email to >> golang-nuts+unsubscr...@googlegroups.com. > > >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
