I'm hugely in favor of bringing the gogoproto enhancements into golang/protobuf, as a spectator to this discussion. I'm not sure what requirements would put generated binary size over marshalling speed, but maybe I haven't worked on large enough projects yet. We have used gogoproto for about 2 years at work without dependency hell, but again I recognize that I have little experience with large open source projects and dependency chains.
It might be interesting to note that the new kid on the gRPC block, Twitch Twirp, mentioned a large runtime library <https://blog.twitch.tv/twirp-a-sweet-new-rpc-framework-for-go-5f2febbf35f> as one of the reasons they implemented their RPC-over-protobuf implementation. Would love to hear more thoughts about this from the go protobuf team. On Thursday, 1 February 2018 08:02:01 UTC, Walter Schulze wrote: > > I cannot recall that I have experienced these problems. The generated code > has also been far easier to maintain than the runtime reflected code that > is repeatedly patched by golang/protobuf. Except for import paths, this is > probably one of the easiest projects a developer will get to work on. It is > deterministic and you have awesome debugging output. The only thing that > tires me is big patches like this one to the runtime reflected code and > libraries that don't recognize gogoprotobuf and only import the > golang/protobuf library. > > I have seriously been pondering a protobuf generator that does not have a > library as a dependency. It would be incredibly easy to maintain, since I > don't have to merge golang/protobuf patches and get around the fact that > grpc libraries only want a single protobuf library. I have one missing > puzzle piece and then I just need time. > > I don't get the argument for small binaries at all, but not importing a > library will also decrease the binary size and given enough flexibility, > which I intend to have, will also result in not generating the code you > don't need. > > A contract which expect your generated struct to have a marshal and > unmarshal method is very simple to maintain. > > I'll be at fosdem this weekend if anyone wants to chat in person and is > there anyway. > > On Thu, 1 Feb 2018, 08:40 Henrik Johansson, <dahan...@gmail.com > <javascript:>> wrote: > >> Surely these issues already exist in gogo-protobuf? How are they handled >> there? >> >> On Thu, Feb 1, 2018, 08:28 'Jisi Liu' via golang-nuts < >> golan...@googlegroups.com <javascript:>> wrote: >> >>> This is not Java specific. I just used Java as an example. For most >>> protobuf implementations, there is a contract between the runtime and >>> generated code which is not meant to be public. e.g. In go-protobuf, >>> runtime expects the generated classes define some internal fields, like >>> XXX_unrecognized, and XXX_sizecache. >>> >>> If we are moving from table-driven to codegen, then I guess there will >>> be more contracts between the runtime and generated code, and potentially >>> introduce contracts also between the containing message and embedded >>> message fields - you probably want to extract duplicate methods into >>> runtime to save code size, and the generate code needs to deal with >>> submessages itself. This is my primary concern. >>> >>> In a single repo development model, someone can change such contract as >>> long as they change the protoc-gen and the runtime library at the same >>> time. However, in open source, such change can cause dependency-hell >>> <https://en.wikipedia.org/wiki/Dependency_hell>issues as the different >>> versions of generated protobufs are no-longer compatible with each other >>> and/or with the runtime. With a more tightly coupled model, we are more >>> prone to such issue. >>> >>> One way to solve this is to make the runtime support all the previous >>> versions of the contract indefinitely and be always backward compatible. >>> Then users can switch to the newest runtime to support all the old >>> libraries. We are following this approach in protobuf-java 3.x, which >>> causes lots of dev overhead. I'm a bit worried that we would need to do the >>> same for Go, if we go with the code-gen approach. >>> >>> >>> On Wed, Jan 31, 2018 at 10:25 PM Walter Schulze <awalter...@gmail.com >>> <javascript:>> wrote: >>> >>>> Could we perhaps get a code snippet example that non java programmers >>>> can follow. >>>> >>>> PS >>>> >>>> When I previously referred to java programmers as real software >>>> developers, I didn't add some much needed context. >>>> >>>> Sometimes in this java dominated world I personally don't feel like a >>>> real software developer. >>>> >>>> On Wed, 31 Jan 2018, 21:44 liujisi via golang-nuts, < >>>> golan...@googlegroups.com <javascript:>> wrote: >>>> >>>>> >>>>> >>>>> On Wednesday, January 31, 2018 at 11:20:31 AM UTC-8, Walter Schulze >>>>> wrote: >>>>> >>>>>> Hi JT please see my inline replies. >>>>>> >>>>>> On Wed, 31 Jan 2018 at 19:05 <thebroke...@gmail.com> wrote: >>>>>> >>>>>>> Thank you, Walter, for your support. >>>>>>> >>>>>>> > gogo/protobuf is disappointed that golang/protobuf still thinks >>>>>>> that runtime reflection is an efficient way of serializing structures. >>>>>>> >>>>>>> The table-driven implementation avoids reflect in the fast and >>>>>>> common path. Instead, are you referring to the fact that we don't >>>>>>> perform >>>>>>> full-code generation of Marshal/Unmarshal like what gogo/protobuf does? >>>>>>> We >>>>>>> are aware that full-code generation will often out-perform the >>>>>>> table-driven >>>>>>> approach we took. However, full code-generation drastically bloats the >>>>>>> binary size when you have many proto messages linked in. Keeping the >>>>>>> binary >>>>>>> size smaller was an important design decision for us and seemed to be a >>>>>>> better default. >>>>>>> >>>>>> >>>>>> Yes, I was referring to the speed of code generation over runtime >>>>>> reflection. >>>>>> What I struggle to understand is why the optimize_for file option >>>>>> that is part of proto 2 and 3 is not considered by golang/protobuf as a >>>>>> way >>>>>> to specify when code generation should be used over runtime reflection. >>>>>> This seems to work for most other languages, including Java, which I >>>>>> heard is quite popular among real software developers. >>>>>> >>>>> >>>>> Note that the code-generation approach used by other languages (mostly >>>>> C++/Java) has its own problem. Mostly because of the tight coupling among >>>>> generated code, runtime and embedded sub messages (generated by a >>>>> different >>>>> party using a different version of protoc). These problem don't exist >>>>> inside of google as we use a single repo build system, but it cause >>>>> significant issues in opensource. For instance, Hadoop is still shipping >>>>> protobuf v2.5 generated code which is incompatible with later version of >>>>> protobufs. All the projects using Hadoop are then version locked to v2.5, >>>>> as an upgrade in any project (including Hadoop itself) will break the >>>>> build. Version upgrade can only happen when all the transitive dependency >>>>> closure upgrade together atomically. >>>>> >>>>> We solve this problem in protobuf-java v3.0+ by introducing ABI >>>>> backward/forward compatibility guarantees on generated code and runtime. >>>>> However, this introduced lots of overhead on code maintenance, reduced >>>>> development velocity and limited the change we could do. We are now >>>>> solving the issue by introduce table driven to Java. The recent benchmark >>>>> result showed performance on par for android platforms, and hopefully we >>>>> can release the new implementation in a few months. >>>>> >>>>> For Go, if we are going to introduce full generated code, I'd strongly >>>>> recommend considering those complications. Major version bump is also >>>>> expensive for protobufs as all the dependency libraries would have to >>>>> bump >>>>> their major version too. >>>>> >>>>> >>>>>> >>>>>> >>>>>>> >>>>>>> We are open to considering an option that allows user to specify >>>>>>> full-code generation for select messages. >>>>>>> >>>>>> >>>>>> This is exactly what gogo/protobuf allows users to do. >>>>>> Using protobuf extensions gogo/protobuf allows the user to specify >>>>>> per message or file whether they want to generate marshalers, >>>>>> unmarshalers, >>>>>> etc. >>>>>> A user can also create a vanity binary to generate these methods if >>>>>> you do not wish to use extensions and want to enforce a specific style >>>>>> across and organization. >>>>>> >>>>>> >>>>>>> >>>>>>> > gogo/protobuf is still open to being merged back into >>>>>>> golang/protobuf and has been since its inception 5 years ago. >>>>>>> >>>>>>> That is good to hear. I have not yet gone through all of >>>>>>> gogo/protobuf to determine what it would to merge, or what should be >>>>>>> merged. This will be future work. >>>>>>> >>>>>> >>>>>> gogo/protobuf is also be open to only being partly merged. >>>>>> >>>>>> One other major advantage of gogo/protobuf is generating the >>>>>> structures you want to use, by allowing you to modify the generated >>>>>> structure using protobuf extensions like customtype. >>>>>> This way you can avoid copying between the protobuf generated >>>>>> structure and a user defined go structure that you actually want to use. >>>>>> This is a huge speed and safety gain and probably the most important >>>>>> feature of gogo/protobuf. >>>>>> proto3 has addressed the biggest concern by allowing the generation >>>>>> of fields without pointers, but there are other cases as well, including >>>>>> casttype, customname for generating more lintable code and even not >>>>>> generating the structure at all, for ultimate customization. >>>>>> I would hope that merging some of these ideas will also be on the >>>>>> table. >>>>>> >>>>>> Looking forward to working together for a change >>>>>> Please let me know how I can help >>>>>> >>>>>> Skeptically hopeful about a new era for protobufs in Go >>>>>> Walter Schulze >>>>>> >>>>>> >>>>> >>>>>>> JT >>>>>>> >>>>>>> On Wednesday, January 31, 2018 at 7:13:38 AM UTC-8, Walter Schulze >>>>>>> wrote: >>>>>>>> >>>>>>>> gogo/protobuf is happy to be acknowledged by Google as an entity in >>>>>>>> the golang protobuf space. >>>>>>>> gogo/protobuf welcomes golang/protobuf to the community and is >>>>>>>> extremely happy to see this kind of transparency. >>>>>>>> >>>>>>>> gogo/protobuf will also merge these changes and as usual try to >>>>>>>> stay as close as possible to golang/protobuf, >>>>>>>> including also following the same version tagging. >>>>>>>> >>>>>>>> gogo/protobuf is disappointed that golang/protobuf still thinks >>>>>>>> that runtime reflection is an efficient way of serializing structures. >>>>>>>> >>>>>>>> go Green go GoGoProtobuf >>>>>>>> >>>>>>>> PS >>>>>>>> >>>>>>>> gogo/protobuf is still open to being merged back into >>>>>>>> golang/protobuf and has been since its inception 5 years ago. >>>>>>>> gogo/protobuf feels for its users, especially those that are not >>>>>>>> acknowledged by grpc-gateway and grpc-go, >>>>>>>> and forced to employ work arounds, to preserve their missions of >>>>>>>> safety and efficiency. >>>>>>>> It knows that its existence is not something that anyone prefers, >>>>>>>> and it welcomes death, >>>>>>>> but only if it can preserve its legacy of fast serailization and >>>>>>>> generating the structures you want to use. >>>>>>>> >>>>>>>> >>>>>>>> On Tuesday, 30 January 2018 23:44:37 UTC+1, joe...@google.com >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Done. I tagged v1.0.0. When we perform the merge in the future, it >>>>>>>>> will be tagged as v1.1.0. >>>>>>>>> >>>>>>>>> On Tuesday, January 30, 2018 at 9:37:23 AM UTC-8, Alexey >>>>>>>>> Palazhchenko wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> Can you please add tags to the repository before that? SemVer or >>>>>>>>>> even tags with _any_ semantic would greatly help to rollback to the >>>>>>>>>> latest >>>>>>>>>> working version when things break. >>>>>>>>>> >>>>>>>>>> –-– >>>>>>>>>> Alexey «AlekSi» Palazhchenko >>>>>>>>>> >>>>>>>>>> -- >>>>>>> You received this message because you are subscribed to a topic in >>>>>>> the Google Groups "golang-nuts" group. >>>>>>> To unsubscribe from this topic, visit >>>>>>> https://groups.google.com/d/topic/golang-nuts/F5xFHTfwRnY/unsubscribe >>>>>>> . >>>>>>> >>>>>> To unsubscribe from this group and all its topics, send an email to >>>>>>> golang-nuts...@googlegroups.com. >>>>>> >>>>>> >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "golang-nuts" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/golang-nuts/F5xFHTfwRnY/unsubscribe. >>>>> >>>> To unsubscribe from this group and all its topics, send an email to >>>>> golang-nuts...@googlegroups.com <javascript:>. >>>> >>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>> >> You received this message because you are subscribed to the Google Groups >>> "golang-nuts" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to golang-nuts...@googlegroups.com <javascript:>. >> >> >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
