As far as I can tell, JSON schemas work for YAML as well, since the parsed forms are identical.
On Fri, Aug 17, 2018 at 4:47 PM Leif Hedstrom <zw...@apache.org> wrote: > > > > On Aug 17, 2018, at 09:51, Alan Carroll <solidwallofc...@oath.com.INVALID> > wrote: > > > > We would not be inventing our own marshaling again, we would be using > YAML. > > > Is there schema definitions for yaml? I think that should be a minimum > requirement at least. > > — Leif > > > >> On Fri, Aug 17, 2018 at 12:45 PM Leif Hedstrom <zw...@apache.org> > wrote: > >> > >> > >> > >>> On Aug 17, 2018, at 06:46, Alan Carroll <solidwallofc...@oath.com > .INVALID> > >> wrote: > >>> > >>> 1) Yes. Two counter points - right now we're not reducing anything > >> because > >>> no one is willing to actually bring in the third party RPC. This > proposal > >>> would IMHO be definitely better than the current one. Second, bringing > in > >>> complete third party RPC will require glue logic to adapt to our > >> > >> > >> All other things said, this is not accurate. There are definitely > >> suggestions on the table to bring in an existing serialization protocol > >> (and willing people). There are plenty well supported libraries, even > CBOR > >> was one of the suggestions. I’m fine piggy backing such libraries on our > >> own transport (say HTTP), but why invent our own marshaling again? > >> > >> Cheers, > >> > >> — Leif > >>> infrastructure. My opinion is this proposal will be less work and > >>> maintenance than that. This is of course speculative because the prior > >>> point means we don't have a basis for comparison. The best I can say is > >>> that I have spent time looking at what it would take and that lead me > to > >>> this proposal. > >>> > >>> 2) RPC communication by YAML or JSON encoded messages is not some brand > >> new > >>> idea of mine. It's actually a common style, sometimes referred to as a > >>> "REST API"[1]. This proposal would make any implementation language > that > >>> supports YAML easily able to communicate. I suspect YAML support is > more > >>> widespread than gRPC or Avro support. > >>> > >>> [1] https://en.wikipedia.org/wiki/Representational_state_transfer > >>> > >>> On Fri, Aug 17, 2018 at 9:12 AM Aaron Canary <acan...@oath.com.invalid > > > >>> wrote: > >>> > >>>> 1. I thought the goal of replacing RPC was to reduce code maintenance. > >> Does > >>>> writing our own do that? How does that effect the other apps we are > >>>> communicating with? > >>>> > >>>> 2. We are discussing using a 3rd party health monitor app which is > >> likely > >>>> to have frequent communication with ATS. I.e. ATS pushes request when > >>>> traffic is dropped, HealthMon pushes when origin looks healthy again. > >>>> (Simplification) this could be RPC or something else. > >>>> > >>>> On Aug 16, 2018 9:53 PM, "Alan Carroll" <solidwallofc...@oath.com > >> .invalid> > >>>> wrote: > >>>> > >>>> The current RPC code is a mess, with bizarre even for TS compiling and > >>>> linking. It won't be really bidirectional even if traffic_manager is > >>>> removed. It's fragile, it's extremely difficult to extend or modify. > You > >>>> have to basically write custom serialization and deserialization logic > >> for > >>>> every message type. On top of that the serialization is very fragile. > >> As I > >>>> noted above, my view is getting the serialization robust and flexible > is > >>>> almost all the work in writing an RPC. That's the part we should > >> leverage. > >>>> Moving bytes in and out of sockets is what we do well, therefore we > >> should > >>>> do that part. Bringing in something like Avro or grpc is a big step > and > >>>> will have a lot of issues with integration. I strongly believe this > >>>> approach will play to our strengths, give us a nice flexible and > >> powerful > >>>> RPC, with the minimum amount of work and additional footprint, > yielding > >>>> something that fits in well with our existing architecture. > >>>> > >>>> > >>>>> On Thu, Aug 16, 2018 at 9:07 PM Leif Hedstrom <zw...@apache.org> > >> wrote: > >>>>> > >>>>> Then what is the motivation to change anything? Let’s just keep the > >>>>> serialization the same as what we have, Peach cleaned that up > >> noticeably. > >>>>> > >>>>> The issue with bidirectional communication gets solved as part of the > >>>>> elimination of traffic_manager. > >>>>> > >>>>> — Leif > >>>>> > >>>>>> On Aug 16, 2018, at 17:40, Alan Carroll <solidwallofc...@oath.com > >>>> .INVALID> > >>>>> wrote: > >>>>>> > >>>>>> Yeah, it's not like we have any expertise in asynchronous I/O on IPC > >>>>>> sockets, that would let us leverage someone else's YAML based > >>>>> serialization > >>>>>> / deserialization library. We should definitely go with something > that > >>>>> has > >>>>>> its own I/O mechanisms (which are sure to be compatible!) and quite > >>>>>> possibly its own threading model too. That'll be *easy*! And I can't > >>>> wait > >>>>>> to play with all the super cool features and adjustable knobs such a > >>>>> thing > >>>>>> will have. > >>>>>> > >>>>>>> On Thu, Aug 16, 2018 at 6:17 PM Leif Hedstrom <zw...@apache.org> > >>>> wrote: > >>>>>>> > >>>>>>> I think rolling our own defeats the purpose of replacing what we > >> have. > >>>>> We > >>>>>>> should pick something where there is a large sets of client > libraries > >>>>>>> already. As you say, performance is not critical, so pick one that > is > >>>>>>> popular. I think gRPC would be fine too. > >>>>>>> > >>>>>>> — Leif > >>>>>>> > >>>>>>>> On Aug 16, 2018, at 11:59, Derek Dagit <der...@oath.com.INVALID> > >>>>> wrote: > >>>>>>>> > >>>>>>>> I was not at Cork, so I am probably missing a lot of the details. > >>>>>>>> > >>>>>>>> It seems having true RPC here would greatly expand the > possibilities > >>>>> for > >>>>>>>> management and monitoring. > >>>>>>>> > >>>>>>>> > >>>>>>>> - What are the current and planned use cases for this RPC? > >>>>>>>> > >>>>>>>> - What if we want to add authentication/authorization for > >>>> multi-tenant > >>>>>>>> management and isolation? > >>>>>>>> > >>>>>>>> - What about security issues and the ongoing cost of maintaining a > >>>>> unique > >>>>>>>> RPC solution, versus using something already battle-tested? > >>>>>>>> > >>>>>>>> - Didn't we try this once already? [0] > >>>>>>>> > >>>>>>>> > >>>>>>>> (I figure there are good answers to all of these questions, and > so I > >>>> am > >>>>>>>> mostly asking for the benefit of those of us not familiar with the > >>>> past > >>>>>>>> discussions.) > >>>>>>>> > >>>>>>>> [0] > >>>>>>> > >>>>> > >> > https://github.com/apache/trafficserver/pull/3504#issuecomment-389927441 > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> What are the current and planned use cases for this RPC? > >>>>>>>> > >>>>>>>> On Thu, Aug 16, 2018 at 2:39 PM, Alan M. Carroll < > >>>>>>>> a...@network-geographics.com> wrote: > >>>>>>>> > >>>>>>>>> I've been looking at how to do the RPC replacement that was > agreed > >>>> on > >>>>> at > >>>>>>>>> Cork. After looking at grpc, Avro, and Thrift, I don't like any > of > >>>>> them > >>>>>>> for > >>>>>>>>> our use case. I've moved to the camp that says we should build > our > >>>> own > >>>>>>> thin > >>>>>>>>> wrapper around YAML messages, very similar to a REST style API. > We > >>>> do > >>>>>>> not > >>>>>>>>> need high performance for this case, the message rate is > generally > >>>>> less > >>>>>>>>> than one per second and the messages are not large. Previously > >> going > >>>>>>> YAML > >>>>>>>>> would have been a major effort but since we've already paid that > >>>> price > >>>>>>> in > >>>>>>>>> the ongoing configuration conversion, building an RPC on top of > >>>>> YAMLCPP > >>>>>>>>> seems quite easy. In this scheme the data is sent via YAML map > >>>>>>> instances. > >>>>>>>>> In each map, the top level keys are the messages, each key being > a > >>>> tag > >>>>>>> that > >>>>>>>>> identifies the message handler. The handler is given the map > entry, > >>>>> key > >>>>>>> and > >>>>>>>>> value, and processes it. If parallelism is needed, the handlers > can > >>>>>>> attach > >>>>>>>>> an "sequence" field inside the map value to match requests to > >>>>> responses. > >>>>>>>>> > >>>>>>>>> The RPC driver is simple. It has a hash of tags to > >>>>>>> handlers/continuations. > >>>>>>>>> It accumulates data from the socket until it gets a successful > YAML > >>>>>>> parse. > >>>>>>>>> Then it walks the top level keys, dispatching an event for each > one > >>>> to > >>>>>>> the > >>>>>>>>> handler associated with the tag. Sending is done handing a YAML > map > >>>>>>> object > >>>>>>>>> to the RPC, which puts it in a queue to be written by a blocking > >>>> write > >>>>>>>>> thread. My view is this would be less work than integrating a > full > >>>>>>> featured > >>>>>>>>> RPC with many features we don't need. > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> Derek > >>>>>>> > >>>>>>> > >>>>>> > >>>>>> -- > >>>>>> *Beware the fisherman who's casting out his line in to a dried up > >>>>> riverbed.* > >>>>>> *Oh don't try to tell him 'cause he won't believe. Throw some bread > to > >>>>> the > >>>>>> ducks instead.* > >>>>>> *It's easier that way. *- Genesis : Duke : VI 25-28 > >>>>> > >>>>> > >>>> > >>>> -- > >>>> *Beware the fisherman who's casting out his line in to a dried up > >>>> riverbed.* > >>>> *Oh don't try to tell him 'cause he won't believe. Throw some bread to > >> the > >>>> ducks instead.* > >>>> *It's easier that way. *- Genesis : Duke : VI 25-28 > >>>> > >>> > >>> > >>> -- > >>> *Beware the fisherman who's casting out his line in to a dried up > >> riverbed.* > >>> *Oh don't try to tell him 'cause he won't believe. Throw some bread to > >> the > >>> ducks instead.* > >>> *It's easier that way. *- Genesis : Duke : VI 25-28 > >> > >> > > > > -- > > *Beware the fisherman who's casting out his line in to a dried up > riverbed.* > > *Oh don't try to tell him 'cause he won't believe. Throw some bread to > the > > ducks instead.* > > *It's easier that way. *- Genesis : Duke : VI 25-28 > > -- *Beware the fisherman who's casting out his line in to a dried up riverbed.* *Oh don't try to tell him 'cause he won't believe. Throw some bread to the ducks instead.* *It's easier that way. *- Genesis : Duke : VI 25-28
