I've been looking at how to do the RPC replacement that was agreed on at Cork. After looking at grpc, Avro, and Thrift, I don't like any of them for our use case. I've moved to the camp that says we should build our own thin wrapper around YAML messages, very similar to a REST style API. We do not need high performance for this case, the message rate is generally less than one per second and the messages are not large. Previously going YAML would have been a major effort but since we've already paid that price in the ongoing configuration conversion, building an RPC on top of YAMLCPP seems quite easy. In this scheme the data is sent via YAML map instances. In each map, the top level keys are the messages, each key being a tag that identifies the message handler. The handler is given the map entry, key and value, and processes it. If parallelism is needed, the handlers can attach an "sequence" field inside the map value to match requests to responses.
The RPC driver is simple. It has a hash of tags to handlers/continuations. It accumulates data from the socket until it gets a successful YAML parse. Then it walks the top level keys, dispatching an event for each one to the handler associated with the tag. Sending is done handing a YAML map object to the RPC, which puts it in a queue to be written by a blocking write thread. My view is this would be less work than integrating a full featured RPC with many features we don't need.
