On Wed, Sep 23, 2020 at 11:52 PM Ryan Patterson <cgamesp...@cgamesplay.com>
wrote:
> Hi Kenton, thanks for weighing in!
>
> > I don't think the generated code would implement the getSchema() RPC
> directly for each type. Instead I think I'd add a new method to the
> `Capability::Server` and `ClientHook` like:
>
> I think we are thinking the same thing here, but to confirm: The
> capability would respond to a standardized method, but the implementation
> of this standard method is provided by the library and not by the generated
> code. Are we in agreement?
>
Right. But I thought you were suggesting the implementation lives in
generated code, whereas I'm suggesting it mostly lives in the RPC system.
> > I'm a bit worried about redundantly transmitting the same schema nodes
> over the wire many times, especially when dealing with many different
> interfaces that implement the same underlying types.
>
> Presumably, a peer would call it on the Bootstrap/Accepted capability,
> which would necessarily pull in every reachable Schema.Node, except for
> through AnyPointer boundaries. So, while the first call might be redundant
> if I did already have the schema, it wouldn't be "many times" in general.
> The worst case is a highly-dynamic interface where every method returns an
> untyped Capability. In such a case, 2 methods (my proposal in the last
> email) makes more sense than a combined method.
>
Technically any capability could implement arbitrary interfaces -- it could
implement a derived class of what it is statically declared to implement,
and that derived class could inherit any other interface as well.
I suppose you could have the calling code explicitly say when it wishes to
query the capability for additional interfaces, rather than assuming the
statically-derived type.
> And via promise pipelining, the 2 method solution shouldn't materially be
> slower than the combined method in the usual case.
>
Not sure if promise pipelining helps here. You'd need to
call supportedInterfaces() first *and wait for it to complete* in order to
find out which type IDs are not in the set you already know about, and only
then could you call schemaForTypes(). So it's still two network round trips.
To do it in one round trip the client would have to tell the server about
types it already knows about. I suppose if the caller already has an
expectation of the capability's type via static typing, it could send that,
so that it only gets a non-empty response if the capability turns out to
implement more than that.
> > There's also the problem that two different servers could have
> conflicting versions of a schema, but could both end up proxying through
> the same third server, meaning you might get different versions of the same
> schema for different capabilities on the same connection. Maybe it's not
> actually feasible to share schemas between capabilities at all?
>
> Let's split this into 2 problems: known (:MyInterface) and unknown
> (:Capability) capabilities. In the first case, a message broker is proxying
> MyInterface between peers which have different underlying versions of the
> same schema. This means that the message broker has a particular schema
> which it supports, and the introspecting peer should treat everything
> coming from the broker as though it were that schema.
>
Cap'n Proto supports passing through messages without knowing their full
schema. This is very important for message brokers/proxies in particular --
you don't want to have to recompile the broker every time you add a new
field to the message. Only the initial producer and final consumer need to
know about a field.
> In the second case, the message broker doesn't have a schema for those
> types, so the introspecting code automatically knows it needs to request
> the schema for the received capability. Because of the case where these
> unknown capabilities might be from different schemas (or different versions
> of one), the introspecting peer shouldn't share schemas between these
> capabilities. The net cost of this is transferring a schema object
> whenever introspecting an untyped Capability (although that transfer may
> not actually be redundant--we won't know ahead of time).
>
But if you assume the types are disjoint, does that mean that if you
discover the new capability implements interface Foo, you can't actually
use it in a call to the original capability that expects a Foo?
Well, I suppose we could allow it to be used as long as the schemas have
the same type ID, independent of whether the schemas are actually
compatible.
So sure, in that case I buy that when people explicitly query a
capability's dynamic type, we could build a whole new schema graph separate
from the previously-known static type. In most cases it won't be necessary
to query the dynamic type anyway since the static type will have everything
you need.
-Kenton
>
> In conclusion, it feels like attempting to cache/reuse schemas makes the
> project big and possibly intractable, but an introspection API which
> doesn't support that has none of the problems and a marginally increased
> wire cost.
>
>
>
> On Saturday, September 19, 2020 at 1:15:11 AM UTC+8 ken...@cloudflare.com
> wrote:
>
>> This is something I've wanted to do for a while (built into the `capnp`
>> tool).
>>
>> I agree that RPC types should be introspectable. I'd design the interface
>> like this:
>>
>> getSchema @0 () -> (typeId :UInt64, nodes :List(Schema.Node))
>>
>> `nodes` would be a list of schema nodes describing the transitive closure
>> of dependencies. (Each `Node` describes one type, like an interface type, a
>> struct type, etc. See schema.capnp for details.)
>>
>> CodeGeneratorRequest isn't quite the right thing to use here, because:
>> - It's specifically designed to provide information needed to generate
>> source code, which is a superset of what you need at runtime. Schema.Node
>> is intended to contain specifically the info that's useful at runtime.
>> - The CodeGeneratorRequest may include a lot of types that aren't
>> actually used by the specific interface.
>> - We don't actually embed the full CodeGeneratorReequest inside the
>> generated code. We embed each Node separately. I wouldn't want to store a
>> second copy of all that data, and trying to rebuild the CGR from the
>> components seems a little weird.
>>
>> Next thing: I don't think the generated code would implement the
>> getSchema() RPC directly for each type. Instead I think I'd add a new
>> method to the `Capability::Server` and `ClientHook` like:
>>
>> virtual kj::Promise<capnp::InterfaceSchema> getSchema() = 0;
>>
>> The generated code for each Server type would implement this virtual
>> method, which would be a one-liner (`return
>> capnp::Schema::from<ThisType>();`).
>>
>> It would be the RPC system's responsibility to actually turn these into
>> RPC calls, and to maintain a capnp::SchemaLoader used to translate
>>
>> I think the RPC system would have to be responsible for maintaining a
>> capnp::SchemaLoader containing all schemas that had been loaded from the
>> remote server, in order to implement the local version of `getSchema()`.
>>
>> I'm a bit worried about redundantly transmitting the same schema nodes
>> over the wire many times, especially when dealing with many different
>> interfaces that implement the same underlying types. I wonder if we need a
>> more complicated mechanism to deal with this. Maybe the RPC interface would
>> let the client specify which schemas they already know about. Or maybe this
>> would be tracked within the RPC system itself on a per-connection basis?
>> There's also the problem that two different servers could have conflicting
>> versions of a schema, but could both end up proxying through the same third
>> server, meaning you might get different versions of the same schema for
>> different capabilities on the same connection. Maybe it's not actually
>> feasible to share schemas between capabilities at all? Ugh.
>>
>> So... yeah, I really want to see this happen, but there's kind of a lot
>> of design questions to work through. Big project.
>>
>> -Kenton
>>
>> On Tue, Sep 15, 2020 at 6:52 AM Ryan Patterson <cgame...@cgamesplay.com>
>> wrote:
>>
>>> I'm in the process of writing an interactive debugger for capnp. It's
>>> easy enough to manually provide a schema when connecting to another peer,
>>> but I think it would be good long-term to support dynamic introspection as
>>> a feature of the RPC system.
>>>
>>> I'm quite new to capnp in general, but after reading over the
>>> documentation, I think the best way to do this is for capnp to dictate a
>>> standardized optional interface which returns the original
>>> CodeGeneratorRequest that was used to implement the peer. Ideally, the code
>>> generator could automatically implement this interface for every generated
>>> interface, or perhaps the capnp library could automatically implement for
>>> on Bootstrap and Accept capabilities. When interacting with services which
>>> don't support introspection (possibly because of space, security, or other
>>> concerns), the method call simply fails because it refers to an invalid
>>> interface ID. The important thing is that such an interface has to have a
>>> fixed ID to be useful, which is why I think it should be established as a
>>> core part of capnp.
>>>
>>> Here is my straw man version of this interface. I've selected AnyPointer
>>> so that implementing this interface doesn't require pulling the entire
>>> capnp schema into your own schema. Since the capnp schema is known to both
>>> parties (even if it's a different version), this is fine.
>>>
>>> interface Introspectable {
>>> codeGeneratorRequest @0 () -> (cgr :AnyPointer);
>>> }
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Cap'n Proto" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to capnproto+...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/capnproto/131b69c1-8782-4935-97d0-6fd11fa31e06n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/capnproto/131b69c1-8782-4935-97d0-6fd11fa31e06n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Cap'n Proto" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to capnproto+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/capnproto/9be29b6b-11db-4aa5-9124-b4fd12008f59n%40googlegroups.com
> <https://groups.google.com/d/msgid/capnproto/9be29b6b-11db-4aa5-9124-b4fd12008f59n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to capnproto+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/capnproto/CAJouXQmTeap4VQHa6cjg4TFQ677ijeC6xZA8xaiqVe3YjiLcgQ%40mail.gmail.com.
