I think the *DiscoverSchemaTransform()* RPC introduced in this proposal and the ability to easily deploy/use available *SchemaTransforms* using an expansion service essentially provide the tooling necessary for implementing such a service. Such a service could even startup expansion services to discover/list transforms available in given artifacts (for example, jar files).
Thanks, Cham On Mon, Aug 8, 2022 at 3:48 PM Byron Ellis <byronel...@google.com> wrote: > I like that idea, sort of like Kafka’s Schema Service but for transforms? > > On Mon, Aug 8, 2022 at 2:45 PM Robert Bradshaw via dev < > dev@beam.apache.org> wrote: > >> This is a great idea. I would like to approach this from the >> perspective of making it easy to provide a catalog of well-defined >> transforms for use in expansion services from typical SDKs and also >> elsewhere (e.g. for documentation purposes, GUIs, etc.) Ideally >> everything about what a transform is (its config, documentation, >> expectations on inputs, etc.) can be specified programmatically in a >> way that's much easier to both author and consume than it is now. >> >> On Thu, Aug 4, 2022 at 6:51 PM Chamikara Jayalath via dev >> <dev@beam.apache.org> wrote: >> > >> > Hi All, >> > >> > I believe we can make the multi-language pipelines offering [1] much >> easier to use by updating the expansion service to be fully aware of >> SchemaTransforms. Additionally this will make it easy to >> register/discover/use transforms defined in one SDK from all other SDKs. >> Specifically we could add the following features. >> > >> > Expansion service can be used to easily initialize and expand >> transforms without need for additional code. >> > Expansion service can be used to easily discover already registered >> transforms. >> > Pipeline SDKs can generate user-friendly stub-APIs based on transforms >> registered with an expansion service, eliminating the need to develop >> language-specific wrappers. >> > >> > Please see here for my proposal: >> https://s.apache.org/easy-multi-language >> > >> > Lemme know if you have any comments/questions/suggestions :) >> > >> > Thanks, >> > Cham >> > >> > [1] >> https://beam.apache.org/documentation/programming-guide/#multi-language-pipelines >> > >> >
