I like that idea, sort of like Kafka’s Schema Service but for transforms?
On Mon, Aug 8, 2022 at 2:45 PM Robert Bradshaw via dev <dev@beam.apache.org> wrote: > This is a great idea. I would like to approach this from the > perspective of making it easy to provide a catalog of well-defined > transforms for use in expansion services from typical SDKs and also > elsewhere (e.g. for documentation purposes, GUIs, etc.) Ideally > everything about what a transform is (its config, documentation, > expectations on inputs, etc.) can be specified programmatically in a > way that's much easier to both author and consume than it is now. > > On Thu, Aug 4, 2022 at 6:51 PM Chamikara Jayalath via dev > <dev@beam.apache.org> wrote: > > > > Hi All, > > > > I believe we can make the multi-language pipelines offering [1] much > easier to use by updating the expansion service to be fully aware of > SchemaTransforms. Additionally this will make it easy to > register/discover/use transforms defined in one SDK from all other SDKs. > Specifically we could add the following features. > > > > Expansion service can be used to easily initialize and expand transforms > without need for additional code. > > Expansion service can be used to easily discover already registered > transforms. > > Pipeline SDKs can generate user-friendly stub-APIs based on transforms > registered with an expansion service, eliminating the need to develop > language-specific wrappers. > > > > Please see here for my proposal: > https://s.apache.org/easy-multi-language > > > > Lemme know if you have any comments/questions/suggestions :) > > > > Thanks, > > Cham > > > > [1] > https://beam.apache.org/documentation/programming-guide/#multi-language-pipelines > > >
