This is a great idea. I would like to approach this from the perspective of making it easy to provide a catalog of well-defined transforms for use in expansion services from typical SDKs and also elsewhere (e.g. for documentation purposes, GUIs, etc.) Ideally everything about what a transform is (its config, documentation, expectations on inputs, etc.) can be specified programmatically in a way that's much easier to both author and consume than it is now.
On Thu, Aug 4, 2022 at 6:51 PM Chamikara Jayalath via dev <dev@beam.apache.org> wrote: > > Hi All, > > I believe we can make the multi-language pipelines offering [1] much easier > to use by updating the expansion service to be fully aware of > SchemaTransforms. Additionally this will make it easy to > register/discover/use transforms defined in one SDK from all other SDKs. > Specifically we could add the following features. > > Expansion service can be used to easily initialize and expand transforms > without need for additional code. > Expansion service can be used to easily discover already registered > transforms. > Pipeline SDKs can generate user-friendly stub-APIs based on transforms > registered with an expansion service, eliminating the need to develop > language-specific wrappers. > > Please see here for my proposal: https://s.apache.org/easy-multi-language > > Lemme know if you have any comments/questions/suggestions :) > > Thanks, > Cham > > [1] > https://beam.apache.org/documentation/programming-guide/#multi-language-pipelines >
