On Fri, Aug 5, 2022 at 9:44 AM Brian Hulette <bhule...@google.com> wrote:
> Thanks Cham! I really like the proposal, I left a few comments. I also had > one higher-level point I wanted to elevate here: > > > Pipeline SDKs can generate user-friendly stub-APIs based on transforms > registered with an expansion service, eliminating the need to develop > language-specific wrappers. > This would be great! I think one point to consider is whether we can do > this statically. We could package up these stubs with releases and include > them in API docs for each language, making them much more discoverable. > That could be an extension on top of your proposal (e.g. as part of its > build, each SDK spins up other known expansion services and generates code > based on the discovery responses), but maybe it could be cleaner if we > don't really need the dynamic version? > So my proposal suggested two solutions for wrappers. * A higher level (dynamic) API (SchemaAwareExternalTransform) that can be used to discover/initialize/use any SchemaTransform. * Developing tooling to generate stubs for each language. This is possible since SchemaTransform gives a cleaner way to define/interpret the construction API of a transform. I think both can be useful. For example, the prior might be useful to quickly test/try out new SchemaTransforms without going through code generation. Also, I agree with you that it might be good to generate such stubs (and corresponding docs) during release time instead of generating and committing stubs to the repo. Thanks, Cham > > Brian > > > On Thu, Aug 4, 2022 at 6:51 PM Chamikara Jayalath via dev < > dev@beam.apache.org> wrote: > >> Hi All, >> >> I believe we can make the multi-language pipelines offering [1] much >> easier to use by updating the expansion service to be fully aware of >> SchemaTransforms. Additionally this will make it easy to >> register/discover/use transforms defined in one SDK from all other SDKs. >> Specifically we could add the following features. >> >> - Expansion service can be used to easily initialize and expand >> transforms without need for additional code. >> - Expansion service can be used to easily discover already registered >> transforms. >> - Pipeline SDKs can generate user-friendly stub-APIs based on >> transforms registered with an expansion service, eliminating the need to >> develop language-specific wrappers. >> >> Please see here for my proposal: https://s.apache.org/easy-multi-language >> >> Lemme know if you have any comments/questions/suggestions :) >> >> Thanks, >> Cham >> >> [1] >> https://beam.apache.org/documentation/programming-guide/#multi-language-pipelines >> >>
