Thanks. I added some comments to the doc.
On Mon, Feb 6, 2023 at 1:33 PM Chamikara Jayalath via dev <dev@beam.apache.org> wrote: > > Hi All, > > Beam PTransforms are currently primarily identified as operations in a > pipeline that perform specific tasks. PTransform implementations were > traditionally linked to specific Beam SDKs. > > With the advent of portability framework, multi-language pipelines, and > expansion services that can be used to build/expand and discover transforms, > we have an opportunity to make this more general and re-introduce Beam > PTransforms as computation units that can serve any use-case that needs to > discover or use Beam transforms. For example, any Beam SDK that runs a > pipeline using a portable Beam runner should be able to use a transform > offered through an expansion service irrespective of the implementation SDK > of the transform or the pipeline. > > I believe we can make such use-cases much easier to manage by introducing a > user-deployable service that encapsulates existing Beam expansion services in > the form of a Kubernetes cluster. The service will offer a single gRPC > endpoint and will include Beam expansion services developed in different > languages. Any Beam pipeline, irrespective of the pipeline SDK, should be > able to use any transform offered by the service. > > This will also offer a way to make multi-language pipeline execution, which > currently relies on locally downloaded large dependencies and locally started > expansion service processes, more robust. > > I have written a proposal for implementing such a service and it's available > at https://s.apache.org/beam-transform-service. > > Please take a look and let me know if you have any comments or questions. > > Thanks, > Cham
