Hey Joey, The best resource to look at, at the moment, is likely Robert Burke's Prism runner that he is implementing ( https://github.com/apache/beam/tree/master/sdks/go/pkg/beam/runners/prism). Runners are pretty complicated and there are a number of primitives that you need to have implemented on the runner side to get executing pipelines.
The Slack has a link on the Beam Contact Us page <https://beam.apache.org/community/contact-us/>, and I'd highly recommend routing questions towards the developer mailing (d...@beam.apache.org) list rather than the user one for runner implementation things. Thanks, Jack McCluskey On Thu, Jun 22, 2023 at 8:40 AM Joey Tran <joey.t...@schrodinger.com> wrote: > Hi Beam community! > > I'm interested in trying to implement a runner with my company's execution > environment but I'm struggling to get started. I've read the docs page > <https://beam.apache.org/contribute/runner-guide/#testing-your-runner> on > implementing a runner but it's quite high level. Anyone have any concrete > suggestions on getting started? > > I've started by cloning and running the hello world example > <https://github.com/apache/beam-starter-python>. I've then subclassed ` > PipelineRunner > <https://github.com/apache/beam/blob/9d0fc05d0042c2bb75ded511497e1def8c218c33/sdks/python/apache_beam/runners/runner.py#L103>` > to create my own custom runner but at this point I'm a bit stuck. My custom > runner just looks like > > class CustomRunner(runner.PipelineRunner): > def run_pipeline(self, pipeline, > options): > self.visit_transforms(pipeline, options) > > And when using it I get an error about not having implemented "Impulse" > > NotImplementedError: Execution of [<Impulse(PTransform) label=[Impulse]>] > not implemented in runner <my_app.app.CustomRunner object at 0x135d9ff40>. > > Am I going about this the right way? Are there tests I can run my custom > runner against to validate it beyond just running the hello world example? > I'm finding myself just digging through the beam source to try to piece > together how a runner works and I'm struggling to get a foothold. Any > guidance would be greatly appreciated, especially if anyone has any > experience implementing their own python runner. > > Thanks in advance! Also, could I get a Slack invite? > Cheers, > Joey >
