On Fri, Jun 23, 2023, 7:37 AM Alexey Romanenko <aromanenko....@gmail.com> wrote:
> If Beam Runner Authoring Guide is rather high-level for you, then, at > fist, I’d suggest to answer two questions for yourself: > - Am I going to implement a portable runner or native one? > The answer to this should be portable, as non-portable ones will be deprecated. - Which SDK I should use for this runner? > The answer to the above question makes this one moot :). On a more serious note, could you tell us a bit more about the runner you're looking at implementing? > Then, depending on answers, I’d suggest to take as an example one of the > most similar Beam runners and use it as a more detailed source of > information along with Beam runner doc mentioned before. > > — > Alexey > > On 22 Jun 2023, at 14:39, Joey Tran <joey.t...@schrodinger.com> wrote: > > Hi Beam community! > > I'm interested in trying to implement a runner with my company's execution > environment but I'm struggling to get started. I've read the docs page > <https://beam.apache.org/contribute/runner-guide/#testing-your-runner> on > implementing a runner but it's quite high level. Anyone have any concrete > suggestions on getting started? > > I've started by cloning and running the hello world example > <https://github.com/apache/beam-starter-python>. I've then subclassed ` > PipelineRunner > <https://github.com/apache/beam/blob/9d0fc05d0042c2bb75ded511497e1def8c218c33/sdks/python/apache_beam/runners/runner.py#L103>` > to create my own custom runner but at this point I'm a bit stuck. My custom > runner just looks like > > class CustomRunner(runner.PipelineRunner): > def run_pipeline(self, pipeline, > options): > self.visit_transforms(pipeline, options) > > And when using it I get an error about not having implemented "Impulse" > > NotImplementedError: Execution of [<Impulse(PTransform) label=[Impulse]>] > not implemented in runner <my_app.app.CustomRunner object at 0x135d9ff40>. > > Am I going about this the right way? Are there tests I can run my custom > runner against to validate it beyond just running the hello world example? > I'm finding myself just digging through the beam source to try to piece > together how a runner works and I'm struggling to get a foothold. Any > guidance would be greatly appreciated, especially if anyone has any > experience implementing their own python runner. > > Thanks in advance! Also, could I get a Slack invite? > Cheers, > Joey > > >
