You should use a side input and not an empty PCollection that you flatten. Since ReadA --> Flatten --> ParDo ReadB -/ can be equivalently executed as: ReadA --> ParDo ReadB --> ParDo
Make sure you access the side input in case a runner evaluates the side input lazily. So your pipeline would look like: Create --> ParDo(DoAction) --> View.asSingleton() named X ... --> ParDo(ProcessElements).withSideInput(X) --> ... An alternative would be to use CoGroupByKey to join the two streams since it is not possible to split the execution like I showed with Flatten. It is wasteful to add the CoGroupByKey but it is a lot less wasteful if you convert a preceding GroupByKey in your pipeline into a CoGroupByKey joining the two streams. On Wed, Feb 28, 2018 at 8:58 AM, Andrew Jones <[email protected]> wrote: > Hi, > > What is the best way to run code before the pipeline starts? Anything in > the `main` function doesn't get called when the pipeline is ran on Dataflow > via a template - only the pipeline. If you're familiar with Spark, then I'm > thinking of code that might be ran in the driver. > > Alternatively, is there a way I can run part of a pipeline first, then run > another part once it's completed? Not sure that makes sense, so to > illustrate with a poor attempt at an ascii diagram, if I have something > like this: > > events > /\ > / \ > | group by key > | | > | do some action > | / > | / > once action is complete, > process all original elements > > I can presumably achieve this by having `do some action` either generating > an empty side input or an empty PCollection which I can then use to create > a PCollectionList along with the original and pass to > Flatten.pCollections() before continuing. Not sure if that's the best way > to do it though. > > Thanks, > Andrew >
