I also notice that stateful DoFn's seem to only be instantiated once in Dataflow, but multiple instances do end up being created in the direct runner. Is there a story behind that?
Jacob On Fri, Nov 17, 2017 at 7:22 PM, Jacob Marble <[email protected]> wrote: > Noticing some related and unexpected differences between batch and > streaming pipelines. > > Why does a stateful DoFn behave like GroupByKey (no data output until all > data input is complete) in a batch pipeline, but not in a streaming > pipeline? It looks like BatchStatefulParDoOverrides has something to do > with it, but I can't figure out why, or how to work around it. > > In this current project: > 1) read 500 million elements > 2) very slow stateful DoFn (rate limited API calls) > 3) write results > > To complete step 1 in a reasonable time, multiple workers are required, > but Dataflow's autoscaling doesn't reduce the worker quantity when step 1 > completes. Since step 2 doesn't speed up with more workers, it would be > best if it could start as soon as step 1 starts. This way, the job > completes faster and uses fewer resources. > > Jacob > > On Fri, Nov 17, 2017 at 12:23 PM, Jacob Marble <[email protected]> > wrote: > >> Here is a small pipeline job that fails using the Dataflow runner, but >> doesn't fail using the direct runner. >> >> https://gist.github.com/jacobmarble/804c2edb9c80a2863f3e671d6851a55f >> >> Jacob >> >> On Fri, Nov 17, 2017 at 9:27 AM, Kenneth Knowles <[email protected]> wrote: >> >>> It is definitely a big deal if @Setup is not getting called! There are >>> no special cases that would skip @Setup. Please do report what you can. >>> >>> That said, lazily doing setup (via null check or some such as you >>> mention) is perfectly fine and often a more robust programming pattern. >>> Upside: you can't accidentally use uninitialized things. Downside: it might >>> mask repeated initialization and only manifest as poor performance. >>> >>> Kenn >>> >>> On Fri, Nov 17, 2017 at 9:00 AM, Jacob Marble <[email protected]> >>> wrote: >>> >>>> I tried to write a simpler DoFn that induces the error, but it works >>>> fine. Working around the issue today by using @StartBundle with a null >>>> check, and that seems to be working. >>>> >>>> If this really is a big deal, then it needs to be reported, so I'll try >>>> to find time to write a broken example. >>>> >>>> Jacob >>>> >>>> On Thu, Nov 16, 2017 at 10:27 PM, Eugene Kirpichov < >>>> [email protected]> wrote: >>>> >>>>> Could you give more details, e.g. a code snippet that reproduces the >>>>> issue, and describe how you determine that @Setup hasn't been called? >>>>> >>>>> On Thu, Nov 16, 2017 at 6:58 PM Derek Hao Hu <[email protected]> >>>>> wrote: >>>>> >>>>>> I've been using DoFn.Setup method in Dataflow and it seems to be >>>>>> working fine. >>>>>> >>>>>> On Thu, Nov 16, 2017 at 4:56 PM, Jacob Marble <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> This one is weird. >>>>>>> >>>>>>> A DoFn I wrote: >>>>>>> - stateful >>>>>>> - used plenty in a streaming pipeline >>>>>>> - direct and dataflow runners >>>>>>> - works fine >>>>>>> >>>>>>> Now: >>>>>>> - new batch pipeline >>>>>>> - @DoFn.Setup method not called >>>>>>> - direct runner works properly (logs from setup method are output) >>>>>>> - dataflow runner simply doesn't call the setup method >>>>>>> >>>>>>> Is this possibly a Beam misuse? Javadoc for DoFn.Setup doesn't hint >>>>>>> at anything, so I'm suspecting Dataflow bug? >>>>>>> >>>>>>> Jacob >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Derek Hao Hu >>>>>> >>>>>> Software Engineer | Snapchat >>>>>> Snap Inc. >>>>>> >>>>> >>>> >>> >> >
