Here is a small pipeline job that fails using the Dataflow runner, but doesn't fail using the direct runner.
https://gist.github.com/jacobmarble/804c2edb9c80a2863f3e671d6851a55f Jacob On Fri, Nov 17, 2017 at 9:27 AM, Kenneth Knowles <[email protected]> wrote: > It is definitely a big deal if @Setup is not getting called! There are no > special cases that would skip @Setup. Please do report what you can. > > That said, lazily doing setup (via null check or some such as you mention) > is perfectly fine and often a more robust programming pattern. Upside: you > can't accidentally use uninitialized things. Downside: it might mask > repeated initialization and only manifest as poor performance. > > Kenn > > On Fri, Nov 17, 2017 at 9:00 AM, Jacob Marble <[email protected]> wrote: > >> I tried to write a simpler DoFn that induces the error, but it works >> fine. Working around the issue today by using @StartBundle with a null >> check, and that seems to be working. >> >> If this really is a big deal, then it needs to be reported, so I'll try >> to find time to write a broken example. >> >> Jacob >> >> On Thu, Nov 16, 2017 at 10:27 PM, Eugene Kirpichov <[email protected]> >> wrote: >> >>> Could you give more details, e.g. a code snippet that reproduces the >>> issue, and describe how you determine that @Setup hasn't been called? >>> >>> On Thu, Nov 16, 2017 at 6:58 PM Derek Hao Hu <[email protected]> >>> wrote: >>> >>>> I've been using DoFn.Setup method in Dataflow and it seems to be >>>> working fine. >>>> >>>> On Thu, Nov 16, 2017 at 4:56 PM, Jacob Marble <[email protected]> >>>> wrote: >>>> >>>>> This one is weird. >>>>> >>>>> A DoFn I wrote: >>>>> - stateful >>>>> - used plenty in a streaming pipeline >>>>> - direct and dataflow runners >>>>> - works fine >>>>> >>>>> Now: >>>>> - new batch pipeline >>>>> - @DoFn.Setup method not called >>>>> - direct runner works properly (logs from setup method are output) >>>>> - dataflow runner simply doesn't call the setup method >>>>> >>>>> Is this possibly a Beam misuse? Javadoc for DoFn.Setup doesn't hint at >>>>> anything, so I'm suspecting Dataflow bug? >>>>> >>>>> Jacob >>>>> >>>> >>>> >>>> >>>> -- >>>> Derek Hao Hu >>>> >>>> Software Engineer | Snapchat >>>> Snap Inc. >>>> >>> >> >
