Thanks Oytun. The problem with doing that is the same data will be have to be stored multiple times wasting memory. In my case there will around million entries which needs to be used by at least two operators for now.
Thanks On Mon, Sep 30, 2019 at 5:42 PM Oytun Tez <oy...@motaword.com> wrote: > This is how we currently use broadcast state. Our states are re-usable > (code-wise), every operator that wants to consume basically keeps the same > descriptor state locally by processBroadcastElement'ing into a local state. > > I am open to suggestions. I see this as a hard drawback of dataflow > programming or Flink framework? > > > > --- > Oytun Tez > > *M O T A W O R D* > The World's Fastest Human Translation Platform. > oy...@motaword.com — www.motaword.com > > > On Mon, Sep 30, 2019 at 8:40 PM Oytun Tez <oy...@motaword.com> wrote: > >> You can re-use the broadcasted state (along with its descriptor) that >> comes into your KeyedBroadcastProcessFunction, in another operator >> downstream. that's basically duplicating the broadcasted state whichever >> operator you want to use, every time. >> >> >> >> --- >> Oytun Tez >> >> *M O T A W O R D* >> The World's Fastest Human Translation Platform. >> oy...@motaword.com — www.motaword.com >> >> >> On Mon, Sep 30, 2019 at 8:29 PM Navneeth Krishnan < >> reachnavnee...@gmail.com> wrote: >> >>> Hi All, >>> >>> Is it possible to access a broadcast state across the pipeline? For >>> example, say I have a KeyedBroadcastProcessFunction which adds the incoming >>> data to state and I have downstream operator where I need the same state as >>> well, would I be able to just read the broadcast state with a readonly >>> view. I know this is possible in kafka streams. >>> >>> Thanks >>> >>
