Re: Initialization of broadcast state before processing main stream

2019-11-15 Thread Vasily Melnik
Maxim, great thanks. We'll try buffering. С уважением, Василий Мельник On Thu, 14 Nov 2019 at 19:36, Maxim Parkachov wrote: > Hi Vasily, > > unfortunately, this is known issue with Flink, you could read discussion > under > https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+

Re: Initialization of broadcast state before processing main stream

2019-11-14 Thread Maxim Parkachov
Hi Vasily, unfortunately, this is known issue with Flink, you could read discussion under https://cwiki.apache.org/confluence/display/FLINK/FLIP-17+Side+Inputs+for+DataStream+API . At the moment I have seen 3 solutions for this issue: 1. You buffer fact stream in local state before broadcast is

Re: Initialization of broadcast state before processing main stream

2019-11-13 Thread vino yang
Hi Vasily, Currently, Flink did not do the coordination between a general stream and broadcast stream, they are both streams. Your scene of using the broadcast state is a special one. In a more general scene, the states need to be broadcasted is an unbounded stream, the state events may be broadca

Initialization of broadcast state before processing main stream

2019-11-13 Thread Vasily Melnik
Hi all. In our task we have two Kafka topics: - one with fact stream (web traffic) - one with dimension We would like to put dimension data into broadcast state and lookup on int with facts. But we see that not all dimension records are put into state before first fact record is processed, so loo