Re: Join with slow changing dimensions/ streams

2019-09-10 Thread Fabian Hueske
t; *From:* Fabian Hueske > *Sent:* Thursday, September 5, 2019 12:59 PM > *To:* Hanan Yehudai > *Cc:* flink-u...@apache.org > *Subject:* Re: Join with slow changing dimensions/ streams > > > > Hi, > > > > Flink does not have good support for mixing bounded and unbo

RE: Join with slow changing dimensions/ streams

2019-09-05 Thread Hanan Yehudai
Thanks Fabian. is there any advantage using broadcast state VS using just CoMap function on 2 connected streams ? From: Fabian Hueske Sent: Thursday, September 5, 2019 12:59 PM To: Hanan Yehudai Cc: flink-u...@apache.org Subject: Re: Join with slow changing dimensions/ streams Hi, Flink

Re: Join with slow changing dimensions/ streams

2019-09-05 Thread Fabian Hueske
Hi, Flink does not have good support for mixing bounded and unbounded streams in its DataStream API yet. If the dimension table is static (and small enough), I'd use a RichMapFunction and load the table in the open() method into the heap. In this case, you'd probably need to restart the job (can b

Join with slow changing dimensions/ streams

2019-09-02 Thread Hanan Yehudai
I have a very common use case -enriching the stream with some dimension tables. e.g the events stream has a SERVER_ID , and another files have the LOCATION associated with e SERVER_ID. ( a dimension table csv file) in SQL I would simply join. but hen using Flink stream API , as far