Re: Enrich streaming data with small lookup data that slowly changes over time

2020-04-20 Thread Mu Kong
Hi Jark Wu, Thanks for your help! I gave the document a quick glimpse, it seems method [1] fits my purpose better. Let me give it a deeper look. Thank you very much!! Best, Mu On Tue, Apr 21, 2020 at 12:06 PM Jark Wu wrote: > Hi Mu, > > Flink SQL does support dimension table join. There are

Re: Enrich streaming data with small lookup data that slowly changes over time

2020-04-20 Thread Jark Wu
Hi Mu, Flink SQL does support dimension table join. There are two ways to join the dimension table. If the data is in your database (e.g. MySQL, HBase), you can use this way [1] to join the data in your database in realtime and enrich fresh data. If the data is in a log stream (change stream), you

Enrich streaming data with small lookup data that slowly changes over time

2020-04-20 Thread Mu Kong
Hi community, I have a stream of traffic data with a service_id in it. I'm enriching this data with a map of (service_id, service_name), which only has 10 ~ 20 pairs and is read from config file. The problem I'm facing now is, this map changes from time to time, and I don't want to redeploy the a