Hi Kant, So your problem statement is "ingest 2 streams into a data warehouse". The main component of the solution, from my view, is that SQL server. You can have a sink function to insert records in your two streams into two different tables (A and B), or upsert into one single table C. That upsert action itself serves as a join function, there's no need to join in Flink at all.
There are many tools out there can be used for that ingestion. Flink, of course, can be used for that purpose. But for me, it's an overkill. Regards, Averell On Thu, 31 Oct. 2019, 8:19 pm kant kodali, <kanth...@gmail.com> wrote: > Hi Averell, > > yes, I want to run ad-hoc SQL queries on the joined data as well as data > that may join in the future. For example, let's say if you take datasets A > and B in streaming mode a row in A can join with a row B in some time in > future let's say but meanwhile if I query the intermediate state using SQL > I want the row in A that have not yet joined with B to also be available to > Query. so not just joined results alone but also data that might be join in > the future as well since its all streaming. > > Thanks! >
