Thanks, Vino & Hequn.
On Mon, Jul 16, 2018 at 5:47 PM Hequn Cheng wrote:
> Hi Shivam,
>
> I think the non-window stream-stream join can solve your problem.
> The non-window join will store all data from both inputs and output joined
> results. The semantics of non-window join is exactly the same
Hi Shivam,
I think the non-window stream-stream join can solve your problem.
The non-window join will store all data from both inputs and output joined
results. The semantics of non-window join is exactly the same with batch
join.
One important thing to note is that the state of join might grow in
Hi Shivam,
Thanks for providing more details about your use case.
So I know you mean two DataStream non-window join. There are two ways to
implement this :
1、user Flink's table/sql non-window join for Streaming : this way the
messages stored in state by Flink, you may not care the state but you
Hi Vino,
First I want to tell you that we are working on Flink SQL so there is no
chance to use Data Stream API.
I will give one example of my use case here:-
Let's say we have two Kafka Topics:
1. UserName to UserId Mapping => {"userName": "shivam", "userId": 123}
2. User transactions in
Hi Shivam,
Can you provide more details about your use case? The join for batch or
streaming? which join type (window or non-window or stream-dimension table
join)?
If it is stream-dimension table join and the table is huge, use Redis or
some cache based on memory, can help to process your proble
Hi Shivam,
Currently, fink sql/table-api support window join and non-window join[1].
If your requirements are not being met by sql/table-api, you can also use
the datastream to implement your own logic. You can refer to the non-window
join implement as an example[2][3].
Best, Hequn
[1]
https://c
Hi,
We have one use case in which we need to persist Table in Flink which can
be later used to join with other tables. This table can be huge so we need
to store it in off-heap but faster access. Any suggestions regarding this?
--
Shivam Sharma
Data Engineer @ Goibibo
Indian Institute Of Informa
