Hi Yaroslav,
I think your approach is correct. Union is perfect to implement multiway
joins if you normalize the type of all streams before. It can simply be
a composite type with the key and a member variable for each stream
where only one of those variables is not null. A keyed process function
can then perform the actual joining.
Regards,
Timo
On 25.02.21 23:29, Yaroslav Tkachenko wrote:
Hello everyone,
I have a question about implementing a join of N datastreams (where N >
2) without any time guarantees. According to my requirements, late data
is not tolerable, so if I have a join between stream A and stream B and
a message with key X arrives in stream B one year after arriving in
stream A, it still should be able to match and emit the result.
I'm currently working on a pipeline with 9 sources and 2 intermediate
joins, so the DAG looks like this:
stream 1 ----->
stream 2 ----->
stream 3 -----> join1 ----->
stream 4 ----->
stream 5 ----->
stream 6 -----> join2 ----->
stream 7 ------------------>
stream 8 ------------------>
stream 9 ------------------> final join
I'm approaching this by creating state variables for all inputs and
using them for lookups (I'm fine with constantly growing keyspace). In
terms of actually joining data streams I see 3 options:
- coGroup with a GlobalWindow and triggers
- connect
- union
coGroup and connect only support two inputs, so implementing the DAG I
described above is very cumbersome. Also, coGroup needs windowing
semantics (even if it's just a single GlobalWindow), and in my
experience with other frameworks, this could add some overhead.
union only works on inputs of the same type, but this can be solved by
introducing a wrapper class.
I'm able to get the union approach working, but I'm still not sure if
it's the best way to implement this pipeline. Any suggestions?
Thank you!
