Asking the response helped me to find the answer (yes, rubber duck
debugging<https://en.wikipedia.org/wiki/Rubber_duck_debugging>) as it seems
that the code below does what I need:
s3 = s1.join(s2)
.where(new KeySelector1()).equalTo(new KeySelector2())
.window(GlobalWindow.create())
.trigger(CountTrigger.of(1))
.apply(new JoinFunction<a,b,c>);
If that's a common use case (in my view it is), a syntax shortcut could help
developers, e.g. something like:
s3 = s1.join(s2)
.where(new KeySelector1()).equalTo(new KeySelector2())
.combineLatest(new JoinFunction<a,b,c>);
Denis
From: Dollfus, Denis (TR Technology & Ops)
Sent: lundi 5 décembre 2016 12:27
To: user@flink.apache.org
Subject: Equivalent of Rx combineLatest() on a join?
Hi all,
[first email here, I'm new to Flink, Java and Scala, sorry if I missed
something obvious]
I'm exploring Flink in the context of streaming calculators. Basically, the
data flow boils down to multiple data streams with variable update rates (ms,
seconds, ..., month) which are joined before being fed to calculators. The kind
of operation I need is very similar to the Rx
combineLatest<https://urldefense.proofpoint.com/v2/url?u=http-3A__reactivex.io_documentation_operators_combinelatest.html&d=CwMFAg&c=4ZIZThykDLcoWk-GVjSLm9hvvvzvGv0FLoWSRuCSs5Q&r=Jtkfol_Mg_eQUL17wcicdBE1Et94D7zgzS-8_bKvGlc&m=XIk_LOnCw3IK_rNsQ5k4Y7nX14-RXxpT2wgnexTL2nM&s=HqeoR2u4r5RCZ6OjcCRP6usy9b8iUuZU3xxU6Sk17yg&e=>
operator, which results in a object being emitted whenever one of the streams
is updated.
As there is no such operator predefined, I think I have to use a GlobalWindow
and provide a custom WindowAssigner. The end result would look like this
(pseudo java 8 code, I hope it's understandable):
DataStream<price1> s1 = env.addSource(..);
DataStream<price2> s2 = env.addSource(..);
S3 = s1.join(s2)
.where(s1 -> id)
.equalTo(s2 -> id)
.window(new MyCustomCombineLatestAssigner())
.apply( ... return new object combining data from s1 and from
s2);
Is the approach correct, or is there a simpler way to achieve the same join +
apply mechanism ?
Thank you,
Denis
________________________________
This e-mail is for the sole use of the intended recipient and contains
information that may be privileged and/or confidential. If you are not an
intended recipient, please notify the sender by return e-mail and delete this
e-mail and any attachments. Certain required legal entity disclosures can be
accessed on our website.<http://site.thomsonreuters.com/site/disclosures/>
