Hi Mans,
I did a quick test on my PC where I simply set breakpoints in map1 and
map2 (someStream has parallelism 1, otherStream 5, my CoMapFunction 8).
Elements of someStream end up in different CoMapTasks (2/8, 7/8 etc.).
So I guess the distribution is a round robin partioning. @Aljoscha might
know more about the internals?
Regards,
Timo
Am 12/31/17 um 10:38 PM schrieb M Singh:
Hi:
Referring to documentation
(https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/index.html)
for ConnectedStreams:
"Connects" two data streams retaining their types. Connect allowing
for shared state between the two streams.
|DataStream<Integer> someStream = //... DataStream<String> otherStream
= //... ConnectedStreams<Integer, String> connectedStreams =
someStream.connect(otherStream);|
If the two connected streams have different number of partitions, eg
(someStream has 4 and otherStream has 2), then how do the elements of
the stream get distributed for the CoMapFunction:
|connectedStreams.map(new CoMapFunction<Integer, String, Boolean>() {
@Override public Boolean map1(Integer value) { return true; }
@Override public Boolean map2(String value) { return false; } });|
I believe that that if the second stream is broadcast, then each
partition of the first will get all the elements of the second. Is my
understanding correct ?
If the streams are not broadcast and since the first stream has 4
partitions and second one had 2, then how are the elements of the
second stream distributed to each partition of the first ?
Also, if the streams are not broadcasted but have same number of
partitions, how are the elements distributed ?
Thanks
Mans
