Re: Apache Flink - Connected Stream with different number of partitions

2018-01-03 Thread M Singh
Thanks Aljoscha and Timo for your answers.  I will try to digest the pointers you provided. Mans On Wednesday, January 3, 2018 3:01 AM, Aljoscha Krettek wrote: Hi, The answer is correct but I'll try and elaborate a bit: the way data is sent to downstream operations depends on a couple

Re: Apache Flink - Connected Stream with different number of partitions

2018-01-03 Thread Aljoscha Krettek
Hi, The answer is correct but I'll try and elaborate a bit: the way data is sent to downstream operations depends on a couple of things in this case: - parallelism of first input operation - parallelism of second input operation - parallelism of co-operation - transmission pattern on first i

Re: Apache Flink - Connected Stream with different number of partitions

2018-01-03 Thread Timo Walther
Hi Mans, I did a quick test on my PC where I simply set breakpoints in map1 and map2 (someStream has parallelism 1, otherStream 5, my CoMapFunction 8). Elements of someStream end up in different CoMapTasks (2/8, 7/8 etc.). So I guess the distribution is a round robin partioning. @Aljoscha mig

Apache Flink - Connected Stream with different number of partitions

2017-12-31 Thread M Singh
Hi: Referring to documentation (https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/stream/operators/index.html) for ConnectedStreams: "Connects" two data streams retaining their types. Connect allowing for shared state between the two streams.DataStream someStream = //... DataStream