Re: Cyclic ConnectedStream

2017-02-05 Thread Matt
I really don't know what you mean, I've been reading the documentation and examples showing iterations. but it just won't work for me I believe. Maybe you can write a quick example? It doesn't matter the details, only the topology. If anyone else has an idea it's very welcome! Matt On Tue, Jan 3

Re: Cyclic ConnectedStream

2017-01-31 Thread Gábor Gévay
I somehow still suspect that iterations might work for your use case. Note, that in the streaming API, iterations are currently nothing more than a back-edge in the topology, i.e. a low-level tool to create a cyclic topology, like as you say with your hypothetical setter syntax. (It's quite differe

Re: Cyclic ConnectedStream

2017-01-29 Thread Matt
Check this image for clarification, this is what I'm trying to do: http://i.imgur.com/iZxPv04.png [image: Inline image 1] The rectangles are the two CoFlatMapFunction, sharing a state between process and update (map1 and map2). It's clear from the image that I need input1 and the green box to cre

Re: Cyclic ConnectedStream

2017-01-28 Thread Matt
I'm aware of IterativeStream but I don't think it's useful in this case. As shown in the example above, my use case is "cyclic" in that the same object goes from *Input* to *predictionStream* (flatMap1), then to *statsStream* (flatMap2, where it's updated with an object from *Input2*) and finally

Re: Cyclic ConnectedStream

2017-01-28 Thread Gábor Gévay
Hello, Cyclic dataflows can be built using iterations: https://ci.apache.org/projects/flink/flink-docs-release-1.2/dev/datastream_api.html#iterations Best, Gábor 2017-01-28 18:39 GMT+01:00 Matt : > I have a ConnectedStream (A) that depends on another ConnectedStream (B), > which depends on th

Cyclic ConnectedStream

2017-01-28 Thread Matt
I have a ConnectedStream (A) that depends on another ConnectedStream (B), which depends on the first one (A). Simplified code: *predictionStream = **input* .connect(*statsStream*) .keyBy(...) .flatMap(CoFlatMapFunction { flatMap1(obj, output) { p = prediction(obj) * ou