Hi all, I have a special use case that I'm not sure how I can fulfill. The use case is: I have my main business processing pipe line that has a MQ source, processFunction1, processFunction2 and MQ sink
PocessFunction1 apart from processing the main business message is also emitting some side effects using side outputs. Those side outputs are send to SideOutputMqSink that sends them to the queue. The requirement is that PocessFunction1 must not send out the main business message further to processFunction2 until side output from processFunction1 is send to the queue via SideOutputMqSink. In general I don't have to use side outputs, although I do some extra processing on them before sending to the sink so having sideOutput stream is nice to have. Never the less, the key requirement is that we should wait with further processing until side utput is send to the queue. I could achieve it in a way that my processFunction1 in processElement method will call MQ directly before sending out the main message, although I dot like that idea. I was thinking is there a way to have a Sink function that would be also a FlatMap function? The best solution would be to be able to process two streams (main and side effect) in some nice way but with some barrier, so the main pipeline will wait until side output is send. Both streams can be keyed. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/
