Re: expose side output stream

2016-08-13 Thread Chen Qin
Stephan & Aljoscha, What I did was a API hack without much thoughts into architect at beginning :-) patch attached, I think it would be "tricky" to achieve backward compatibility with current architect. We might need to encapsulate "Collectors" to ProcessContext as starting point. I would try to

Re: expose side output stream

2016-08-12 Thread Aljoscha Krettek
Hi Chen, could you maybe share the code that you have so far? If you wan't you can start a google doc and then we can work together on fleshing out an API/implementation that we can present to the Flink community as a FLIP. Cheers, Aljoscha On Thu, 11 Aug 2016 at 14:40 Stephan Ewen wrote: > Hi

Re: expose side output stream

2016-08-11 Thread Chen Qin
Stephan Humm... I see. Back off one step, how do Flink deal with corrupted input data right now, like a dead letter queue? Thanks, Chen On Thu, Aug 11, 2016 at 5:40 AM, Stephan Ewen wrote: > Hi! > > This is a very big change, both on the semantics, the runtime classes. > These changes are tric

Re: expose side output stream

2016-08-11 Thread Stephan Ewen
Hi! This is a very big change, both on the semantics, the runtime classes. These changes are tricky to get in, and usually work best if you document the changes and all implications well. Something like a deep design doc, or a FLIP would be great for this. https://cwiki.apache.org/confluence/disp