Re: SideOutput doesn't receive anything if filter is applied after the process function

Thanks Chesnay, So I think to support multi input and multiple output model like data flow paper indicates, Flink needs to get credit based scheduling as well as side input ready and doing a new set of data stream apis that doesn’t constrained with backwards compatibility issues. Only then can

Re: SideOutput doesn't receive anything if filter is applied after the process function

I've opened https://issues.apache.org/jira/browse/FLINK-8437 Unfortunately i doubt we can fix this properly. The proposed solution will not work if we ever allow arbitrary functions to use side-outputs. On 16.01.2018 08:59, Juho Autio wrote: Could someone with knowledge of the right terms crea

Re: SideOutput doesn't receive anything if filter is applied after the process function

Could someone with knowledge of the right terms create this in JIRA, please? I guess I could also create it if needed.. On Mon, Jan 15, 2018 at 3:15 PM, Chesnay Schepler wrote: > yes, i meant that process() returns the special operator. This would > definitely deserve a JIRA issue. > > > On 15.0

Re: SideOutput doesn't receive anything if filter is applied after the process function

yes, i meant that process() returns the special operator. This would definitely deserve a JIRA issue. On 15.01.2018 14:09, Juho Autio wrote: Thanks for the explanation. Did you meant that process() would return a SingleOutputWithSideOutputOperator? Any way, that should be enough to avoid the

Re: SideOutput doesn't receive anything if filter is applied after the process function

Thanks for the explanation. Did you meant that process() would return a SingleOutputWithSideOutputOperator? Any way, that should be enough to avoid the problem that I hit (and it also seems like the best & only way). Maybe the name should be something more generic though, like ProcessedSingleOutp

Re: SideOutput doesn't receive anything if filter is applied after the process function

It would mean that getSideOutput() would return a SingleOutputWithSideOutputOperator which extends SingleOutputOperator offering getSideOutput(). Other transformations would still return a SingleOutputOperator. With this the following code wouldn't compile. stream .process(...) .filte

Re: SideOutput doesn't receive anything if filter is applied after the process function

> sideoutput might deserve a seperate class which inherit form singleoutput. It might prevent lot of confusions Thanks, but how could that be done? Do you mean that if one calls .process(), then the stream would change to another class which would only allow calls like .getMainOutput() or .getSide

Re: SideOutput doesn't receive anything if filter is applied after the process function

Hi Juho, I think sideoutput might deserve a seperate class which inherit form singleoutput. It might prevent lot of confusions. A more generic question is whether datastream api can be mulitple ins and mulitple outs natively. It's more like scheduling problem when you come from single process syst

Re: SideOutput doesn't receive anything if filter is applied after the process function

Maybe I could express it in a slightly different way: if adding the .filter() after .process() causes the side output to be somehow totally "lost", then I believe the .getSideOutput() could be aware that there is not such side output to be listened to from upstream, and throw an exception. I mean,

Re: SideOutput doesn't receive anything if filter is applied after the process function

Hi Juho, Now that I think of it this seems like a bug to me: why does the call to getSideOutput succeed if it doesn't provide _any_ input? With the way side outputs work, I don’t think this is possible (or would make sense). An operator does not know whether or not it would ever emit some elem