Re: Fold vs Reduce in DataStream API

2015-12-31 Thread Aljoscha Krettek
No, the internal behavior is not documented right now. But I think it will change before the 1.0 release. > On 30 Dec 2015, at 18:44, Brian Chhun wrote: > > Thanks for the clarification. Is there a resource besides the code that has > these kinds of things documented? Understandable if there i

Re: Fold vs Reduce in DataStream API

2015-12-30 Thread Brian Chhun
Thanks for the clarification. Is there a resource besides the code that has these kinds of things documented? Understandable if there isn't much out there yet and that these things are still in flux. On Wed, Dec 30, 2015 at 11:14 AM, Aljoscha Krettek wrote: > Yes, this is correct right now. It s

Re: Fold vs Reduce in DataStream API

2015-12-30 Thread Aljoscha Krettek
Yes, this is correct right now. It should not be too hard to add the pre-aggregation behavior for fold, however. > On 30 Dec 2015, at 17:31, Brian Chhun wrote: > > Hi All, > > Are certain considerations when using these functions on windowed streams? > > From reading the code, it looks using

Re: Fold vs Reduce in DataStream API

2015-12-30 Thread Brian Chhun
Hi All, Are certain considerations when using these functions on windowed streams? >From reading the code, it looks using reduce (or another aggregation function) on a windowed stream will pre-aggregate the result value as elements are added to the window, keeping the size of window constant. On

Re: Fold vs Reduce in DataStream API

2015-11-19 Thread Stephan Ewen
Hi Ron! Yes, we had to change a few things in the API between 0.9 and 0.10. The API in 0.9 had quite a few problems. This one now looks good, we are confident that it will stay. Greetings, Stephan On Thu, Nov 19, 2015 at 8:15 PM, Ron Crocker wrote: > Thanks Stephan, that helps quite a bit. Lo

Re: Fold vs Reduce in DataStream API

2015-11-19 Thread Ron Crocker
Thanks Stephan, that helps quite a bit. Looks like another one of those API changes that I'll be struggling with for a little bit. On Thu, Nov 19, 2015 at 10:40 AM, Stephan Ewen wrote: > Hi Ron! > > You are right, there is a copy/paste error in the docs, it should be a > FoldFunction that is pas

Re: Fold vs Reduce in DataStream API

2015-11-19 Thread Stephan Ewen
Hi Ron! You are right, there is a copy/paste error in the docs, it should be a FoldFunction that is passed to fold(), not a ReduceFunction. In Flink-0.10, the FoldFunction is only available on - KeyedStream ( https://ci.apache.org/projects/flink/flink-docs-release-0.10/api/java/org/apache/flin

Re: Fold vs Reduce in DataStream API

2015-11-19 Thread Ron Crocker
Hi Fabian - Thanks Fabian, that is a helpful description. That document WAS my source of information and it seems to also be the source of my confusion. Further, it appears to be wrong - there is a FoldFunction ( https://ci.apache.org/projects/flink/flink-docs-release-0.10/api/java/org/apache/fli

Re: Fold vs Reduce in DataStream API

2015-11-18 Thread Kostas Tzoumas
Granted, both are presented with the same example in the docs. They are modeled after reduce and fold in functional programming. Perhaps we should have a bit more enlightening examples. On Wed, Nov 18, 2015 at 6:39 PM, Fabian Hueske wrote: > Hi Ron, > > Have you checked: > https://ci.apache.org/

Re: Fold vs Reduce in DataStream API

2015-11-18 Thread Fabian Hueske
Hi Ron, Have you checked: https://ci.apache.org/projects/flink/flink-docs-release-0.10/apis/streaming_guide.html#transformations ? Fold is like reduce, except that you define a start element (of a different type than the input type) and the result type is the type of the initial value. In reduce,

Fold vs Reduce in DataStream API

2015-11-18 Thread Ron Crocker
Is there a succinct description of the distinction between these transforms? Ron — Ron Crocker Principal Engineer & Architect ( ( •)) New Relic rcroc...@newrelic.com M: +1 630 363 8835