Re: Flink streaming questions

2017-01-09 Thread Henri Heiskanen
Hi, Unfortunately I can not use reduce function. I am now going with WindowFunction and see how it works on our production load. Br, Henkka On Wed, Jan 4, 2017 at 2:46 PM, Fabian Hueske wrote: > Hi Henri, > > can you express the logic of your FoldFunction (or WindowFunction) as a > combinatio

Re: Flink streaming questions

2017-01-04 Thread Fabian Hueske
Hi Henri, can you express the logic of your FoldFunction (or WindowFunction) as a combination of ReduceFunction and WindowFunction [1]? ReduceFunction should be supported by a merging WindowAssigner and has the same resource consumption as a FoldFunction, i.e., a single record per window. Best, F

Re: Flink streaming questions

2017-01-03 Thread Henri Heiskanen
Hi, Actually it seems "Fold cannot be used with a merging WindowAssigner" and workaround I found was to use generic window function. It seems that I would need to use the window apply anyway. Functionality is then all there, but I am really concerned on the resource utilisations. We have quite man

Re: Flink streaming questions

2017-01-02 Thread Jamie Grier
Hi Henri, #1 - This is by design. Event time advances with the slowest input source. If there are input sources that generate no data this is indistinguishable from a slow source. Kafka topics where some partitions receive no data are a problem in this regard -- but there isn't a simple solutio

Flink streaming questions

2017-01-02 Thread Henri Heiskanen
Hi, I have few questions related to Flink streaming. I am on 1.2-SNAPSHOT and what I would like to accomplish is to have a stream that reads data from multiple kafka topics, identifies user sessions, uses an external user user profile to enrich the data, evaluates an script to produce session aggr