Re: How to preserve KeyedDataStream

2015-11-04 Thread Aljoscha Krettek
Hi, I’m afraid there is no other solution besides calling keyBy() again if you require a keyed data stream. This has to do with the data model of Flink, where the key is part of the regular data element. This is different from systems that have a (Key, Value) data model. Those systems can provid

Re: How to preserve KeyedDataStream

2015-11-03 Thread Martin Neumann
Here is some code of my current solution, if there is some better way of doing it let me know. KeyedStream hostStream = inStream .keyBy(t -> t.getHost()); KeyedStream,String> freqStream = hostStreams .timeWindow(Time.of(WINDOW_SIZE, TimeUnit.MILLISECONDS)) .fold(new Tuple2(0, ""), new Count()

Re: How to preserve KeyedDataStream

2015-11-03 Thread Aljoscha Krettek
Hi, where are you storing the results of each window computation to? Maybe you could also store it from inside a custom WindowFunction where you just count the elements and then store the results. On the other hand, adding a (1) field and doing a window reduce (à la WordCount) is going to be wa

How to preserve KeyedDataStream

2015-11-03 Thread Martin Neumann
Hej, I want to do the following thing: 1. Split a Stream of incoming Logs by host address. 2. For each Key, create time based windows 3. Count the number of items in the window 4. Feed it into a statistical model that is maintained for each host Since I don't have fields to sum upon, I use a (win