Re: Aggregation by key hierarchy

2017-08-21 Thread Nico Kruber
Hi Basant, no, you cannot add data streams or re-wire your program during runtime. As for any other program changes, you'd have to take a savepoint (to keep operator state and exactly-once semantics) and restart the new program code from there. For a few combinations, I'd probably choose the sec

Re: Aggregation by key hierarchy

2017-08-16 Thread Basanth Gowda
Thanks Nico. As there are 2 ways to achieve this which is better ? 1st option -> dataStream.flatMap( ... ) -> this takes in out and provides me N number of outputs, depending on my key combination . On each of the output the same windowing logic is applied or the one you suggested 2nd option ->

Re: Aggregation by key hierarchy

2017-08-16 Thread Nico Kruber
[back to the ml...] also including your other mail's additional content... > I have been able to do this by the following and repeating this for every > key + window combination. So in the above case there would be 8 blocks like > below. (4 combinations and 2 window period for each combination) >

Re: Aggregation by key hierarchy

2017-08-14 Thread Nico Kruber
Hi Basanth, Let's assume you have records of the form Record = {timestamp, country, state, city, value} Then you'd like to create aggregates, e.g. the average, for the following combinations? 1) avg per country 2) avg per state and country 3) avg per city and state and country * You could create

Re: Aggregation by key hierarchy

2017-08-13 Thread Basanth Gowda
For example - this is a sample model from one of the Apache Apex presentation. I would want to aggregate for different combinations, and different time buckets. What is the best way to do this in Flink ? {"keys":[{"name":"campaignId","type":"integer"}, {"name":"adId","type":"integer"}, {"name":

Aggregation by key hierarchy

2017-08-13 Thread Basanth Gowda
Hi, I want to aggregate hits by Country, State, City. I would these as tags in my sample data. How would I do aggregation at different levels ? Input data would be single record Should I do flatMap transformation first and create 3 records from 1 input record, or is there a better way to do it ?