Re: Question about SlidingPreReducers

2015-04-30 Thread Szabó Péter
So, would it be a reasonable solution to just modify the WindowUtils.isParallelPolicy() method to return false in case of "eviction instanceof CountEvictionPolicy && trigger instanceof TimeTriggerPolicy" ? 2015-04-30 12:21 GMT+02:00 Gyula Fóra : > I'm referring to Peter's problem. If we just crea

Re: Question about SlidingPreReducers

2015-04-30 Thread Gyula Fóra
I'm referring to Peter's problem. If we just create more count discretizers it doesn't really break the semantics given the network guarantees but it is not very intuitive. On Thursday, April 30, 2015, Aljoscha Krettek wrote: > @Gyula: Are you referring to the pre-aggregator or the thing Peter >

Re: Question about SlidingPreReducers

2015-04-30 Thread Aljoscha Krettek
@Gyula: Are you referring to the pre-aggregator or the thing Peter mentioned? On Thu, Apr 30, 2015 at 11:12 AM, Gyula Fóra wrote: > The problem is in the WindowUtils.isParallel policy method. It makes count > policies automatically parallel as well. > > On Thursday, April 30, 2015, Aljoscha Krett

Re: Question about SlidingPreReducers

2015-04-30 Thread Gyula Fóra
The problem is in the WindowUtils.isParallel policy method. It makes count policies automatically parallel as well. On Thursday, April 30, 2015, Aljoscha Krettek wrote: > Hi, > no, I think the two are unrelated. But that's another problem we need > to tackle then. > > Cheers, > Aljoscha > > On T

Re: Question about SlidingPreReducers

2015-04-30 Thread Aljoscha Krettek
Hi, no, I think the two are unrelated. But that's another problem we need to tackle then. Cheers, Aljoscha On Thu, Apr 30, 2015 at 9:15 AM, Szabó Péter wrote: > Hey, > > our intern, Pablo pointed out that there is some problem with mixed > windowing policies. When you write > ... > .window(C

Re: Question about SlidingPreReducers

2015-04-30 Thread Szabó Péter
Hey, our intern, Pablo pointed out that there is some problem with mixed windowing policies. When you write ... .window(Count ...) .every(Time ...) .mapWindow(...) ... The result makes no sense, as the window is not of the specified length. Maybe, there is some conflict between Time and

Re: Question about SlidingPreReducers

2015-04-29 Thread Aljoscha Krettek
I am on it, yes. :D On Apr 29, 2015 8:31 PM, "Gyula Fóra" wrote: > Hey, > You are right, so there seems to be some error in the logic that checks for > the end of the slide. > I checked the count pre-reducer and the error there is clear, but if I fix > it it breaks some tests so I think some othe

Re: Question about SlidingPreReducers

2015-04-29 Thread Gyula Fóra
Hey, You are right, so there seems to be some error in the logic that checks for the end of the slide. I checked the count pre-reducer and the error there is clear, but if I fix it it breaks some tests so I think some other stuff must not work properly either. I cannot work on this until I get bac

Re: Question about SlidingPreReducers

2015-04-29 Thread Aljoscha Krettek
Oh, I saw them, but in updateCurrent() (both grouped and non-grouped) the else path is never taken when I run the windowed WordCount example. I have a branch where I just throw a RuntimeException in the else branches and the whole things still works: mvn verify doesn't fail, the examples still run

Re: Question about SlidingPreReducers

2015-04-29 Thread Gyula Fóra
Hey, They actually work :P Although I have to admit I need to do some refactoring of the method names and parameters. I made some quick refactoring and added some comments for the key methods: https://github.com/mbalassi/flink/blob/dfa028ce5102b1f0d098eaec6c4b4d3744bee379/flink-staging/flink-str