Re: streaming GroupBy + Fold

2015-10-14 Thread Márton Balassi
Thanks for the update. On Wed, Oct 14, 2015 at 10:12 AM, Martin Neumann wrote: > Hej, > > I checked the last Flink trunk version together with Aljoscha and the > problems are gone by now. (Just to close this discussion thread now) > > cheers Martin > > On Wed, Oct 7, 2015 at 1:21 PM, Aljoscha Kr

Re: streaming GroupBy + Fold

2015-10-14 Thread Martin Neumann
Hej, I checked the last Flink trunk version together with Aljoscha and the problems are gone by now. (Just to close this discussion thread now) cheers Martin On Wed, Oct 7, 2015 at 1:21 PM, Aljoscha Krettek wrote: > Hi, > I ran it using the attached TimeShift.java and I didn't get any key > cr

Re: streaming GroupBy + Fold

2015-10-07 Thread Aljoscha Krettek
Hi, I ran it using the attached TimeShift.java and I didn't get any key cross-talk. Could you please try my example, or verify that the problem still persists on your side? I replaced the source by a source that just creates random strings. On Tue, 6 Oct 2015 at 09:56 Martin Neumann wrote: >

Re: streaming GroupBy + Fold

2015-10-06 Thread Martin Neumann
The window is actually part of the workaround we currently using (should have commented it out) where we use a window and a MapFunction instead of a Fold. Original I was running fold without a window facing the same problems. The workaround works for now so there is no urgency on that one. I just

Re: streaming GroupBy + Fold

2015-10-05 Thread Aljoscha Krettek
Hi, If you are using a fold you are using none of the new code paths. I will add support for Fold to the new windowing implementation today, though. Cheers, Aljoscha On Mon, 5 Oct 2015 at 23:49 Márton Balassi wrote: > Martin, I have looked at your code and you are running a fold in a window, >

Re: streaming GroupBy + Fold

2015-10-05 Thread Márton Balassi
Martin, I have looked at your code and you are running a fold in a window, that is a very important distinction - the code paths are separate. Those code paths have been recently touched by Aljoscha if I am not mistaken. I have mocked up a simple example and could not reproduce your problem unfort

Re: streaming GroupBy + Fold

2015-10-05 Thread Márton Balassi
Thanks, I am checking it out tomorrow morning. On Mon, Oct 5, 2015 at 9:59 PM, Martin Neumann wrote: > Hej, > > Sorry it took so long to respond I needed to check if I was actually > allowed to share the code since it uses internal datasets. > > In the appendix of this email you will find the ma

Re: streaming GroupBy + Fold

2015-10-05 Thread Martin Neumann
Hej, Sorry it took so long to respond I needed to check if I was actually allowed to share the code since it uses internal datasets. In the appendix of this email you will find the main class of this job without the supporting classes or the actual dataset. If you want to run it you need to repla

Re: streaming GroupBy + Fold

2015-10-03 Thread Márton Balassi
Hey, Thanks for reporting the problem, Martin. I have not merged the PR Stephan is referring to yet. [1] There I am cleaning up some of the internals too. Just out of curiosity, could you share the code for the failing test please? [1] https://github.com/apache/flink/pull/1155 On Fri, Oct 2, 201

Re: streaming GroupBy + Fold

2015-10-02 Thread Martin Neumann
One of my colleagues found it today when we where hunting bugs today. We where using the latest 0.10 version pulled from maven this morning. The program we where testing is new code so I cant tell you if the behavior has changed or if it was always like this. On Fri, Oct 2, 2015 at 7:46 PM, Stepha

Re: streaming GroupBy + Fold

2015-10-02 Thread Stephan Ewen
I think these operations were recently moved to the internal state interface. Did the behavior change then? @Marton or Gyula, can you comment? Is it per chance not mapped to the partitioned state? On Fri, Oct 2, 2015 at 6:37 PM, Martin Neumann wrote: > Hej, > > In one of my Programs I run a Fol

streaming GroupBy + Fold

2015-10-02 Thread Martin Neumann
Hej, In one of my Programs I run a Fold on a GroupedDataStream. The aim is to aggregate the values in each group. It seems the aggregator in the Fold function is shared on operator level, so all groups that end up on the same operator get mashed together. Is this the wanted behavior? If so, what