Re: threads, parallelism and task managers

2016-03-25 Thread Flavio Pompermaier
Any help here? I think that the problem is that the JobManager creates the executionContext of the scheduler with val executionContext = ExecutionContext.fromExecutor(new ForkJoinPool()) and thus the number of concurrently running threads is limited to the number of cores (using the defaul

Re: TumblingProcessingTimeWindow and ContinuousProcessingTimeTrigger

2016-03-25 Thread Kostas Kloudas
Hello Aljoscha and Hironori, Nice initiative! I totally agree with the proposals in the document. I also left some comments and I will soon start working on some of the issues there. Kostas > On Mar 25, 2016, at 12:53 PM, Hironori Ogibayashi > wrote: > > Aljoscha, > > Yes, it's reproducibl

Re: TumblingProcessingTimeWindow and ContinuousProcessingTimeTrigger

2016-03-25 Thread Hironori Ogibayashi
Aljoscha, Yes, it's reproducible as long as I tried. Here is the code and procedure: https://gist.github.com/ogibayashi/402153bcd79138c35d6a Thank you for your explanation about the removal of windows. I didn't know that calling .trigger() will replace default window trigger. I have read your doc

Re: does reduce function has a bug

2016-03-25 Thread Balaji Rajagopalan
Never mind Till figured out a way, instead of doing the aggregation in reduce, I moved that logic to apply of the window function. On Thu, Mar 24, 2016 at 11:33 PM, Balaji Rajagopalan < balaji.rajagopa...@olacabs.com> wrote: > Till, > > Thanks for your reply, may be I should have given more det