Re: FLIP-6 and running many "small" jobs

2016-11-04 Thread Maciek Próchniak
Hi Max, thanks for answer. I still have to wrap my head around it, but I hope we'll manage to work it out - maybe when 1.3.x arrives I'll have access to some nice mesos cluster... or not... we'll see :) thanks, maciek On 25/10/2016 17:49, Maximilian Michels wrote: Hi Maciek, Your use ca

Re: FLIP-6 and running many "small" jobs

2016-10-25 Thread Maximilian Michels
Hi Maciek, Your use case will be covered by the FLIP-6 "Sessions". Sessions are similar to how the on-premise Flink or the Yarn session operates today. We will have a long-running dispatcher, resource manager, and task managers. We will bring up a job manager for each job but the overhead for this

FLIP-6 and running many "small" jobs

2016-10-20 Thread Maciek Próchniak
Hi, we're looking at FLIP-6 and while it looks really great we started to wonder how it fits in our use case. We currently have around 20 processes but the idea is to have many more of them. Many of them are pretty "small" - them don't large sources, are stateless, mainly filtering data. A