Re: Auto/Dynamic scaling in Flink

2018-11-15 Thread 罗齐
> On Mon, Nov 12, 2018 at 5:50 PM Nauroz Khan Nausherwani > <mailto:nauroz-khan.nausherw...@etu.univ-cotedazur.fr>> wrote: > Dear Flink Contributors and users, > > I am a PhD student and I was interested to know, using which matrices, and > when does Flink perform

Re: Auto/Dynamic scaling in Flink

2018-11-13 Thread Tzu-Li Chen
es, >> and when does Flink performs scaling-in or scaling out of resources? I did >> search the flink's website where I could only find information about how >> dynamic scaling is performed in stateless or stateful operator. It would >> be interesting to know which mat

Re: Auto/Dynamic scaling in Flink

2018-11-13 Thread Tzu-Li (Gordon) Tai
te: > Dear Flink Contributors and users, > > I am a PhD student and I was interested to know, using which matrices, and > when does Flink performs scaling-in or scaling out of resources? I did > search the flink's website where I could only find information about how > dynamic

Auto/Dynamic scaling in Flink

2018-11-12 Thread Nauroz Khan Nausherwani
Dear Flink Contributors and users, I am a PhD student and I was interested to know, using which matrices, and when does Flink performs scaling-in or scaling out of resources? I did search the flink's website where I could only find information about how dynamic scaling is perform

Re: Flink dynamic scaling 1.5

2018-07-10 Thread Anil
Thanks for the reply Till. Resubmitting the job is an option. I was wondering if there's any way that Flink could be configured to detect issues like a memory issue and rescale without me submitting the job again. -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.co

Re: Flink dynamic scaling 1.5

2018-07-09 Thread Till Rohrmann
Hi Anil, you can use `flink modify -p ` to scale a running job under JOB_ID to the NEW_PARALLELISM. Cheers, Till On Mon, Jul 9, 2018 at 9:45 AM Anil wrote: > This is regarding dynamic rescaling in Flink 1.5 > > I am using Yarn for running Flink jobs. I start these jobs with a static > resourc

Flink dynamic scaling 1.5

2018-07-09 Thread Anil
This is regarding dynamic rescaling in Flink 1.5 I am using Yarn for running Flink jobs. I start these jobs with a static resource. Is there any option to scale out these job by itself in specific conditions like if there's a memory issues. In Flink 1.5 release notes - Applications can be rescal

Flink dynamic scaling 1.5

2018-07-09 Thread Anil
This is regarding dynamic rescaling in Flink 1.5 I am using Yarn for running Flink jobs. I start these jobs with a static resource. Is there any option to scale out these job by itself in specific conditions like if there's a memory issues. In Flink 1.5 release notes - Applications can be rescal

Re: Flip-6 + Dynamic scaling

2018-01-26 Thread Gary Yao
Hi Jayant, I am working on FLIP-6, and I can tell you that we are aiming at shipping it with Flink 1.5. For the scope and release timeline see this blog post: https://flink.apache.org/news/2017/11/22/release-1.4-and-1.5-timeline.html Best, Gary On Wed, Jan 24, 2018 at 7:02 AM, Jayant Ameta w

Flip-6 + Dynamic scaling

2018-01-23 Thread Jayant Ameta
Hi, There hasn't been any activity on https://issues.apache.org/jira/browse/FLINK-4319 since September 21. Any ETA on this? I required dynamic upscaling and downscaling based on the volume of data. Jayant Ameta

Re: Dynamic Scaling

2017-01-02 Thread Jamie Grier
Hi Govind, In Flink 1.2 (feature complete, undergoing test) you will be able to scale your jobs/operators up and down at will, however you'll have to build a little tooling around it yourself and scale based on your own metrics. You should be able to integrate this with Docker Swarm or Amazon aut

Re: Dynamic Scaling

2016-12-26 Thread Govindarajan Srinivasaraghavan
Hi All, It would great if someone can help me with my questions. Appreciate all the help. Thanks. > On Dec 23, 2016, at 12:11 PM, Govindarajan Srinivasaraghavan > wrote: > > Hi, > > We have a computation heavy streaming flink job which will be processing > around 100 million message at pea

Dynamic Scaling

2016-12-23 Thread Govindarajan Srinivasaraghavan
Hi, We have a computation heavy streaming flink job which will be processing around 100 million message at peak time and around 1 million messages in non peak time. We need the capability to dynamically scale so that the computation operator can scale up and down during high or low work loads resp

Re: Dynamic scaling in flink

2016-08-29 Thread Robert Metzger
I understand guaranteeing exactly once semantics with the > dynamic scaling is tough. If I were to let go of the exactly once > requirement, is it not possible in current version? It would be really > great if you can point me to the JIRA tracking this work. > > On Mon, Aug 29, 2016 at

Re: Dynamic scaling in flink

2016-08-29 Thread Abhishek Agarwal
Thanks Stephan. I understand guaranteeing exactly once semantics with the dynamic scaling is tough. If I were to let go of the exactly once requirement, is it not possible in current version? It would be really great if you can point me to the JIRA tracking this work. On Mon, Aug 29, 2016 at 2:30

Re: Dynamic scaling in flink

2016-08-29 Thread Stephan Ewen
Hi! There is a lot of work in progress on that feature, and it looks like you can expect the next version to have some upscale/downscale feature that maintains exactly-once semantics. Stephan On Mon, Aug 29, 2016 at 9:00 AM, Abhishek Agarwal wrote: > Is it possible to upscale or downscale a f

Dynamic scaling in flink

2016-08-29 Thread Abhishek Agarwal
Is it possible to upscale or downscale a flink application without re-deploying (similar to rebalancing in storm)? -- Regards, Abhishek Agarwal