Hey Sonam, I'm wondering whether it may be helpful to have a min and max parallelism, > and the actual parallelism be determined by the scaling policy mentioned > next?
Yes, that's certainly possible. Thanks a lot for your input on the design of a scaling policy. Your input is very valuable for scoping the evolution of reactive mode for Flink 1.14. Once I know more about the planning for the 1.14 release, I can tell you where we are headed. Support for the active resource managers is very high on the list in my opinion. Best, Robert On Thu, Mar 11, 2021 at 6:57 PM Sonam Mandal <soman...@linkedin.com> wrote: > Hi Robert, > > Thanks for getting back to me. We are currently assessing Flink Standalone > on Kubernetes and Native Flink on Kubernetes and haven't yet decided on > which model we intend to use. We want to ensure that whichever model we > choose, we'll be able to get the benefits of the new features added by the > community. > > >> We are certainly aware that support for active deployments is a much > desired feature. The "problem" with the 1.13 implementation of reactive > mode is that it will try to acquire infinite resources from an active > resource manager. > > Good point, thanks for explaining why this is a challenge for active mode. > I'm wondering whether it may be helpful to have a min and max parallelism, > and the actual parallelism be determined by the scaling policy mentioned > next? > > >> For integration with an active deployment, how would you like to > control the scaling behavior of Flink? (for example via a REST API call to > Flink's JobManager, or via a programmatic scaling policy, or a configured > scaling policy? If you prefer a scaling policy, which metric would you like > to consider?) > > In the long term, I think having some kind of pluggable/extensible scaling > policy would be best for users to allow flexibility in choosing metrics > that are important for their use case. Making it configurable might make it > easier to pick and choose different policies if they are available, without > needing to make code changes. > > Some possible metrics to start with could be related to resource > utilization, such as CPU, memory, or other characteristics such as how much > the job is lagging? > > Since we are in early stages of just assessing what kind of deployment > model we'd like to use, it's hard to say what will work best for us. We > just want to see if reactive mode will be available in the future so that > we can leverage it when we have more data. > > Thanks, > Sonam > > > ------------------------------ > *From:* Robert Metzger <rmetz...@apache.org> > *Sent:* Thursday, March 11, 2021 5:28 AM > *To:* Sonam Mandal <soman...@linkedin.com> > *Cc:* user@flink.apache.org <user@flink.apache.org> > *Subject:* Re: Question about Reactive mode support > > Hey Sonam, > > I'm very happy to hear that you are interested in reactive mode. Your > understanding of the limitations for 1.13 is correct. Note that you can > deploy standalone Flink on Kubernetes [1]. I'm actually currently preparing > a demo for this [2]. > > We are certainly aware that support for active deployments is a much > desired feature. The "problem" with the 1.13 implementation of reactive > mode is that it will try to acquire infinite resources from an active > resource manager. > > For integration with an active deployment, how would you like to control > the scaling behavior of Flink? (for example via a REST API call to Flink's > JobManager, or via a programmatic scaling policy, or a configured scaling > policy? If you prefer a scaling policy, which metric would you like to > consider?) > > Best, > Sonam > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/ > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fci.apache.org%2Fprojects%2Fflink%2Fflink-docs-master%2Fdocs%2Fdeployment%2Fresource-providers%2Fstandalone%2Fkubernetes%2F&data=04%7C01%7Csomandal%40linkedin.com%7C3bd03c356faa4ac2d9f708d8e491a82f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637510661534528574%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=JocGO%2BfYHQf9ZtjjVjGzG0Mu0o1Oz3u4FTaTZZD3BiU%3D&reserved=0> > [2] https://github.com/rmetzger/flink-reactive-mode-k8s-demo > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frmetzger%2Fflink-reactive-mode-k8s-demo&data=04%7C01%7Csomandal%40linkedin.com%7C3bd03c356faa4ac2d9f708d8e491a82f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637510661534538566%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=xQJ4z821CA8XmO4S4Wu5AIiR9k4xb9LXahLQOT4sCk4%3D&reserved=0> > (attention, this is really work in progress!) > > > > > On Wed, Mar 10, 2021 at 5:32 PM Sonam Mandal <soman...@linkedin.com> > wrote: > > Hello, > > We were going through FlIP-159 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FFLINK%2FFLIP-159%253A%2BReactive%2BMode&data=04%7C01%7Csomandal%40linkedin.com%7C3bd03c356faa4ac2d9f708d8e491a82f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637510661534538566%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=FXX%2FhtNZfNXdi5DrQGvFoO0o%2BDVovHLLM1izxLTAL2g%3D&reserved=0> > and FLIP-160 > <https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FFLINK%2FFLIP-160%253A%2BAdaptive%2BScheduler&data=04%7C01%7Csomandal%40linkedin.com%7C3bd03c356faa4ac2d9f708d8e491a82f%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637510661534548560%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Yqafz7xR69QOZFcAXZd0OVqmFWzgl%2FsWo3Db4AyGvkw%3D&reserved=0> > and > found this feature of interest to us for auto-scaling purposes. The > limitations indicate that Flink 1.13 will release this for standalone only > and for application mode deployments only. > > Will this be extended in future releases to other active deployments such > as Native Flink on Kubernetes? What about session mode? > > Thanks, > Sonam > >
