> > Currently, I saw that Magnum only send CPU and Memory metric to Ceilometer, > > and Heat can grab these to decide the right scaling method. IMO, this > > approach have some problems, please take a look and give feedbacks: > > - The AutoScaling Policy and AutoScaling Resource of Heat cannot handle > > complex scaling policies. For example: > > If CPU > 80% then scale out > > If Mem < 40% then scale in > > -> What if CPU = 90% and Mem = 30%, the conflict policy will appear. > > There are some WIP patch-set of Heat conditional logic in [1]. But IMO, the > > conditional logic of Heat also cannot resolve the conflict of scaling > > policies. For example: > > If CPU > 80% and Mem >70% then scale out If CPU < 30% or Mem < 50% > > then scale in > > -> What if CPU = 90% and Mem = 30%. > > What would you like Heat to do in this scenario ? Is it that you would like > to have a user defined logic option as well as basic conditionals ?
Thank you Tim for feedback. Yes, I'd like Heat to validate the user-defined policies along with heat template-validate mechanism. > > I would expect the same problem to occur in pure Heat scenarios also so a > user defined scaling policy would probably be of interest there too and avoid > code duplication. > > Tim Currently, there are some bp like [1] related to auto scaling policies but I cannot see the interest related to this problem. Hope someone can show me. The Magnum scaler can be a centralize spot for both Magnum and Zun to scale COE node via Heat or container via COE scaling API (k8s already have auto scaling engine). [1]. https://blueprints.launchpad.net/heat/+spec/as-lib -----Original Message----- From: Tim Bell [mailto:tim.b...@cern.ch] Sent: Thursday, August 18, 2016 3:19 PM To: OpenStack Development Mailing List (not for usage questions) <openstack-dev@lists.openstack.org> Subject: Re: [openstack-dev] [Magnum] Next auto-scaling feature design? > On 18 Aug 2016, at 09:56, hie...@vn.fujitsu.com wrote: > > Hi Magnum folks, > > I have some interests in our auto scaling features and currently testing with > some container monitoring solutions such as heapster, telegraf and > prometheus. I have seen the PoC session corporate with Senlin in Austin and > have some questions regarding of this design: > - We have decided to move all container management from Magnum to Zun, so is > there only one level of scaling (node) instead of both node and container? > - The PoC design show that Magnum (Magnum Scaler) need to depend on > Heat/Ceilometer for gathering metrics and do the scaling work based on auto > scaling policies, but is Heat/Ceilometer is the best choice for Magnum auto > scaling? > > Currently, I saw that Magnum only send CPU and Memory metric to Ceilometer, > and Heat can grab these to decide the right scaling method. IMO, this > approach have some problems, please take a look and give feedbacks: > - The AutoScaling Policy and AutoScaling Resource of Heat cannot handle > complex scaling policies. For example: > If CPU > 80% then scale out > If Mem < 40% then scale in > -> What if CPU = 90% and Mem = 30%, the conflict policy will appear. > There are some WIP patch-set of Heat conditional logic in [1]. But IMO, the > conditional logic of Heat also cannot resolve the conflict of scaling > policies. For example: > If CPU > 80% and Mem >70% then scale out If CPU < 30% or Mem < 50% > then scale in > -> What if CPU = 90% and Mem = 30%. What would you like Heat to do in this scenario ? Is it that you would like to have a user defined logic option as well as basic conditionals ? I would expect the same problem to occur in pure Heat scenarios also so a user defined scaling policy would probably be of interest there too and avoid code duplication. Tim > Thus, I think that we need to implement magnum scaler for validating the > policy conflicts. > - Ceilometer may have troubles if we deploy thousands of COE. > > I think we need a new design for auto scaling feature, not for Magnum only > but also Zun (because the scaling level of container maybe forked to Zun > too). Here are some ideas: > 1. Add new field enable_monitor to cluster template (ex baymodel) and show > the monitoring URL when creating cluster (bay) complete. For example, we can > use Prometheus as monitoring container for each cluster. (Heapster is the > best choice for k8s, but not good enough for swarm or mesos). > 2. Create Magnum scaler manager (maybe a new service): > - Monitoring enabled monitor cluster and send metric to ceilometer if need. > - Manage user-defined scaling policy: not only cpu and memory but also other > metrics like network bw, CCU. > - Validate user-defined scaling policy and trigger heat for scaling > actions. (can trigger nova-scheduler for more scaling options) > - Need highly scalable architecture, first step we can implement simple > validator method but in the future, there are some other approach such as > using fuzzy logic or AI to make an appropriate decision. > > Some use case for operators: > - I want to create a k8s cluster, and if CCU or network bandwidth is high > please scale-out X nodes in other regions. > - I want to create swarm cluster, and if CPU or memory is too high, please > scale-out X nodes to make sure total CPU and memory is about 50%. > > What do you think about these above ideas/problems? > > [1]. > https://blueprints.launchpad.net/heat/+spec/support-conditions-functio > n > > Thanks, > Hieu LE. > > > ______________________________________________________________________ > ____ OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
