Hey Nathan, Strange, so just an update of cloudstack version cause this to break? Ill check with my guys later today.
After the upgrade, VR system template version should need an upgrade. Is that done? If not, will a cleanup work? Its not really solving the root problem but it is a stop gap solution. Regards, Bryan On 16 Aug 2024 at 8:31 AM +0800, Nathan Gleason <[email protected]>, wrote: > Hi Bryan, thanks for the replies. > > We are able to create ASGs without issue as well, it’s just the scaling that > doesn’t work. We don’t have any orphan VMs from the LB. We did see the > Windows nugget, but these are all just regular Ubuntu VMs. We haven’t had any > node failures. > > Thank you, > Nathan > > > On Aug 15, 2024, at 20:05, Bryan Tiang <[email protected]> wrote: > > > > Hey Nathan, > > > > As for the metrics not showing right, do you happen to be using Windows > > Guest VMs for the autoscale? > > > > I remember something about Windows VMs not being able to get metrics on > > cloudstack correctly unless some setting was made in the hypervisor. > > > > Regards, > > Bryan > > On 16 Aug 2024 at 8:03 AM +0800, Bryan Tiang <[email protected]>, > > wrote: > > > Hey Nathan, > > > > > > Our company uses around 20 Autoscale Groups at the moment with 4.19.1.1 > > > with KVM. > > > > > > Since the upgrade, we definitely tested being able to create new ASGs > > > without issue, but ive not tested the scale up and down scenario. > > > > > > But we did found a bug where if an ASG VM is restarted after a node > > > failure, the scale down policies dont work anymore. Github ticket below: > > > > > > https://github.com/apache/cloudstack/issues/9336 > > > > > > Might affect Scale Up policies too. > > > > > > What we did was recreate the VR with Cleanup. If that doesnt work, turn > > > off the policy, delete affected ASG VMs and turn it on again so all VMs > > > are new again. > > > > > > Another possible scenario i could think is that your the ASG VM became an > > > orphan for some reason (again from restart after node failure) where the > > > VM is restarted but not recorded under the LB. You can go to ASG Group > > > > LB > Press + icon to display all the VMs and see if yours is there. If > > > not, this can explain why the scale up/down policies no longer work. > > > > > > https://github.com/apache/cloudstack/issues/9145 > > > > > > Regards, > > > Bryan > > > On 16 Aug 2024 at 6:19 AM +0800, Nathan Gleason > > > <[email protected]>, wrote: > > > > Hello, > > > > > > > > We’ve recently upgraded Cloudstack from 4.18.1.0 to 4.19.1.0, then to > > > > 4.19.1.1. While testing Autoscale we’ve found that a ScaleUp policy for > > > > “VM CPU - average percentage” does not work. We run cpuburn on the VM > > > > to load the CPU to 100%. We see the metrics in the UI as well as in the > > > > autoscale_vmgroup_statistics table. But scale up never happens. We’ve > > > > set the threshold anywhere from 1% to 50% but it does not work. We have > > > > restarted cloudstack-mangement, cloudstack-agent, libvirtd, etc… Has > > > > anyone encountered this? > > > > > > > > This may be unrelated but we have also noticed that memory metrics for > > > > all of the VMs are incorrect. We load the memory with memtester and > > > > while the VM shows the memory usage properly, the metrics do not. We > > > > found this while testing “VM Memory - average percentage” in ScaleUp > > > > policies. > > > > > > > > Versions: > > > > > > > > OS: Ubuntu 22.04 > > > > Cloudstack: 4.19.1.1 > > > > Hypervisor: KVM > > > > > > > > Libvirt: > > > > Compiled against library: libvirt 8.0.0 > > > > Using library: libvirt 8.0.0 > > > > Using API: QEMU 8.0.0 > > > > Running hypervisor: QEMU 6.2.0 > > > > > > > > Thank you, > > > > Nathan >
