Damans227 opened a new pull request, #13478: URL: https://github.com/apache/cloudstack/pull/13478
### Description This PR adds resource alerts. You set a rule on a VM, volume, host or storage pool, like "cpu above 80%", and CloudStack checks it every minute. When it goes over, an alert is fired and sent out. - rules can be on one resource or on all resources of a type - alerts go out by webhook, email and the event bus - a cooldown per rule so the same alert doesnt keep repeating every minute - alert history on the rule, and an Alerts tab on each resource page - users, domain admins and root admins each see and manage only what they own. hosts, storage pools and email are root admin only - a resource can opt out of "all resources" rules with a tag - a rule on one resource only overrides your own "all resources" rule, so users cant switch off admin rules on their VMs - volume rules watch used space, in GB or as % of the disk size. disk read/write metrics are on VMs, since CloudStack doesnt collect disk IO per volume by default - deleting a rule deletes its alert history too - only one management server runs the checks, so alerts are not doubled when there are more servers - small change in the webhook plugin so alert deliveries show in the webhook deliveries tab and can be resent New global settings: resourcealert.evaluation.interval, resourcealert.repeat.interval.default, resourcealert.history.retention.days, resourcealert.per.user.limit Doc PR: https://github.com/apache/cloudstack-documentation/pull/686 ### Types of changes - [ ] Breaking change (fix or feature that would cause existing functionality to change) - [x] New feature (non-breaking change which adds functionality) - [ ] Bug fix (non-breaking change which fixes an issue) - [ ] Enhancement (improves an existing feature and functionality) - [ ] Cleanup (Code refactoring and cleanup, that may add test cases) - [ ] Build/CI - [ ] Test (unit or integration test code) ### Feature/Enhancement Scale or Bug Severity #### Feature/Enhancement Scale - [x] Major - [ ] Minor ### Screenshots (if appropriate): Rules list  Settings  New rule form  Rule details  Alert history of a rule  Alerts tab on a resource page  Alert received by a webhook  Alert deliveries on the webhook  Alert email  Alert on the event bus (RabbitMQ)  ### How Has This Been Tested? Tested on a KVM lab with 2 management servers, 2 hosts and NFS storage. Went through the UI end to end, and checked everything with cmk and the management server logs. Main thing tested was delivery, all three ways: - webhook: a small test receiver got the alerts, signed, with rule, resource, value and severity. filters on the webhook work, so an excluded alert is not sent but still shows in history. deliveries show in the webhook tab and resend works - email: a test mail server got the alert email only for rules with email on - event bus: RabbitMQ got the alerts on the cloudstack events exchange next to the normal events Also tested: - rules on one resource and on all resources, for VMs, volumes, hosts and storage pools - cooldown, alerts only repeat after the cooldown - rules under the threshold stay quiet, and metrics that a resource doesnt report are skipped - user, domain admin and root admin each only see and edit their own stuff. a user cant open or change another users rule - opt-out tag - unit tests for the plugin and the command error codes - new Marvin smoke test test_resource_alerts.py: rule lifecycle, bad input, user and domain admin limits, alert firing with delivery to a real webhook receiver, all-resources scope and opt-out, history removed with the rule, rule removed when the VM is expunged. The ones that need real VM stats are marked required_hardware=true, the rest run in CI on the simulator #### How did you try to break this feature and the system with this change? - bad input like a percentage over 100 or negative numbers - trying to create host rules or turn on email as a normal user through the api - opening, editing and deleting other users rules - both management servers up at once. checked there are no double alerts - webhook filters blocking alerts -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
