Alerting / Notifications

[jantman]

My employer would like built-in support in awslimitchecker for notifying/alerting someone when there are warning or critical thresholds crossed (or, in theory, if the run fails). We use PagerDuty but we'd like to make it easy for others to contribute integrations for other notification methods. I'm being allowed to work on this during my work time.

My thought is implementing something similar to the plan in #418 for metrics outputs - we'd have a NotificationProvider base class with a notify() method that takes a list of limits with critical thresholds passed and a list of limits with warning thresholds passed, and some configuration information from CLI options. It should also handle sending a notification if the run failed. The first subclass would be PagerDuty, but we'd accept PRs for additional ones.

[jantman]

Implementation plan is something similar to Metrics providers...

• Similar CLI options as notification providers - list, specify class, and config options
• Initialize before running the limit checks
• wrap everything after initialize in a try/finally, so we can notify on failures/exceptions
• send metrics first then notification
• PagerDuty notifier
  • by default, if service_key is specified, send success/resolve if no issues
  • include formatted text output as well as data dict for warnings/criticals
  • config option for notification title/subject
  • config option for additional information to be included at end of notification
  • config options for an account name/alias
  • config option to specify a severity, instead of using the one from thresholds

[jantman]

Docs updates needed for this:

• cli_usage.rst.template - alerts section like metrics
• cli_usage.rst regenerate
• development.rst section on adding alert providers
• README.rst addition to features list
• getting_started.rst addition to feature list, and addition to IAM permissions admonition
• docs/build_generated_docs.py addition to IAM policy admonition and addition to examples

[jantman]

This has been fixed by #425 and will be in the next release.

[jantman]

This was merged to master in #427 and released as 7.1.0. The release is now live on PyPI and the Docker image should be live on Docker Hub shortly.