Upgrade OpsGenie notifier to v2 API

[Tom-Fawcett]

Closes #902

The OpsGenie Web API (referred to as 'old' below) is deprecated.

This upgrades Alertmanager to the OpsGenie REST Alert API (referred to as 'new' below).

---

Key changes made

• Use new paths.
• Use new auth header.
• Adjust response code logic.
  • Docs
  • Expand retry to include new "429 - Too Many Requests" throttling.
  • We no longer need to handle closing an already closed alert, as such a request is now asynchronous.

Transition

• No changes in config.
• New integration still uses alert alias, so it is backwards compatible.
  • i.e. you can open a request on the old API and close it on the new API.

Tests

• Open an alert with Alertmanager, then close with Alertmanager. ✓
• Open an alert with Alertmanager, close elsewhere, then observe Alertmanager attempting to close without erroring. ✓

Asynchronous

Unlike the old API, the new API is asynchronous.

Alert creation, deletion and action requests are processed asynchronously to provide higher availability and scalability, therefore valid requests for those endpoints are responded with HTTP status 202 - Accepted.

link

• It seems reasonable to keep things simple, and not attempt to write a load of logic around requesting the status of previous requests.
• This works to our advantage when attempting to close an already closed alert, as we are returned 202.

[stuartnelson3]

Looks good, thanks!

[Tom-Fawcett]

Great 😄