feat: add DataDog notification provider

[mrparkers]

This PR adds support for DataDog as a notification provider.

This provider publishes events that look like this:

...given a provider config that looks like this:

apiVersion: notification.toolkit.fluxcd.io/v1beta2
kind: Provider
metadata:
  name: datadog
  namespace: flux-system
spec:
  secretRef:
    name: datadog-secret
  type: datadog
  address: https://api.datadoghq.com
  channel: my-k8s-cluster-name

...and a secret that looks like this:

apiVersion: v1
kind: Secret
metadata:
  name: datadog-secret
  namespace: flux-system
stringData:
  token: "datadog-api-key"
  headers: |
    my-custom-tag: tag-value

...and an alert that looks like this:

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: datadog-info
  namespace: flux-system
spec:
  eventSeverity: info
  eventSources:
    - kind: Kustomization
      name: "*"
    - kind: HelmRelease
      name: "*"
    - kind: GitRepository
      name: "*"
  providerRef:
    name: datadog

I made a few assumptions along the way based on the way other events / tags look within the DataDog ecosystem. I tried to leave comments in the code that explains the rationale behind each of these.

[stefanprodan]

Please use eventMetadata to set the custom tags instead of things in the secret along with the standard ones like revision and summary. The cluster name, as specified in docs should come from eventMetadata, the env can be set from the channel.

[mrparkers]

Thanks for the review @stefanprodan. I made the changes that you requested. I'm setting kube_cluster_name via the event summary if it exists, falling back to channel if it doesn't exist.

[mrparkers]

Okay, I think that make sense. I've removed the code that derives kube_cluster_name from the summary field of the event metadata.

[cep21]

Thanks for this change! We use datadog and this will be very helpful.

[stefanprodan]

@mrparkers can you please squash the commits and rebase with upstream main.

[mrparkers]

Done! Thanks again for the review!

[stefanprodan]

LGTM

Thanks @mrparkers 🏅

PS. We'll need another rebase with upstream main to merge it.

[mrparkers]

Hi @stefanprodan, I just rebased with main again. Let me know if there's anything else you need from me 😄

[stefanprodan]

@mrparkers can you please run make tidy and force push the changes?

[mrparkers]

@stefanprodan Done!