Standardize event metadata and notification template

[phillebaba]

Background

There have been some discussions notification-controller issues regarding extending configuration of the notifications being sent. Some of the are related to commit status notifications and other two chat notifications. What they have in common is that they could all be solved with allowing end users to create their own templates for the notifications.

fluxcd/notification-controller#194
fluxcd/notification-controller#195

Image automation already has a similar feature implemented so we should try to make this solution as similar to the existing one as possible.
https://fluxcd.io/docs/guides/image-update/#configure-the-commit-message

Proposal

Looking around there is already a project that solves this type of problem. Alertmanager allows notification templating for individual alerts using go templating, and I don't see a reason why we should re invent the wheel. Another solution which would extend its functionality would be to use Sprig which would add functionality that Helm has.
https://prometheus.io/docs/alerting/latest/notification_examples/
http://masterminds.github.io/sprig/

If we are going to expose templates to the end user we need to decide and document the parameters from the Event which should be available for the end user. It will be difficult to change names or remove parameters once the initial version is released so this should also be the time to make any changes to the event metadata.
Given that there is already a feature request to implement Sprig in Image Automation I expect that people would request it in the Notification Controller if it is not initially included.
fluxcd/image-automation-controller/issues/160

Metadatata

Currently chat notifications will for the most part use the events InvolvedObjects Kind, Name, and Namespace as the authors name. Then it will take the whole message from the event and use it as the message.

The git status notifications are a bit different as they do not use the message at all. This is mostly because there are character limits imposed by the Git providers. Git providers have two fields a key (also called genre, or context) and then a description. This could in theory be considered the same as an author and message. Currently the key is created from the InvolvedObject Kind and Name while the description is created from the event Reason. On top of this the commit status notification relies on the revision metadata in the event which is only set by the Kustomize Controller right now.

Given these two use cases which should at minimum expose the following parameters from the event to be used in the template.

• InvolvedObject
  • Kind
  • Namespace
  • Name
• Message
• Reason
• Revision

Configuration

The chat and commit status notification may look different but I think there are enough similarities to allow for a common set of configuration which could be used in both. My suggestion is to add two new parameters to the alert object. An author template and a message template. For chat notifications these are self explanatory and for commit status notifications the author is used for the key and message is used for the description.

The yaml could look something like this.

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: webapp
  namespace: default
spec:
  providerRef: 
    name: on-call-slack
  eventSeverity: info
  eventSources:
    - kind: GitRepository
      name: webapp
  template:
    author:  {{ .InvolvedObject.Kind }}/{{ .InvolvedObject.Name }}
    message: |
      Some extra text {{ . Revision }}

Of course the default behavior should be used if the template is not specified to keep backwards compatibility.

Extra Features

• Ability to add additional parameters from a ConfigMap. This could be useful if users want to include the cluster name or environment in the notification without having to modify the alert object for each environment.

[stefanprodan]

I propose we introduce a new API type called CommitStatus so that in v1beta2 the Alert type will only target chat providers and event hubs.

[phillebaba]

After some discussions I agree that the best way forward is to create a new CRD and move the git providers to it. During the last year the features in the chat and commit status providers have moved further away forcing a compromise solution in the Alert resource which is specific to neither. For backwards compatibility until v1beta2 is released the Alert will still have to support both provider types.

The original chat providers will remain with the Alert resource. Currently there is a summary field under .spec which will have to me moved to under .spec.template. For backwards compatibility both fields should be available and the field that is set should be used. If both are set the resource should error. Additionally i think we should add the ability to read extra template parameters from a configmap to make it easier to inject the environment name or cluster name.

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Alert
metadata:
  name: webapp
  namespace: default
spec:
  providerRef: 
    name: on-call-slack
  eventSeverity: info
  eventSources:
    - kind: GitRepository
      name: webapp
  template:
    author:  {{ .InvolvedObject.Kind }}/{{ .InvolvedObject.Name }}
    summary: Short description
    message: |
      Some extra text {{ . Revision }}
    parameters:
      - kind: ConfigMap
        key: foo

The new CommitStatus CRD will look very similar to the Alert CRD, it will only have slightly different parameters in its template field. Currently the only controller that sends events containing revision in the metadata is the kustomize-controller, it might be good to limit the eventSource kind for now. There are also undocumented length limits in the status descriptions in all git providers which we need to figure out what they are and protect the user from creating descriptions that are to large by mistake.

apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: CommitStatus
metadata:
  name: webapp
  namespace: default
spec:
  providerRef: 
    name: github
  eventSeverity: info
  eventSources:
    - kind: Kustomization
      name: webapp
  template:
    context/key/sender:  {{ .InvolvedObject.Kind }}/{{ .InvolvedObject.Name }}
    description: {{ .Summary }}
    parameters:
      - kind: ConfigMap
        key: foo

I am still a bit unsure of the template field name, maybe there is a better name for it.

[dominikdrzewiecki-tomtom]

Any chance of getting this in fluxcd anytime soon? I'd rather pick fluxcd over argocd as our gitops solution ;)

[bcarrier92]

I'd love to help out here. I plan to get started on this within the week.

[xjxtree]

suggest support customizing Generic webhook provider's playload(post body data).
here is a example : https://github.com/resmoio/kubernetes-event-exporter/blob/master/README.md#customizing-payload