[Alertmanager] Alertmanager (standalone chart) - unsure if it supports secrets instead of configmaps for alertmanager.yaml

[crackedupcorson]

Describe the bug a clear and concise description of what the bug is.

Hi.

I'm trying to setup a separate alertmanager using this helm chart.
We have an existing alertmanager deployed with kube-prometheus-stack, but want to deploy a second alertmanager (so we can keep the alertmanager.yaml configs separate for a second team).

I don't believe this chart supports using a secret for storing the alertmanager.yaml (or at least it isn't explained well).
If it is supported, the templates section within the configmap isn't well described/documented/visualised on the status page

What's your helm version?

3.9.4

What's your kubectl version?

1.23.0

Which chart?

alertmanager

What's the chart version?

0.25.0

What happened?

kube-prometheus-stack alertmanager supports using an existing secret, but it appears that the standalone alertmanager uses a configmap instead.
https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml#L507
As a workaround I had tried loading the secret directly as an "extraSecretMount", and load a template from the /etc/secrets path that it provides.

However, it appears to just load it as a template, but it doesn't make it clear if that template (which contains routes and receivers) is actually active or will be used? I read here: https://prometheus.io/blog/2016/03/03/custom-alertmanager-templates/ but it's still not clear if the alerting config contains the routes and receivers I've added using the secret

What you expected to happen?

I would expect the standalone helm chart to be somewhat similar to the kube-prometheus-stack, or at least be compatible with secrets for storing the alertmanager.yaml.
It contains slack webhooks and pagerduty routing keys, so I would consider those secure values that shouldn't be in a configmap (or at least not stored in git directly)

Using imagePullSecrets on its own does not seem to add the necessary config either

How to reproduce it?

By using the values.yaml I have supplied, along with creating a secret containing alertmanager.yaml

Enter the changed values of values.yaml?

Overview

alertmanager:
image:
repository: quay.io/prometheus/alertmanager
pullPolicy: IfNotPresent
#tag: ""
config:
templates:
- '/etc/secrets/alertmanager.yaml'
resources:
limits:
memory: "150Mi"
cpu: "0.5"
requests:
memory: "100Mi"
extraSecretMounts:
- name: alertmanager-config-secrets
mountPath: /etc/secrets
subPath: ""
secretName: alertmanager-config
readOnly: false
imagePullSecrets:
- name: alertmanager-config

---

ALSO LISTED IS THE SECRET WE USE TO STORE THESE VALUES

apiVersion: v1
data:
alertmanager.yaml: redacted
api_url: redacted
dev_slack_url: redacted
pagerduty_routing_key: redacted
kind: Secret
metadata:
creationTimestamp: "2023-01-26T16:08:46Z"
name: alertmanager-config
namespace: alertmanager
ownerReferences:

• apiVersion: kubernetes-client.io/v1
  controller: true
  kind: ExternalSecret
  name: alertmanager-config
  uid: redacted
  resourceVersion: "232633145"
  uid: redacted

---

LISTED BELOW IS THE ALERTMANAGER.YAML LOADED FROM SECRET

global:
resolve_timeout: 5m
route:
receiver: 'slack'
repeat_interval: 1h
group_interval: 10m
repeat_interval: 1h
group_interval: 10m
group_wait: 1m
group_by: [prom_server, alertname]
routes:

• match_re:
  alertname: Watchdog
  receiver: 'null'
  continue: false
• match:
  severity: warning
  owner: engineers
  receiver: 'slack'
• match:
  severity: warning
  owner: operations
  receiver: 'dev-slack'
• match:
  severity: critical
  owner: operations
  receiver: 'dev-slack'
• match:
  pagerduty: page
  receiver: 'pagerduty'
  receivers:
• name: 'dev-slack'
  slack_configs:
  • channel: '#redacted'
    send_resolved: true
    icon_emoji: ':fire:'
    api_url: https://hooks.slack.com/services/redacted
    text: redacted
• name: 'pagerduty'
  pagerduty_configs:
  • routing_key: redacted
    send_resolved: true
    description: "{{ range .Alerts }}{{ .Annotations.summary }} {{ end }}"
    details:
    redacted

---

Enter the command that you execute and failing/misfunctioning.

No command is failing, I think it's missing functionality for storing alertmanager.yaml in a secret (or docs that explain this better)

Anything else we need to know?

Here is the alertmanager config from the status page

global:
resolve_timeout: 5m
http_config:
follow_redirects: true
enable_http2: true
smtp_hello: localhost
smtp_require_tls: true
pagerduty_url: https://events.pagerduty.com/v2/enqueue
opsgenie_api_url: https://api.opsgenie.com/
wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
telegram_api_url: https://api.telegram.org
webex_api_url: https://webexapis.com/v1/messages
route:
receiver: default-receiver
continue: false
group_wait: 10s
group_interval: 5m
repeat_interval: 3h
receivers:

• name: default-receiver
  templates:
• /etc/secrets/alertmanager.yaml

alertmanager.yaml is populated and has a global, route and receivers sections - does it override the values shown here?
It doesn't show it on the status page, which I think it should

[zeritti]

kube-prometheus-stack alertmanager supports using an existing secret, but it appears that the standalone alertmanager uses a configmap instead.

For the former, indeed, Prometheus operator supports alertmanagers.spec.configSecret and the chart makes it available. For the latter, as mentioned, the chart creates a configmap out of the value config. There is currently no support for providing the configuration file from an existing secret.

As a workaround I had tried loading the secret directly as an "extraSecretMount", and load a template from the /etc/secrets path that it provides.

Regarding the value extraSecretMounts, these mounts/files will be made available to alertmanager but one has to tell alertmanager how to use them.

Apart from using it for a custom notification template as shown above, as an example, when using opsgenie, one has to specify an api key. This is a secret and hence, one might prefer using global.opsgenie_api_key_file over global.opsgenie_api_key. The file would come from a secret mounted through extraSecretMounts as the mount path and the key are known in advance. Using these secrets depends on what alertmanager can do with the resulting files - any file-supporting configuration parameter could refer to such a file (Ref.).

alertmanager.yaml is populated and has a global, route and receivers sections - does it override the values shown here?

I understand you mean /etc/secrets/alertmanager.yaml from secret alertmanager-config-secrets and included underneath global.templates. The file will not be used as a configuration file, will be used as a custom template for notifications only (Ref.).

[crackedupcorson]

Thanks for the detailed response, it's a lot clearer now.

The file would come from a secret mounted through extraSecretMounts as the mount path and the key are known in advance.

I'll try this today and see if it works for our usecase.

I understand you mean /etc/secrets/alertmanager.yaml from secret alertmanager-config-secrets and included underneath global.templates. The file will not be used as a configuration file, will be used as a custom template for notifications only (Ref.).

Upon reading it again, it makes perfect sense. Amazing what I night of sleep can do.
So I can populate the values of the configmap from the secret I load in. I'll go back and make those changes today, thanks!

[jeremych1000]

I'm running into the same problem. I'm trying to configure a webhook to send to bigpanda, not directly to slack.

Due to the nature of my CICD setup I cannot have another wrapper script that adds injects the secret at helm compile time.

I have a secret in the Kubernetes cluster.

How do I get alertmanager to read this? The config file supports filepath config for opsgenie/slack etc., but not for a generic webhook?

[zeritti]

How do I get alertmanager to read this? The config file supports filepath config for opsgenie/slack etc., but not for a generic webhook?

That is correct. At the moment, alertmanager supports reading some parameters' values from files. The topic is known in the alertmanager project, see e.g. issues #3108, #2498.

[jeremych1000]

How do I get alertmanager to read this? The config file supports filepath config for opsgenie/slack etc., but not for a generic webhook?

That is correct. At the moment, alertmanager supports reading some parameters' values from files. The topic is known in the alertmanager project, see e.g. issues #3108, #2498.

Thanks - understood. I'll use an alternative method of templating instead then.

Namely, external-secrets templating out a Kubernetes secret with the alertmanager config, and then getting alertmanager to read this secret instead (as a full config).

Will revert back if that doesn't work.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[spcaipers-arm]

I am facing similar issue, I dont know how to workaround it:

config:
      global:
        slack_api_url: "MY_API"
      receivers:
      - name: "slack-notifications"
        slack_configs:
        - channel: "#my-channels"
          icon_url: "https://avatars3.githubusercontent.com/u/3380462"
          title: |-
                [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}
          text: >-
            {{ range .Alerts }}
              {{ .Annotations.description }}
            {{ end }}
            PVC: {{ .CommonLabels.persistentvolumeclaim }}
          send_resolved: true
      route:
        receiver: 'slack-notifications'

It deploys as a secret, but the autocreated gz secret which is mounted into alertmanager pod has not been updated.
I am using kube-prometheus-stack chart.

[strainovic]

I've faced the same issue as @spcaipers-arm described. Unable to set alertmanager, only default value is loaded.

[strainovic]

I've managed to provide a custom configuration @spcaipers-arm. Please try this:

config:
      global:
        slack_api_url: "MY_API"
      receivers:
      - name: "slack-notifications"
        slack_configs:
        - channel: "#my-channels"
          icon_url: "https://avatars3.githubusercontent.com/u/3380462"
          title: |-
                [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }}
          text: >-
            {{ range .Alerts }}
              {{ .Annotations.description }}
            {{ end }}
            PVC: {{ .CommonLabels.persistentvolumeclaim }}
          send_resolved: true
      route:
        receiver: 'slack-notifications'
        routes:
        - matchers:
          - alertname =~ "InfoInhibitor|Watchdog"
          receiver: 'slack-notifications'

I added:

        routes:
        - matchers:
          - alertname =~ "InfoInhibitor|Watchdog"
          receiver: 'slack-notifications'

Also, you need this:

  alertmanagerSpec:
    configSecret: "alertmanager-monitoring-kube-prometheus-alertmanager"

I hope it will work for you as well!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Any further update will cause the issue/pull request to no longer be considered stale. Thank you for your contributions.

[stale]

This issue is being automatically closed due to inactivity.