Alertmanager doesn't template pagerduty service key

[zeeshen]

What did you do?
Send alerts to pagerduty via alertmanager

What did you expect to see?
Send alerts

What did you see instead? Under which circumstances?

level=error ts=2017-12-25T09:11:28.340342692Z caller=dispatch.go:266 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="cancelling notify retry for \"pagerduty\" due to unrecoverable error: unexpected status code 400"

Environment

• System information:

  Linux 3.13.0-48-generic x86_64

• Alertmanager version:

alertmanager, version 0.12.0 (branch: HEAD, revision: fc33cc78036f82ef8d4734c197a96f7cb6c952a3)
  build user:       root@c9169eb10d06
  build date:       20171215-14:13:20
  go version:       go1.9.2

• Alertmanager configuration file:

...
- name: 'pd-application'
  pagerduty_configs:
    - send_resolved: true
      service_key: '{{ template "pagerduty.XXX.app_key" . }}'

• Logs:

level=error ts=2017-12-25T09:11:28.340342692Z caller=dispatch.go:266 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="cancelling notify retry for \"pagerduty\" due to unrecoverable error: unexpected status code 400

I think this 400 is caused by this commit. In pagerduty api v1 implement, the ServiceKey is template-expanded but the new implement does't expand it. Same for RoutingKey field.

And according to docs, this service_key should be a tmpl_secret field. Besides, in pagerduty faq, it's API V1 not Event API v1 that will be deprcated. So many people may still use this ServiceKey field.

[reachlin]

@zeeshen could you tell me what is pagerduty.XXX.app_key and where you define this?

[zeeshen]

@reachlin pagerduty.XXX.app_key is just template that defined in my customer template files.