Determine user for notification based on alert data

[nixikanius]

Some incidents occurring in my company cannot be normally pre-assigned to a specific user in the escalation chain. There are 2 examples:

1. Missed call notification for B2B sales. We want to:
   • open an incident for a specific user who missed a call based on alert data from the PBX (username, email).
   • escalate an incident to the head of the sales team if no reaction after some time (e.g. 1h).
2. Project budget control. We want to open an incident if project logged hours have exceeded the budget. The incident should be assigned to the project manager determined by alert data from the PM system.

I understand that these examples not so much from the technical, but from the business domain. However, I would very much like to use Grafana OnCall pretty features to manage alerting not only for the technical team.

Now it looks like there are no suitable feature to solve my problem. I could use direct paging, but:

1. It does not have a public API, only internal exists.
2. The internal API identifies users by their Grafana ID, when we do not have them in alert source systems (only username or email is possible to have).
3. It's not possible to use escalation chains with direct paging, which we want to use for missed call notifications.

Checked Open Source Grafana OnCall v1.2.39.

Tasks

Preview Give feedback

No tasks being tracked yet.

[Matvey-Kuk]

Hi! This use-case should be able to fulfill using OnCall. You don't need triggering "Direct Paging" using API, you could raise Alert Groups using "Formatted Webhook": https://grafana.com/docs/oncall/latest/integrations/webhook/ . It will register new Alert Group & you will be able to use routes to route to a specific person: https://grafana.com/docs/oncall/latest/escalation-chains-and-routes/

[nixikanius]

@Matvey-Kuk thank you for the reply. As I understand correctly, in the solution you suggested we need to create individual routes for each user we need to notify. So we need to create a personal route each time we add a new user to Grafana (we talk about 50+ users who change regularly): this is not as convenient as a single rule that will identify the user, for example, by his email.

[Matvey-Kuk]

@nixikanius OnCall agree, this use-case is not what the tool was designed to. Routes are mostly designed to route to teams by labels, who exactly will be notified is usually managed by on-call rotation, or round robin, or other tools provided by escalation chains. If you want to achieve 1:1 routing label:user, looks like you are doing user mapping at the upstream which is not a typical use-case. If you still would like to use OnCall for this, I suggest leveraging API: https://grafana.com/docs/oncall/latest/oncall-api-reference/routes/

[mderynck]

If we were to implement this feature we could create an escalation step that uses a jinja template to figure out who to page. This has some other applications like controlling for logic not encapsulated by a schedule.