Skip to content

Commit

Permalink
Update direct paging docs (#2600)
Browse files Browse the repository at this point in the history
# What this PR does

Updates public docs on direct paging after introducing numerous changes
to the feature.

## Preview


https://github.com/grafana/oncall/assets/20116910/f141e8b1-ac2c-455c-81f8-9c62c4c75234


## Checklist

- [x] Unit, integration, and e2e (if applicable) tests updated
- [x] Documentation added (or `pr:no public docs` PR label added if not
required)
- [x] `CHANGELOG.md` updated (or `pr:no changelog` PR label added if not
required)
  • Loading branch information
vstpme authored Jul 20, 2023
1 parent bf3ea1e commit ed6389c
Show file tree
Hide file tree
Showing 5 changed files with 102 additions and 28 deletions.
4 changes: 4 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## Unreleased

### Changed

- Update direct paging docs by @vadimkerr ([#2600](https://github.com/grafana/oncall/pull/2600))

### Fixed

- Fix Slack direct paging issue when there are more than 100 schedules by @vadimkerr ([#2594](https://github.com/grafana/oncall/pull/2594))
Expand Down
89 changes: 73 additions & 16 deletions docs/sources/integrations/manual/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,27 +9,84 @@ keywords:
- on-call
- Alertmanager
- Prometheus
title: Raising alerts manually
- Direct paging
title: Page people manually
weight: 300
---

# Raising alerts manually
# Page people manually

Sometimes you need to page a specific person (following their preferred notification policy), or need help from people
in some particular team. In that case you can trigger an alert group providing some context information as well as
defining who to notify about it, a user or the person on-call in a given team's schedule.
Grafana OnCall relies on automated and pre-configured workflows, such as [integrations][integrations],
[routes, and escalation chains][escalation-chains-and-routes] to handle most of the incident response process.
However, sometimes you might need to page a [team][manage-teams] or request assistance from specific people that
are not part of these pre-defined rules.

You can create a manual alert group using the "+ Manual alert group" button (in the Alert Groups page), and set
its escalation options to page a specific person or group of people.
For such ad-hoc scenarios, Grafana OnCall allows you to create an alert group, input necessary information, and decide
who will be alerted – a team, a user, or an on-call user from a specific schedule.

> The same feature is also available as **/escalate** slack command.
## Page a team

- You need to define a title for your alert, an optional description, and select the responders which could be a
specific user in your team, a particular schedule, or multiple instances of those.
- When selecting a user, a few checks will be performed before adding them to the list of responders: user should have
a notification policy set, and ideally be on-call.
- If the user is not on-call at the time, you will get alternative users to choose instead from the OnCall schedules
that user is part of. You can still page the original user if you confirm that is what you want.
- When selecting a schedule, the user(s) on-call when the alert is triggered will be notified.
Click on "+ New alert group" on the "Alert groups" page to start creating a new alert group.
From there, you can configure the alert group to notify a particular team and optionally include additional users or
schedules. Here are the inputs you need to fill in:

> **NOTE:** for each responder (user or schedule) you can choose the notification policy to use: default or important.
- **Title**: Write a brief and clear title for your alert group.
- **Message**: Optionally, add a message to provide more details or instructions.
- **Team**: Select the team you want to page. The team's
[direct paging integration](#learn-the-flow-and-handle-warnings) will be used for notification.
- **Additional Responders**: Optionally, include more responders for the alert group.
These could be any combination of users and schedules.
For each additional responder (user or schedule), you can select a notification policy: [default or important][notify].

> The same feature is also available as [**/escalate**][slack-escalate] Slack command.
## Add responders for an existing alert group

If you want to page more people for an existing alert group, you can do so using the "Notify additional responders"
button on the specific alert group's page. Here you can select more users, or choose users who are on-call for specific
schedules. The same functionality is available in Slack using the "Responders" button in the alert group's message.

Notifying additional responders doesn't disrupt or interfere with the escalation chain configured for the alert group;
it simply adds more responders and notifies them immediately.

> It's not possible to page a team for an existing alert group. To page a specific team, you need to
[create a new alert group](#page-a-team).

## Learn the flow and handle warnings

When you pick a team to page, Grafana OnCall will automatically use the right direct paging integration for the team.
"Direct paging" is a special kind of integration in Grafana OnCall that is unique per team and is used to send alerts
to the team's ChatOps channels and start an appropriate escalation chain.

If a team hasn't set up a direct paging integration, or if the integration doesn't have any escalation chains connected,
Grafana OnCall will issue a warning. If this happens, consider
[setting up a direct paging integration](#set-up-direct-paging-for-a-team) for the team
(or reach out to the relevant team and suggest doing so).

## Set up direct paging for a team

To create a direct paging integration for a team, click "+ New alert group" on the "Alert groups" page, choose the team,
and create an alert group, **regardless of any warnings**. This action automatically triggers Grafana OnCall to generate
a [direct paging integration](#learn-the-flow-and-handle-warnings) for the chosen team.

After setting up the integration, you can customize its settings, link it to an escalation chain,
and configure associated ChatOps channels.
To confirm that the integration is functioning as intended, [create a new alert group](#page-a-team)
and select the same team for a test run.

{{% docs/reference %}}
[integrations]: "/docs/oncall/ -> /docs/oncall/<ONCALL VERSION>/integrations"
[integrations]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/integrations"

[escalation-chains-and-routes]: "/docs/oncall/ -> /docs/oncall/<ONCALL VERSION>/escalation-chains-and-routes"
[escalation-chains-and-routes]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/escalation-chains-and-routes"

[notify]: "/docs/oncall/ -> /docs/oncall/<ONCALL VERSION>/notify#configure-user-notification-policies"
[notify]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/notify#configure-user-notification-policies"

[slack-escalate]: "/docs/oncall/ -> /docs/oncall/<ONCALL VERSION>/notify/slack#slack-escalate-command"
[slack-escalate]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/notify/slack#slack-escalate-command"

[manage-teams]: "/docs/oncall/ -> /docs/oncall/<ONCALL VERSION>/user-and-team-management#manage-teams-in-grafana-oncall"
[manage-teams]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/user-and-team-management#manage-teams-in-grafana-oncall"
{{% /docs/reference %}}
25 changes: 17 additions & 8 deletions docs/sources/notify/slack/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,19 +92,25 @@ teams of their on-call shifts. Admins can configure shift notification behavior

The Grafana OnCall Slack app includes helpful message shortcuts and slash commands.

### Slack commands
### Slack `/escalate` command

Use the `/oncall` Slack command to create a new alert group directly from Slack targetting a team and/or route.
Use `/escalate` to page a team (and additional responders) directly from Slack.

1. Type `/oncall` in the message box of the desired Slack channel then click **Send**.
1. Fill out the **Start New Escalation** creation form then click **Submit**.
1. Type `/escalate` in the message box of any Slack channel then click **Send**.
1. Fill out the **Create Alert Group** form then click **Submit**.
1. Once the Grafana OnCall app sends a Slack message with the newly created alert, the alert group is open and firing.

Use the `/escalate` Slack command to create a new alert group directly from Slack and specifically paging a user or
a schedule.
It's also possible to page additional responders for an existing alert group. To do so, use the "Responders" button
in the alert group message. [Learn more about paging people manually.][integrations-manual]

1. Type `/escalate` in the message box of any Slack channel then click **Send**.
1. Fill out the **Create alert group** form then click **Submit**.
### Slack `/oncall` command

> **DEPRECATED: `/oncall` is deprecated and WILL BE REMOVED in a future release. Use `/escalate` instead.**
Use the `/oncall` Slack command to create a new alert group directly from Slack targetting a team and/or route.

1. Type `/oncall` in the message box of the desired Slack channel then click **Send**.
1. Fill out the **Start New Escalation** creation form then click **Submit**.
1. Once the Grafana OnCall app sends a Slack message with the newly created alert, the alert group is open and firing.

### Message shortcuts
Expand All @@ -121,4 +127,7 @@ Use message shortcuts to add resolution notes directly from Slack. Message short
{{% docs/reference %}}
[open-source]: "/docs/oncall/ -> /docs/oncall/<ONCALL VERSION>/open-source"
[open-source]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/open-source"

[integrations-manual]: "/docs/oncall/ -> /docs/oncall/<ONCALL VERSION>/integrations/manual"
[integrations-manual]: "/docs/grafana-cloud/ -> /docs/grafana-cloud/alerting-and-irm/oncall/integrations/manual"
{{% /docs/reference %}}
4 changes: 2 additions & 2 deletions engine/apps/slack/scenarios/paging.py
Original file line number Diff line number Diff line change
Expand Up @@ -557,14 +557,14 @@ def _get_team_select_context(organization, team):
":warning: *Direct paging integration missing*\n"
"The selected team doesn't have a direct paging integration configured and will not be notified. "
"If you proceed with the alert group, an empty direct paging integration will be created automatically for the team. "
"<https://grafana.com/docs/oncall/latest/integrations/manual/|Learn more.>"
"<https://grafana.com/docs/oncall/latest/integrations/manual/#learn-the-flow-and-handle-warnings|Learn more.>"
)
elif not escalation_chains_exist:
context_text = (
":warning: *Direct paging integration not configured*\n"
"The direct paging integration for the selected team has no escalation chains configured. "
"If you proceed with the alert group, the team likely will not be notified. "
"<https://grafana.com/docs/oncall/latest/integrations/manual/|Learn more.>"
"<https://grafana.com/docs/oncall/latest/integrations/manual/#learn-the-flow-and-handle-warnings|Learn more.>"
)
else:
context_text = f"Integration <{alert_receive_channel.web_link}|{alert_receive_channel.verbal_name} ({team_name})> will be used for notification."
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -168,7 +168,9 @@ const ManualAlertGroup: FC<ManualAlertGroupProps> = (props) => {
<br />
If you proceed with the alert group, the team likely will not be notified. <br />
<a
href={'https://grafana.com/docs/oncall/latest/integrations/manual/'}
href={
'https://grafana.com/docs/oncall/latest/integrations/manual/#learn-the-flow-and-handle-warnings'
}
target="_blank"
rel="noreferrer"
className={cx('link')}
Expand All @@ -188,7 +190,9 @@ const ManualAlertGroup: FC<ManualAlertGroupProps> = (props) => {
If you proceed with the alert group, an empty direct paging integration will be created automatically
for the team. <br />
<a
href={'https://grafana.com/docs/oncall/latest/integrations/manual/'}
href={
'https://grafana.com/docs/oncall/latest/integrations/manual/#learn-the-flow-and-handle-warnings'
}
target="_blank"
rel="noreferrer"
className={cx('link')}
Expand Down

0 comments on commit ed6389c

Please sign in to comment.