Setting Up Webhook Channels for Sending Alert Notifications

This section describes how to set up webhooks in your CoPilot system so they can be selected as a notification channel when configuring user alerts. The section provides the following information:

  • How to specify webhook payload URLs in your CoPilot system.

  • How to customize the webhooks Aviatrix CoPilot generates for sending to external systems (such as Slack and PagerDuty). See CoPilot Webhooks Customization.

  • An example for setting up PagerDuty to receive CoPilot alerts via webhooks. See CoPilot Webhooks Customization.

Specifying the Webhook Payload URL

This section describes how to set up the webhook payload URLs in your CoPilot system so they can be selected as a notification channel when configuring user alerts.

  1. Go to Monitor > Notifications > Recipients tab and click Webhooks to open the Webhooks pane.

  2. Click the + Webhook button.

  3. For Alias, enter the name or alias of the person or team who will receive the alert notification.

  4. For Webhook Payload URL, type the URL of the server who will receive the webhook POST request of the alert notification.

  5. (Optional) Set an authorization header. Passing a custom authorization header may be used for integrating with downstream third-party alerting systems. When set, the contents are included in the header of the webhook request as:

    Authorization: <value typed in Authorization Header field>

  6. (Optional) Set a secret for use in authenticating CoPilot as the sender of the POST requests.

  7. (Optional) Set a tag or label to be sent in the payload for use to integrate with your system.

  8. Click Save.

    The webhook payload URL is saved and is now available for selection when configuring CoPilot user alerts.

Testing a CoPilot Webhook

To test a CoPilot webhook:

  1. Go to CoPilot > Monitor > Notifications > Recipients.

  2. Click the Webhooks tab.

  3. In the table, locate the CoPilot webhook you previously created that you want to test.

  4. Click the vertical ellipsis and select Test Webhook.

  5. The webhook is tested with an example alert to check that the webhook is configured correctly. Note that CoPilot does not simulate a real condition for this test.

    Webhook test control

CoPilot Webhooks Customization

You can customize the webhooks Aviatrix CoPilot generates for sending to external systems (such as Slack) by using the Handlebars templating language. Examples are provided in this topic for high level variables that are exposed in CoPilot notification alerts.

Variables in CoPilot alerts used in webhook customization

CoPilot alerts expose the following high level variables (objects):

  • alert

  • event

  • webhook

Each object exposes additional variables that can be accessed.

Alert

The alert object exposes :

"alert": {
  "closed": false,
  "metric": "CPU Utilization",
  "name": "High CPU Usage",
  "status": "OPEN",
  "threshold": 80,
  "unit": "%"
}

Event

The event object exposes :

"event": {
  "receiveSeparateAlert": false,
  "exceededOrDropped": "Exceeded",
  "newlyAffectedHosts": ["spoke1", "spoke1-hagw"],
  "recoveredHosts": ["spoke2"],
  "message": "Alert Updated",
  "timestamp": "2021-05-22T17:49:20.547Z"
}

where:

  • newlyAffectedHosts represents the hosts that are now affected but were not affected before. These hosts usually need the user’s attention the most.

  • recoveredHosts represents the hosts that are now recovered.

  • receiveSeparateAlert is for individual host alerts.

Webhook

The webhook object exposes :

"webhook": {
  "name": "",
  "secret": "",
  "tags": [],
  "url": ""
}

Creating a custom webhook and accessing individual fields

Example 1: If individual alerts for hosts is ON, receive a string. Else receive an array. :

{
  "status": "{{#if alert.closed}}ok{{else}}critical{{/if}}",
  "check": {{alert.name}},
  "copilotstatus": {{alert.status}},
  "host": {{#if event.receiveSeparateAlert}}
  {{#if event.newlyAffectedHosts}}
    {{event.newlyAffectedHosts.[0]}}
  {{else}}
    {{event.recoveredHosts.[0]}}
  {{/if}}
{{else}}
  {{#if event.newlyAffectedHosts}}
    {{event.newlyAffectedHosts}}
  {{else}}
    {{event.recoveredHosts}}
  {{/if}}
{{/if}},
  "alert_timestamp": "Received <<alert.metric>> at <<event.timestamp>>"
}

Example 2 :

{
  "myAlert": {{alert.name}},
  "triggeredAt": {{event.timestamp}},
  "eventMessage": {{event.message}},
  "triggeredMetric": {{alert.metric}},
  "status": {{alert.status}},
  "webHookName": {{webhook.name}},
  "webHookTags": {{webhook.tags}}
}​

Output: :

{
  "myAlert": "High CPU Usage",
  "triggeredAt": "2021-05-22T18:06:34.143Z",
  "eventMessage": "Alert Updated",
  "triggeredMetric": "CPU Utilization",
  "status": "OPEN",
  "webHookName": "test",
  "webHookTags": [
    "customTag"
  ]
}​

Templates support JSON and String formatted output as values.

In situations where you want to specifically format the value of an output, it needs to be converted from JSON (default) to a string value.

“webhook”: \{{webhook}}→ produces JSON: :

{
  "webhook": {
    "name": "",
    "secret": "",
    "tags": [
      "test",
      "123",
      "emergency"
    ],
    "url": ""
  }
}​

“webhook”: “[webhook]” → produces STRING: :

{
  "webhook": "{\n  \"name\": \"\",\n  \"secret\": \"\",\n  \"tags\": [\n    \"test\",\n    \"123\",\n    \"emergency\"\n  ],\n  \"url\": \"\"\n}"
}

String escaped values allow for custom messages to be used in values. :

{
  "webhook": "My Custom Webhook message <<webhook>>"
}

Output: :

{
  "webhook": "My Custom Webhook message {\n  \"name\": \"\",\n  \"secret\": \"\",\n  \"tags\": [\n    \"test\",\n    \"123\",\n    \"emergency\"\n  ],\n  \"url\": \"\"\n}"
}

Looping over lists in templates using #attribute…​ . …​/attribute. Any content between the # and / is expanded once for each list item, and the special attribute . can be used to refer to it.

Some attributes refer to a list of results:

  • webhook.tags list of optional user-defined strings, configured on a per-webhook basis.

  • event.newlyAffectedHosts represents the hosts that are now affected but were not affected before. These hosts usually need the user’s attention the most.

  • event.recoveredHosts represents the hosts that are now recovered.

{
  "webHookTags": {{webhook.tags}}
}

Output: :

{
  "webHookTags": [
    "customTag",
    "Slack",
    "Emergency"
  ]
}

If you want to customize the output for list items: :

{
  "webhook": "<<#webhook.tags>> tag:<<.>> <</webhook.tags>>"
}

Output: :

{
  "webhook": " tag:test  tag:123  tag:emergency "
}

Escaping quotes for return values when creating custom values is performed automatically for strings within << >>.

{{{some_quoted_var}}} disables escapes altogether, which should be avoided, as it can unexpectedly cause embedded strings to form invalid JSON, for example, an alert name of A “great” alert, quotes, newlines, tabs, and so on are not allowed in JSON strings.

Input: :

{
  "alertStatus": "Name:{{{alert.name}}} Metric:{{{alert.metric}}} alert"
}

Output: :

{
  "alertStatus": "Name:High CPU Usage Metric:CPU Utilization alert"
}

Custom Slack Webhook example (slack document: https://app.slack.com/block-kit-builder/): :

{
  "blocks":[
    {
      "type":"header",
      "text":{
        "type":"plain_text",
        "text":":fire:<<alert.name>>:fire:"
      }
    },
    {
      "type":"divider"
    },
    {
      "type":"section",
      "text":{
        "type":"mrkdwn",
        "text":"newly affected hosts:\n <<#event.newlyAffectedHosts>>:arrow_down:<<.>>\n<</event.newlyAffectedHosts>>"
      }
    },
    {
      "type":"actions",
      "elements":[
        {
          "type":"button",
          "text":{
            "type":"plain_text",
            "text":"Confirm",
            "emoji":true
          },
          "value":"click_me_123",
          "action_id":"actionId-0"
        }
      ]
    },
    {
      "type":"section",
      "text":{
        "type":"mrkdwn",
        "text":"status: <<alert.status>>\nthreshold: <<alert.threshold>><<alert.unit>>\ntime: <<event.timestamp>>\nmesssage: <<event.message>>"
      }
    }
  ]
}

Webhook Example

webhookImage

Example: PagerDuty Webhook Payload URL

If you want to set up PagerDuty to receive CoPilot alerts via webhooks, this section provides an overview of the steps, including generating the PagerDuty webhook payload URL to which CoPilot will send POST requests. You specify the URL when you configure CoPilot notifications in Settings > Notifications > Webhooks.

For the most current information about receiving HTTP callbacks in your PagerDuty account, always refer to the PagerDuty support documentation.

Prerequisite: Before you begin, create a PagerDuty developer account at the PagerDuty Developer Platform site.

Summary of Steps:

  • (In PagerDuty) Log in to your PagerDuty Developer Console.

  • (In PagerDuty) Create the PagerDuty application service.

  • (In CoPilot) Configure CoPilot notifications via webhooks to send alerts to your new service.

To set up PagerDuty to receive CoPilot alerts via webhooks:

  1. Log in to your PagerDuty Developer Console and click Create New App.

    If you do not see the Create New App button after logging in, navigate to Integrations > Developer Tools > App Registration from the console menu and click + New App.

    notifs_pager_create_app

  2. Fill in the requested fields, such as App Name and Description, and then select the Events Integration checkbox.

    notifs_pager_build_app

  3. Click Next.

  4. In the Configure Events Integration section, fill in the requested fields:

    • Transform Event Data: Yes

    • Send Change Events: No

  5. Click Register App.

  6. To create a Test Service, navigate to Services > Service Directory and click + New Service.

  7. Enter the name of the service and provide a description.

    notifs pager create test

  8. Click Next to assign an escalation policy on the Assign an Escalation Policy page.

  9. After generating a new escalation policy, or selecting an existing escalation policy, click Next to configure alert grouping and transient alerts on the Reduce Noise page.

  10. After selecting alert grouping and transient alerts options, click Next to configure integrations on the Integrations page.

  11. After selecting the Events API V2 integration, click Create Service.

  12. Navigate to Services > Service Directory.

    pagerduty directory

  13. Select the newly created service and then navigate to Integrations. The existing integration with Events API v2 are displayed.

    pagerduty directory api

  14. Copy the integration URL (Alert Events).

  15. Log into CoPilot and go to Monitor > Notifications > Recipients and click Webhooks.

  16. Click + Webhook to create a new webhook configuration for PagerDuty.

  17. In the Create Webhook Recipient dialog, fill in the necessary fields. For Webhook Payload URL, paste the copied Integration URL (Alert Events).

  18. Click Save.

  19. To test the webhook, click the vertical ellipsis 25 icon and select Test Webhook. If the test message is successful, a Success message displays.

  20. You can check the alert in PagerDuty on the Incidents tab.

Note About Webhooks Migrated to 3.7.1

3.7.1 offers a new Alert UI that provides the ability to configure multi-metric conditions for alerts. Existing webhooks were for single condition alerts.

The Webhook Format contains the following new fields:

  • alert.defId: The id of the alert that sends out the webhook

  • alert.metrics: An array of strings that contains all the metric(s) the alert has. i.e. ['CPU Used (%)', 'Memory Used (%)', 'Disk Free (%)']

  • alert.version: This field is always 'v2'. It’s used to distinguish from the previous webhook format.

  • alert.query: This field is a more detailed version of alert.metrics. For each metric, it also has its duration, comparator, threshold, and unit.

    For example:

    [
              {
                duration: 15,
                comparator: 'more than',
                metric: 'cpu_used_per',
                threshold: 90,
                unit: 'Percent'
              },
              {
                duration: 15,
                comparator: 'more than',
                metric: 'memory_used_per',
                threshold: 90,
                unit: 'Percent'
              },
              {
                duration: 15,
                comparator: 'less than',
                metric: 'hdisk_free_per',
                threshold: 5,
                unit: 'Percent'
              }
            ] ​

  • event.condition: This is a human-readable string that explains when the alert would be triggered.

    For example:

    '(CPU Used (%) more than 90%) OR (Memory Used (%) more than 90%) OR (Disk Free (%) less than 5%) '

Pre-existing webhooks are migrated to contain the new format. Existing webhook high level objects remain the same.

The Webhook Format contains the following modifications on some old fields:

  • alert.metric: If the alert is a multi-metric alert, this field would be "Please refer to alert.query key".

  • event.exceededOrDropped: If the alert is a multi-metric alert, this field would be "Please refer to event.condition or alert.query key".

Any existing webhooks (for single condition alert) created prior to release 3.7.1 will have no changes to existing fields, and only new fields would be added. Newly added multi-metric alerts will have modifications to the two old fields as they are explained above.