You can configure webhook URLs for alert ingestion. To do this, you must create an alert ingestion service and an alert ingestion application in the Log Service console. Then, Log Service can receive alerts from external monitoring systems.

Step 1: Create an alert ingestion service and an alert ingestion application

  1. Log on to the Log Service console.
  2. Go to the Alert Ingestion page.
    1. In the Projects section, click a project.
    2. In the left-side navigation pane, click Alerts.
    3. Choose Alert Management > Alert Ingestion.
  3. Create an alert ingestion service.
    1. Click Create.
    2. In the Add Service dialog box, specify the ID and name of the service, and click Save.
  4. Create an alert ingestion application.
    1. Click Application in the Actions column of the service.
    2. In the Application Management dialog box, click Create.
    3. In the Add Application dialog box, set the parameters and click Save. The following table describes the parameters.
      Parameter Description
      ID The ID of the alert ingestion application.
      Name The name of the alert ingestion application.
      Protocol Verification
      Protocol The format of alerts. Valid values:
      • AlertManager: receives Alertmanager alerts.
      • Prometheus: receives Prometheus alerts.
      • Zabbix: receives Zabbix alerts.
      • Grafana: receives Grafana alerts.
      • Datadog: receives Datadog alerts.
      • CloudMonitor: receives CloudMonitor alerts.
      • Loki: receives Loki alerts.
      Alert Policy The alert policy that is used to merge, silence, and suppress alerts.
      • If you select Simple Mode or Standard Mode, you do not need to configure alert policies. By default, Log Service uses the sls.builtin.dynamic alert policy to manage alerts.
      • If you select Advanced Mode, you can select a built-in or custom alert policy to manage alerts. For information about how to create an alert policy, see Create an alert policy.
      Action Policy The action policy that is used to manage alert notification methods and the frequency at which alert notifications are sent.
      • If you set Alert Policy to Simple Mode, you need only to configure an action group.
        After you configure an action group, Log Service creates an action policy named Rule name-Action policy. Alert notifications are sent based on the action policy for all alerts that are triggered based on the alert monitoring rule. For more information, see Notification methods.
        Notice You can modify the settings of an action policy on the Action Policy tab. For more information, see Create an action policy. If you add evaluation when you modify an action policy, the value of the Alert Policy parameter is automatically changed to Standard Mode.
      • If you set Alert Policy to Standard Mode or Advanced Mode, you can select a built-in or custom action policy to send alert notifications. For information about how to create an action policy, see Create an action policy.

        If you set Alert Policy to Advanced Mode, you can enable or disable Custom Action Policy. For more information, see Dynamic action policy mechanism.

      Cycle If duplicate alerts are triggered in the specified cycle, the action policy that you select is executed only once, and only one alert notification is sent.
      Whitelist If you turn on the Whitelist switch, you must specify an AccessKey ID. Only the webhook URL that contains a valid AccessKey ID can be used to access alerts.
      For example, if you specify an AccessKey ID as AEDC****ERT and set the {ACCESS_KEY_ID} variable of the path_prefix parameter in Prometheus to AEDC****ERT, the related alerts can be ingested into the alerting system of Log Service.
      Notice You must grant the AliyunLogOpenEventWrite permission to the RAM user to which the AccessKey pair belongs. For more information, see Appendix: Obtain an AccessKey ID.
      Filter
      Filter by Keyword The alert ingestion system ingests only the alerts that contain one or more specified keywords.
      Enrichment
      Add Label Add labels to alerts. Labels are formatted in key-value pairs. For example, you can set the key of a label to Environment and set the value to Staging environment. For more information, see Labels.
      Add Annotation Add annotations to alerts. Annotations are formatted in key-value pairs. For example, you can set the key of an annotation to title and set the value to Prometheus alert. For more information, see Annotations.

      If you set the Protocol parameter to CloudMonitor, Log Service automatically adds the __user_language__ annotation based on the language of the console. The valid values of the annotation are en and cn. The value en indicates English and the value cn indicates Chinese.

      Notice When you set the Add Annotation parameter, you cannot reference alert template variables.
      Quota
      Peak Requests The maximum number of alerts that the alert ingestion application can receive per minute. When the threshold is reached, no more alerts can be ingested. Valid values: 100 to 10000. Unit: requests per minute.

Step 2: Obtain webhook URLs

When you configure parameters in Prometheus or Grafana, you must use the related webhook URL.

  1. In the Application Management dialog box, click Webhook URLs of the alert ingestion application.
  2. In the Webhook URLs dialog box, select the region where alerts are ingested.
    Note If your Prometheus or Grafana is deployed on an Elastic Compute Service (ECS) instance, we recommend that you select the region where the ECS instance resides and use an internal endpoint that can be accessed over a LAN or a virtual private cloud (VPC). You can also use the public endpoint that can be accessed over the Internet in a region.
  3. Move the pointer over the blank area next to the text box and copy the full URL, domain name, or subpath of the webhook URL.

    Replace the {ACCESS_KEY_ID} variable in a webhook URL with the AccessKey ID of the AccessKey pair that you want to use. For information about how to obtain an AccessKey ID, see Appendix: Obtain an AccessKey ID.

    Webhook URLs

Appendix: Obtain an AccessKey ID

To ensure the security of your Alibaba Cloud account, we recommend that you use a RAM user rather than an Alibaba Cloud account to ingest alerts. Before you can use a RAM user to ingest alerts, you must grant the AliyunLogPutOpenEventPolicy permission to the RAM user. To obtain an AccessKey ID, perform the following steps:

  1. Create a RAM user. For more information, see Create a RAM user.
  2. Grant the AliyunLogPutOpenEventPolicy permission to the RAM user. For more information, see Grant permissions to a RAM user.
  3. Create an AccessKey pair that includes an AccessKey ID for the RAM user. For more information, see Create an AccessKey pair for a RAM user.

What to do next