X-Pack Watcher is a monitoring and alerting service developed for Elasticsearch. If you configure X-Pack Watcher for your cluster, X-Pack Watcher can trigger actions when specific conditions are met. For example, if the logs index contains errors, X-Pack Watcher triggers the system to send alert notifications by using emails or DingTalk messages. This topic describes how to configure a DingTalk chatbot to receive alert notifications from X-Pack Watcher.

Background information

X-Pack Watcher allows you to create watches. A watch consists of a trigger, an input, a condition, and actions.
  • Trigger

    Determines when a watch starts to run. You must configure a trigger for each watch. X-Pack Watcher allows you to create various types of triggers. For more information, see Schedule Trigger.

  • Input

    Loads data to the payload of a watch. Inputs are used as filters to match the specified type of index data. For more information, see Inputs.

  • Condition

    Controls whether a watch performs actions.

  • Actions

    Determines the actions that a watch performs when the specified condition is met. In this example, the webhook action is used.

Prerequisites

  • A single-zone Alibaba Cloud Elasticsearch cluster is created.
    For more information, see Create an Alibaba Cloud Elasticsearch cluster.
    Note In the original network architecture, X-Pack Watcher is available only for single-zone Elasticsearch clusters. In the new network architecture, X-Pack Watcher is available for both single-zone Elasticsearch clusters and multi-zone Elasticsearch clusters.
  • X-Pack Watcher is enabled for the Elasticsearch cluster. By default, X-Pack Watcher is disabled.

    For more information, see Configure the YML file.

  • An Elastic Compute Service (ECS) instance is created in your virtual private cloud (VPC), and the required applications are deployed on the ECS instance. For more information, see Create an instance by using the wizard.
    Note
    • The ECS instance is used as a backend server to receive requests that are forwarded by a Server Load Balancer (SLB) instance. The ECS instance can be deployed in a zone that is different from the SLB instance but must be deployed in the same VPC and region as the SLB instance.
    • X-Pack Watcher cannot directly access the Internet. It must use the internal endpoint of your Elasticsearch cluster to access the Internet. Therefore, you must configure source network address translation (SNAT) or an elastic IP address for an ECS instance that is deployed in a VPC and use the ECS instance as a proxy to forward requests.

Precautions

Since October 2020, the network architecture of Alibaba Cloud Elasticsearch in different regions has been adjusted. The adjustment has the following impacts on clusters:

  • Clusters that are created before October 2020 are deployed in the original network architecture. In this architecture, clusters are deployed in the VPCs that are created by users. If you want a cluster that is deployed in this architecture to access the Internet, you can use an ECS instance that is configured with SNAT or use an NGINX proxy to forward requests.
  • Clusters that are created in October 2020 or later are deployed in the new network architecture. If you want to use X-Pack Watcher for an Elasticsearch cluster that is created in October 2020 or later, you must first use the PrivateLink service to establish private connections between VPCs. For more information, see Configure a private connection for an Elasticsearch cluster. If you want a cluster that is deployed in the new network architecture to access the Internet, configure an NGINX proxy to forward requests.

Procedure

  1. Configure a private connection to the Elasticsearch cluster and obtain the domain name of the related endpoint. The domain name is used to access external services.
    For more information, see Configure a private connection for an Elasticsearch cluster.
    Note This step is required only for a cluster that is deployed in the new network architecture.
  2. Configure a security group rule for the ECS instance.
    1. Log on to the ECS console.
    2. In the left-side navigation pane, click Instances.
    3. On the Instances page, find the ECS instance and choose More > Network and Security Group > Configure Security Group in the Actions column.
    4. On the Security Groups tab, find your security group and click Add Rules in the Actions column.
    5. On the Inbound tab, click Add Rule.
    6. Configure parameters.
      Add a security group rule
      Parameter Description
      Action Select Allow.
      Priority Retain the default value.
      Protocol Type Select Custom TCP.
      Port Range Set this parameter to the port that you frequently use. If you want to configure an NGINX proxy, you must configure this parameter. In this example, port 8080 is used.
      Authorization Object Enter the IP addresses of all the nodes in the Elasticsearch cluster.
      Note For more information about how to obtain the IP addresses of the nodes, see View the basic information of nodes.
      Description The description of the rule.
    7. Click Save.
  3. Configure an NGINX proxy.
    1. Install NGINX on the ECS instance.
    2. Configure the nginx.conf file.
      Replace the server configuration in the nginx.conf file with the following code. Server configuration in the nginx.conf file 
      server
  {
    listen 8080;# Listening port
    server_name localhost;# Domain name
    index index.html index.htm index.php;
    root /usr/local/webserver/nginx/html;# Website directory
      location ~ .*\.(php|php5)?$
    {
      #fastcgi_pass unix:/tmp/php-cgi.sock;
      fastcgi_pass 127.0.0.1:9000;
      fastcgi_index index.php;
      include fastcgi.conf;
    }
    location ~ .*\.(gif|jpg|jpeg|png|bmp|swf|ico)$
    {
      expires 30d;
      # access_log off;
    }
    location / {
      proxy_pass <Webhook URL of the DingTalk chatbot>;
    }
    location ~ .*\.(js|css)?$
    {
      expires 15d;
      # access_log off;
    }
    access_log off;
  }
      Replace <Webhook URL of the DingTalk chatbot> with the webhook URL of the DingTalk chatbot that you configured to receive alert notifications.
      Note To obtain the webhook URL of the DingTalk chatbot, create an alert group in DingTalk. In the upper-right corner of the DingTalk group, click the Group Settings icon. In the Group Settings panel, click Group Assistant. In the Group Assistant panel, click Add Robot. In the ChatBot dialog box, click the Add icon on the right side of Add Robot to add a chatbot that you can access by using a webhook. Then, you can view the webhook URL of the DingTalk chatbot.
    3. Reload the NGINX configuration file and restart NGINX. 
      /usr/local/webserver/nginx/sbin/nginx -s reload            # Reload the NGINX configuration file.
/usr/local/webserver/nginx/sbin/nginx -s reopen            # Restart NGINX.
  4. Create a watch for alerting.
    1. Log on to the Kibana console of the Elasticsearch cluster.
      Note For more information, see Log on to the Kibana console.
    2. In the left-side navigation pane, click Dev Tools.
    3. On the Console tab of the page that appears, run the following command to create a watch.
      In this example, a watch named log_error_watch is created to search the logs index for errors every 10 seconds. If more than 0 errors are found, an alert is triggered. 
      PUT _xpack/watcher/watch/log_error_watch
{
  "trigger": {
    "schedule": {
      "interval": "10s"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": ["logs"],
        "body": {
          "query": {
            "match": {
              "message": "error"
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "actions" : {
  "test_issue" : {
    "webhook" : {
      "method" : "POST",
      "url" : "http://<yourAddress>:8080",
      "body" : "{\"msgtype\": \"text\", \"text\": { \"content\": \"An error is found. Handle the error immediately.\"}}"
    }
  }
}
}
      Table 1. Parameters
      Parameter Network architecture type Configuration object Description
      <yourAddress> New network architecture Domain name of the endpoint In the new network architecture, private connections need to be established between VPCs, and the domain name of the related endpoint is used to forward requests.
      Original network architecture IP address of the NGINX proxy The NGINX proxy in the same VPC as the Elasticsearch cluster is used to forward requests over the Internet.
      Webhook URL of the DingTalk chatbot The SNAT feature must be enabled. This feature enables an ECS instance in a VPC to access the Internet if no public IP address is associated with the ECS instance.
      Notice
      • If the error No handler found for uri [/_xpack/watcher/watch/log_error_watch_2] and method [PUT] is returned after you run the preceding command, X-Pack Watcher is disabled for the Elasticsearch cluster. In this case, enable X-Pack Watcher and run the command again. For more information, see Configure the YML file.
      • When you create a DingTalk chatbot, you must configure security settings. This is because the body parameter in the preceding code must be specified based on the security settings. For more information, see Configure security settings. In this example, Security Settings is set to Custom Keywords and the error keyword is specified. In this case, the DingTalk chatbot sends alert notifications only if the content field in the body parameter contains error.
      If you no longer require this watch, you can run the following command to delete the watch: 
      DELETE _xpack/watcher/watch/log_error_watch