Configure a DingTalk chatbot to receive alert notifications from X-Pack Watcher

Background information

Prerequisites

• A single-zone Alibaba Cloud Elasticsearch cluster is created.
  For more information, see Create an Alibaba Cloud Elasticsearch cluster.

  Note In the original network architecture, X-Pack Watcher is available only for single-zone Elasticsearch clusters. In the new network architecture, X-Pack Watcher is available for both single-zone Elasticsearch clusters and multi-zone Elasticsearch clusters.
• X-Pack Watcher is enabled for the Elasticsearch cluster. By default, X-Pack Watcher is disabled.

  For more information, see Configure the YML file.

• An Elastic Compute Service (ECS) instance is created in your virtual private cloud (VPC), and the required applications are deployed on the ECS instance.

  For more information, see Create an instance by using the wizard.

  Note

  • When you use PrivateLink to establish private connections between VPCs, the ECS instance is used as a backend server to receive requests that are forwarded by a Server Load Balancer (SLB) instance. The ECS instance can be deployed in a zone that is different from the SLB instance but must be deployed in the same VPC and region as the SLB instance.
  • X-Pack Watcher cannot directly access the Internet. It must use the internal endpoint of your Elasticsearch cluster to access the Internet. In this case, you can enable source network address translation (SNAT) for or associate an elastic IP address (EIP) with an ECS instance that is deployed in a VPC. This way, you can use the ECS instance as a proxy to forward requests.

Precautions

The network architecture of Alibaba Cloud Elasticsearch in different regions has been adjusted since October 2020. The adjustment has the following impacts on clusters:

• Clusters that are created before October 2020 are deployed in the original network architecture. In this architecture, clusters are deployed in the VPCs that are created by users. If you want a cluster that is deployed in this architecture to access the Internet, you can use an ECS instance for which SNAT is enabled or use an NGINX proxy to forward requests.
• Clusters that are created in October 2020 or later are deployed in the new network architecture. If you want to use X-Pack Watcher for an Elasticsearch cluster that is created in October 2020 or later, you must first use the PrivateLink service to establish private connections between VPCs. For more information, see Configure a private connection for an Elasticsearch cluster. If you want a cluster that is deployed in the new network architecture to access the Internet, you can configure an NGINX proxy to forward requests.

Procedure

1. Configure a private connection to the Elasticsearch cluster and obtain the domain name of the related endpoint. The domain name is used to access external services.
   For more information, see Configure a private connection for an Elasticsearch cluster.

   Note This step is required only for a cluster that is deployed in the new network architecture.
2. Configure a security group rule for the ECS instance.
   1. Log on to the ECS console.
   2. In the left-side navigation pane, click Instances.
   3. On the Instances page, find the ECS instance and choose More > Network and Security Group > Configure Security Group in the Actions column.
   4. On the Security Groups tab, find your security group and click Add Rules in the Actions column.
   5. On the Inbound tab, click Add Rule.
   6. Configure parameters.
      

      Parameter	Description
      Action	Select Allow.
      Priority	Retain the default value.
      Protocol Type	Select Custom TCP.
      Port Range	Set this parameter to the port that you frequently use. If you want to configure an NGINX proxy, you must configure this parameter. In this example, port 8080 is used.
      Authorization Object	Enter the IP addresses of all the nodes in the Elasticsearch cluster. Note For more information about how to obtain the IP addresses of the nodes, see View the basic information of nodes.
      Description	The description of the rule.

   7. Click Save.
3. Configure an NGINX proxy.
   1. Install NGINX on the ECS instance.
   2. Configure the nginx.conf file.
      Replace the server configuration in the nginx.conf file with the following code.

      server
        {
          listen 8080;# Listening port
          server_name localhost;# Domain name
          index index.html index.htm index.php;
          root /usr/local/webserver/nginx/html;# Website directory
            location ~ .*\.(php|php5)?$
          {
            #fastcgi_pass unix:/tmp/php-cgi.sock;
            fastcgi_pass 127.0.0.1:9000;
            fastcgi_index index.php;
            include fastcgi.conf;
          }
          location ~ .*\.(gif|jpg|jpeg|png|bmp|swf|ico)$
          {
            expires 30d;
            # access_log off;
          }
          location / {
            proxy_pass <Webhook URL of the DingTalk chatbot>;
          }
          location ~ .*\.(js|css)?$
          {
            expires 15d;
            # access_log off;
          }
          access_log off;
        }

      Replace <Webhook URL of the DingTalk chatbot> with the webhook URL of the DingTalk chatbot that you configured to receive alert notifications.

      Note To obtain the webhook URL of the DingTalk chatbot, create an alert group in DingTalk. In the upper-right corner of the DingTalk group, click the Group Settings icon. In the Group Settings panel, click Group Assistant. In the Group Assistant panel, click Add Robot. In the ChatBot dialog box, click the Add icon on the right side of Add Robot to add a chatbot that you can access by using a webhook. Then, you can view the webhook URL of the DingTalk chatbot.
   3. Reload the NGINX configuration file and restart NGINX.

      /usr/local/webserver/nginx/sbin/nginx -s reload            # Reload the NGINX configuration file.
      /usr/local/webserver/nginx/sbin/nginx -s reopen            # Restart NGINX.

4. Create a watch for alerting.
   1. Log on to the Kibana console of the Elasticsearch cluster.
      Note For more information, see Log on to the Kibana console.
   2. In the left-side navigation pane, click Dev Tools.
   3. On the Console tab of the page that appears, run the following command to create a watch.
      In this example, a watch named log_error_watch is created to search the logs index for errors every 10 seconds. If more than 0 errors are found, an alert is triggered.

      PUT _xpack/watcher/watch/log_error_watch
      {
        "trigger": {
          "schedule": {
            "interval": "10s"
          }
        },
        "input": {
          "search": {
            "request": {
              "indices": ["logs"],
              "body": {
                "query": {
                  "match": {
                    "message": "error"
                  }
                }
              }
            }
          }
        },
        "condition": {
          "compare": {
            "ctx.payload.hits.total": {
              "gt": 0
            }
          }
        },
        "actions" : {
        "test_issue" : {
          "webhook" : {
            "method" : "POST",
            "url" : "http://<yourAddress>:8080",
            "body" : "{\"msgtype\": \"text\", \"text\": { \"content\": \"An error is found. Handle the issue immediately.\"}}"
          }
        }
      }
      }

      Table 1. Parameters

      Parameter	Network architecture type	Value	Description
      <yourAddress>	New network architecture	Domain name of the endpoint	In the new network architecture, private connections need to be established between VPCs, and the domain name of the related endpoint is used to forward requests. Notice You must set the parameter to the domain name of the related endpoint rather than the domain name of the related endpoint service. For more information about how to obtain the domain name of an endpoint, see View the domain name of an endpoint.
      	Original network architecture	IP address of the NGINX proxy	The NGINX proxy in the same VPC as the Elasticsearch cluster is used to forward requests over the Internet.
      		Webhook URL of the DingTalk chatbot	The SNAT feature must be enabled. This feature enables an ECS instance in a VPC to access the Internet if no public IP address is associated with the ECS instance.

      Notice

      • If the error No handler found for uri [/_xpack/watcher/watch/log_error_watch_2] and method [PUT] is returned after you run the preceding command, X-Pack Watcher is disabled for the Elasticsearch cluster. In this case, enable X-Pack Watcher and run the command again. For more information, see Configure the YML file.
      • When you create a DingTalk chatbot, you must configure security settings. This is because the body parameter in the preceding code must be specified based on the security settings. For more information, see Configure security settings. In this example, Security Settings is set to Custom Keywords and the error keyword is specified. In this case, the DingTalk chatbot sends alert notifications only if the content field in the body parameter contains error.
      If you no longer require this watch, you can run the following command to delete the watch:

      DELETE _xpack/watcher/watch/log_error_watch