Elasticsearch:Configure a DingTalk chatbot for X-Pack Watcher alerts

Step 1: Create and configure a DingTalk chatbot

1. Create a DingTalk group to receive alerts.

2. In the upper-right corner of the group, click the icon. In the Group Settings panel, click Robot.

3. In the Robot Management dialog box, click Add Bot, select Custom, and then click Add.

4. Enter a Bot Name, select Custom Keywords, and enter the keywords.

   Important

   The keywords must be included in the alert message that you configure.

5. Read and select Custom Robot Services and Disclaimers, and then click Complete.

6. Click Webhook next to Copy to copy the webhook URL of the chatbot.

   Keep the webhook URL secure. Do not share it publicly to avoid security risks.

Step 2: Configure the Nginx proxy and security group

1. Configure the Nginx proxy on the ECS instance.

   The Nginx proxy forwards alert messages. The X-Pack Watcher service sends alert messages to Nginx, which then forwards the messages to services such as DingTalk or WeCom.

   1. Install Nginx on the ECS instance.

   2. Configure the nginx.conf file.

      Replace the server section in the nginx.conf file with the following:

      server
        {
          listen 8080;# The listening port.
          server_name localhost;# The domain name.
          index index.html index.htm index.php;
          root /usr/local/webserver/nginx/html;# The website directory.
            location ~ .*\.(php|php5)?$
          {
            #fastcgi_pass unix:/tmp/php-cgi.sock;
            fastcgi_pass 127.0.0.1:9000;
            fastcgi_index index.php;
            include fastcgi.conf;
          }
          location ~ .*\.(gif|jpg|jpeg|png|bmp|swf|ico)$
          {
            expires 30d;
            # access_log off;
          }
          location / {
            proxy_pass <Webhook_URL>;
          }
          location ~ .*\.(js|css)?$
          {
            expires 15d;
            # access_log off;
          }
          access_log off;
        }

      Replace <Webhook_URL> with the webhook URL of the DingTalk chatbot.

   3. Reload the modified configuration file and restart Nginx.

      /usr/local/webserver/nginx/sbin/nginx -s reload            # Reload the configuration file.
      /usr/local/webserver/nginx/sbin/nginx -s reopen            # Restart Nginx.

2. Configure the security group for the ECS instance.

   Allow the Nginx proxy on the ECS instance to receive alert messages from the Elasticsearch instance.

   1. Log on to the Alibaba Cloud ECS console.

   2. In the left-side navigation pane, choose Instances & Images > Cluster.

   3. On the Cluster page, click the name of the target instance.

   4. Click the Security Group tab.

   5. On the Security Group List tab, click the ID of the target security group.

   6. On the Inbound tab, click Add Rule.

   7. Set the following parameters.

      Parameter	Description
      Action	Select Permitted.
      Priority	Retain the default value.
      Protocol	Select Custom TCP.
      Source	Add the IP addresses of all nodes in your Alibaba Cloud Elasticsearch instance. Note To obtain the IP addresses of the nodes, see View the basic information of a node.
      Destination	Enter the port that Nginx is configured to listen on. In this example, port 8080 is used.
      Description	Enter a description for the rule.

   8. Click OK.

Step 3: Configure a Watcher alert

1. Log on to the Kibana console of your Elasticsearch cluster.

   For instructions, see Log on to the Kibana console.

   Note

   Examples here use Elasticsearch V6.7.0. Operations may vary slightly for other versions.

2. In the left navigation menu, choose Management > Dev Tools.

3. In the Console, run the following command to create a watch.

   In this example, a log_error_watch document is created. Every 10s, the document queries the logs index for error logs and triggers an alarm if the log count exceeds 0.

   PUT _xpack/watcher/watch/log_error_watch
   {
     "trigger": {
       "schedule": {
         "interval": "10s"
       }
     },
     "input": {
       "search": {
         "request": {
           "indices": ["logs"],
           "body": {
             "query": {
               "match": {
                 "message": "error"
               }
             }
           }
         }
       }
     },
     "condition": {
       "compare": {
         "ctx.payload.hits.total": {
           "gt": 0
         }
       }
     },
     "actions" : {
     "test_issue" : {
       "webhook" : {
         "method" : "POST",
         "url" : "http://<your_ECS_IP_address>:8080",
         "body" : "{\"msgtype\": \"text\", \"text\": { \"content\": \"An error log is found. Please handle it as soon as possible.\"}}"
       }
     }
   }
   }

   Table 1. Key parameter descriptions

   Parameter	Network architecture	Description
   url	New network architecture	Set this parameter to the domain name of the endpoint. To obtain the endpoint domain name, see Configure private communication for an Alibaba Cloud Elasticsearch instance.
   	Old network architecture	Set this parameter to the following address: The private IP address of the ECS instance where the Nginx proxy is running. The public webhook URL of the DingTalk chatbot is not supported in this architecture.
   body	New network architecture Old network architecture	Configure this parameter based on the security settings of your DingTalk chatbot. For example, if you set the Security Settings to Custom Keywords and add the custom keyword error in Step 1: Create and configure a DingTalk chatbot, the content field in the body must contain error for the DingTalk chatbot to accept and display the alert.

   Note

   If you receive the No handler found for uri [/_xpack/watcher/watch/log_error_watch_2] and method [PUT] error when you run the command, it indicates that X-Pack Watcher is not enabled for your Alibaba Cloud Elasticsearch instance. Enable this feature and run the command again. For more information, see Configure YML parameters.

Step 4: Verify the results

If the configuration is correct, you will receive an alert message such as An error log is found. Please handle it as soon as possible. in your DingTalk group when your cluster's data meets the conditions defined in Step 3.

DELETE _xpack/watcher/watch/log_error_watch