Elasticsearch:Configure a DingTalk bot for X-Pack Watcher alerts

Step 1: Create and configure a DingTalk bot

1. Create a DingTalk group to receive alerts.

2. In the upper-right corner of the group, click the icon. In the Group Settings panel, click Bot.

3. In the Robot Management dialog box, click Add Robot, select Custom, and then click Add.

4. Enter a Robot Name, select Custom Keywords, and then enter your keywords.

   Important

   The keywords must be included in the alert message that you configure.

5. Select I have read and agree to the "Custom Robot Service and Disclaimer Terms", and click Complete.

6. Next to Webhook, click Copy to copy the bot's webhook URL.

   Keep the webhook URL secure. Sharing it on external websites creates a security risk.

Step 2: Configure NGINX proxy and ECS security group

1. Configure the NGINX proxy on your ECS instance.

   X-Pack Watcher sends alerts to an NGINX proxy on your ECS instance, which then forwards them to DingTalk or WeCom.

   1. Install NGINX on the ECS instance.

   2. Configure the nginx.conf file.

      Replace the server section in the nginx.conf file with the following configuration.

      server
        {
          listen 8080;# Listener port
          server_name localhost;# Domain name
          index index.html index.htm index.php;
          root /usr/local/webserver/nginx/html;# Site directory
            location ~ .*\.(php|php5)?$
          {
            #fastcgi_pass unix:/tmp/php-cgi.sock;
            fastcgi_pass 127.0.0.1:9000;
            fastcgi_index index.php;
            include fastcgi.conf;
          }
          location ~ .*\.(gif|jpg|jpeg|png|bmp|swf|ico)$
          {
            expires 30d;
            # access_log off;
          }
          location / {
            proxy_pass <Webhook URL>;
          }
          location ~ .*\.(js|css)?$
          {
            expires 15d;
            # access_log off;
          }
          access_log off;
        }

      Replace <Webhook URL> with the webhook URL of the DingTalk bot that you copied in Step 1.

   3. Reload the configuration file and restart NGINX.

      /usr/local/webserver/nginx/sbin/nginx -s reload            # Reload the configuration file
      /usr/local/webserver/nginx/sbin/nginx -s reopen            # Restart NGINX

2. Configure the ECS security group.

   Allow inbound traffic from your Alibaba Cloud Elasticsearch instance to the NGINX proxy on the ECS instance.

   1. Log on to the Alibaba Cloud ECS console.

   2. In the left-side navigation pane, choose Instances & Images > Cluster.

   3. On the Cluster page, click the name of the target instance.

   4. Click the Security Group tab.

   5. On the Security Groups tab, click the name of the target security group.

   6. On the Inbound tab, click Add Rule.

   7. Configure the parameters.

      Parameter	Description
      Action	Select Permitted.
      Priority	Keep the default value.
      Protocol	Select Custom TCP.
      Access Source	Enter the IP addresses of all nodes in your Alibaba Cloud Elasticsearch instance. Note To get the IP addresses of the nodes, see View basic information about nodes.
      Destination	Enter the port that NGINX is configured to listen on. This topic uses port 8080 as an example.
      Description	Enter a description for the rule.

   8. Click OK.

Step 3: Configure a Watcher alert

1. Log on to the Kibana console of your Elasticsearch cluster.

   For instructions, see Log on to the Kibana console.

   Note

   Examples here use Elasticsearch V6.7.0. Operations may vary slightly for other versions.

2. In the left navigation menu, choose Management > Dev Tools.

3. In the Console, run the following command to create a watch.

   The following example creates a watch named log_error_watch. It queries the logs index every 10 seconds for logs that contain error. If more than 0 matching logs are found, an alert is triggered.

   PUT _xpack/watcher/watch/log_error_watch
   {
     "trigger": {
       "schedule": {
         "interval": "10s"
       }
     },
     "input": {
       "search": {
         "request": {
           "indices": ["logs"],
           "body": {
             "query": {
               "match": {
                 "message": "error"
               }
             }
           }
         }
       }
     },
     "condition": {
       "compare": {
         "ctx.payload.hits.total": {
           "gt": 0
         }
       }
     },
     "actions" : {
     "test_issue" : {
       "webhook" : {
         "method" : "POST",
         "url" : "http://<yourAddress>:8080",
         "body" : "{\"msgtype\": \"text\", \"text\": { \"content\": \"An error log has appeared. Please handle it as soon as possible.\"}}"
       }
     }
   }
   }

   Table 1. Key parameters

   Parameter	Network type	Configuration
   url	New network architecture	Set this to the endpoint domain name. To obtain the endpoint domain name, see Configure a PrivateLink connection for an Alibaba Cloud Elasticsearch instance.
   	Old network architecture	Set this to one of the following addresses: The IP address of the NGINX proxy: Requests are forwarded by the NGINX proxy in the same VPC over the public internet. The webhook URL of the DingTalk bot.
   body	New network architecture Old network architecture	Configure this parameter based on the security settings of the DingTalk bot. For example, in Step 1: Create and configure a DingTalk bot, if you set Security Settings to Custom Keywords and add the custom keyword error, the DingTalk bot pushes alert messages only if the content field in the body contains error.

   Note

   If you see the error No handler found for uri [/_xpack/watcher/watch/log_error_watch_2] and method [PUT] when you run the command, X-Pack Watcher is not enabled for your Alibaba Cloud Elasticsearch instance. You must enable it and then run the command again. For more information, see Configure YML parameters.

Step 4: Verify the alert

After the configuration is complete, your DingTalk group receives a message like An error log has appeared. Please handle it as soon as possible. whenever the alert condition from Step 3 is met.

DELETE _xpack/watcher/watch/log_error_watch