Elasticsearch:Set up CCR exception alerts with X-Pack Watcher

Step 1: Create and configure a DingTalk bot

1. Create a DingTalk group to receive alerts.

2. In the upper-right corner of the group, click the icon. In the Group Settings panel, click Bot.

3. In the Robot Management dialog box, click Add Robot, select Custom, and then click Add.

4. Enter a Robot Name, select Custom Keywords, and then enter your keywords.

   Important

   The keywords must be included in the alert message that you configure.

5. Select I have read and agree to the "Custom Robot Service and Disclaimer Terms", and click Complete.

6. Next to Webhook, click Copy to copy the bot's webhook URL.

   Keep the webhook URL secure. Sharing it on external websites creates a security risk.

Step 2: Configure NGINX proxy and ECS security group

1. Configure the NGINX proxy on your ECS instance.

   X-Pack Watcher sends alerts to an NGINX proxy on your ECS instance, which then forwards them to DingTalk or WeCom.

   1. Install NGINX on the ECS instance.

   2. Configure the nginx.conf file.

      Replace the server section in the nginx.conf file with the following configuration.

      server
        {
          listen 8080;# Listener port
          server_name localhost;# Domain name
          index index.html index.htm index.php;
          root /usr/local/webserver/nginx/html;# Site directory
            location ~ .*\.(php|php5)?$
          {
            #fastcgi_pass unix:/tmp/php-cgi.sock;
            fastcgi_pass 127.0.0.1:9000;
            fastcgi_index index.php;
            include fastcgi.conf;
          }
          location ~ .*\.(gif|jpg|jpeg|png|bmp|swf|ico)$
          {
            expires 30d;
            # access_log off;
          }
          location / {
            proxy_pass <Webhook URL>;
          }
          location ~ .*\.(js|css)?$
          {
            expires 15d;
            # access_log off;
          }
          access_log off;
        }

      Replace <Webhook URL> with the webhook URL of the DingTalk bot that you copied in Step 1.

   3. Reload the configuration file and restart NGINX.

      /usr/local/webserver/nginx/sbin/nginx -s reload            # Reload the configuration file
      /usr/local/webserver/nginx/sbin/nginx -s reopen            # Restart NGINX

2. Configure the ECS security group.

   Allow inbound traffic from your Alibaba Cloud Elasticsearch instance to the NGINX proxy on the ECS instance.

   1. Log on to the Alibaba Cloud ECS console.

   2. In the left-side navigation pane, choose Instances & Images > Cluster.

   3. On the Cluster page, click the name of the target instance.

   4. Click the Security Group tab.

   5. On the Security Groups tab, click the name of the target security group.

   6. On the Inbound tab, click Add Rule.

   7. Configure the parameters.

      Parameter	Description
      Action	Select Permitted.
      Priority	Keep the default value.
      Protocol	Select Custom TCP.
      Access Source	Enter the IP addresses of all nodes in your Alibaba Cloud Elasticsearch instance. Note To get the IP addresses of the nodes, see View basic information about nodes.
      Destination	Enter the port that NGINX is configured to listen on. This topic uses port 8080 as an example.
      Description	Enter a description for the rule.

   8. Click OK.

Step 3: Configure the watch

1. Log on to the Kibana console of your Elasticsearch cluster.

   For instructions, see Log on to the Kibana console.

   Note

   Examples here use Elasticsearch V6.7.0. Operations may vary slightly for other versions.

2. In the left navigation menu, choose Management > Dev Tools.

3. In the Console, run the following command to create a watch.

   PUT _watcher/watch/ccr_watcher
   {
     "trigger": {
       "schedule": {
         "interval": "10s"
       }
     },
     "input": {
       "search": {
         "request": {
           "indices": [
             ".monitoring-es*" 
           ],
           "body": {
             "size": 0,
             "sort": [
               {
                 "timestamp": {
                   "order": "desc"
                 }
               }
             ],
             "query": {
               "bool": {
                 "must": [
                   {
                     "range": {
                       "timestamp": {
                         "gte": "now-10m"
                       }
                     }
                   },
                   {
                     "term": {
                       "type": {
                         "value": "ccr_stats"
                       }
                     }
                   },
                   {
                     "bool": {
                       "should": [
                         {
                           "range": {
                             "ccr_stats.time_since_last_read_millis": {
                               "gte": 600000
                             }
                           }
                         },
                         {
                           "script": {
                             "script": "long gap = doc['ccr_stats.leader_global_checkpoint'].value - doc['ccr_stats.follower_global_checkpoint'].value;\n            return gap>1000;"
   
   
                           }
                         }
                       ]
                     }
                   }
                 ]
               }
             },
             "aggs": {
               "NAME": {
                 "terms": {
                   "field": "ccr_stats.follower_index",
                   "size": 1000
                 }
               }
             }
           }
         }
       }
     },
     "condition": {
       "compare": {
         "ctx.payload.hits.total": {
           "gt": 0
         }
       }
     },
     "transform": {
       "script": """
       StringBuilder message = new StringBuilder();
   for (def bucket : ctx.payload.aggregations.NAME.buckets) {
     message.append(bucket.key).append('  ')
   }
       return [ 'delay_indices' : message.toString().trim()  ]
   """
     },
     "actions" : {
        "add_index": {
         "index": {
           "index": "ccr_delay_indices",
           "doc_type": "doc"
         }
       },
        "my_webhook": {
        "webhook" : {
           "method" : "POST",
           "url" : "http://<yourAddress>:8080",
           "body" : "{\"msgtype\": \"text\", \"text\": { \"content\": \"Please note： {{ctx.payload}}\"}}"
         }
       }
     }
   
   }

   The following table describes the key parameters.

   Parameter	Description
   trigger	The watch trigger interval. In this example, the interval is 10 seconds. Adjust this value as needed.
   input.search.request.indices	The target indexes to query. The .monitoring-es* indexes store all monitoring metrics for the cluster, including CCR metrics.
   input.search.request.body	The query body. This example queries CCR status from the last 10 minutes. The watch proceeds if either of the following conditions is met: ccr_stats.time_since_last_read_millis > 600000 ms (10 minutes): More than 10 minutes have elapsed since the last read operation from the leader cluster. Adjust this threshold as needed. The difference between ccr_stats.leader_global_checkpoint and ccr_stats.follower_global_checkpoint is greater than 1,000. This means the follower checkpoint is more than 1,000 operations behind the leader checkpoint. Adjust this threshold as needed.
   condition	The alert trigger condition. In this example, the alert triggers if the conditions in input.search.request.body are met and the query returns more than zero matching documents.
   transform	A preprocessing step. This example iterates through the bucket keys and extracts the names of lagging indexes.
   actions	The actions to execute if the condition is true. This example specifies two actions: add_index: Writes the results to an index, which is useful for debugging the watch configuration. my_webhook: Sends an alert by using a webhook.
   <yourAddress>	The address of the server that receives alerts: For clusters that use the new network architecture, set this parameter to the endpoint of the private connection. For more information about how to obtain the endpoint, see Configure a private connection for an Elasticsearch cluster. For clusters that use the original network architecture, set this parameter to one of the following addresses: The IP address of the NGINX proxy. The NGINX proxy forwards requests from within the VPC to the internet. The webhook URL of the DingTalk chatbot.
   body	Configure this value according to the security settings of the DingTalk chatbot. For example, if you select Custom Keywords as the Security Settings and add note as a keyword in Step 1: Create and configure a DingTalk chatbot, the content field in the body must contain the word note for the chatbot to process the alert.

   Note

   If you see the error No handler found for uri [/_xpack/watcher/watch/log_error_watch_2] and method [PUT] when you run the command, X-Pack Watcher is not enabled for your Alibaba Cloud Elasticsearch instance. You must enable it and then run the command again. For more information, see Configure YML parameters.

Step 4: View the alert

If the CCR status meets the alert condition configured in Step 3: Configure the watch, you receive an alert in your DingTalk group.

DELETE _watcher/watch/ccr_watcher