X-Pack Watcher is a monitoring and alerting service that is developed for Elasticsearch. If you configure X-Pack Watcher for your Elasticsearch cluster, X-Pack Watcher can trigger actions when specific conditions are met. For example, if the logs index contains errors, X-Pack Watcher triggers the system to send alert notifications by using emails or DingTalk messages. This topic describes how to configure a DingTalk chatbot to receive alert notifications from X-Pack Watcher.

Background information

X-Pack Watcher allows you to create watches. A watch consists of a trigger, an input, a condition, and actions.
  • Trigger

    Determines the interval at which the operation specified in the input definition is performed. X-Pack Watcher allows you to create various types of triggers. For more information, see Schedule Trigger.

  • Input
    Loads data into the execution context of a watch when the watch is triggered. This payload is accessible during the subsequent execution phases of the watch. If no input is specified for a watch, an empty payload is loaded. For more information, see Inputs. A watch supports the following input types:
    • simple: Load static data into the execution context. For example, you can enter a piece of data for alerting.
    • search: Load the results of a search into the execution context. For example, the results of a full-text search that is performed based on a keyword are used for alerting.
    • http: Load the results of an HTTP request into the execution context. For example, the results of calling an Elasticsearch API operation that is used to query the health status and node status of an Elasticsearch cluster are used for alerting.
    • chain: Load a series of input data into the execution context. In most cases, the input data has different sources.
  • Condition
    Controls whether a watch performs actions. If specific conditions are met, the actions are triggered. If you do not specify the condition type for a watch, the conditions of the always type are used by default. For more information, see Conditions. A watch supports the following condition types:
    • always: The conditions are always met, and the actions are always performed.
    • never: The conditions are never met, and the actions are never performed.
    • compare: The values in the payload are compared to determine whether to perform the actions.
    • array_compare: An array of values in the payload is compared with a specific value to determine whether to perform the actions.
    • script: A script is used to determine whether to perform the actions.
  • Actions
    Determines the actions that a watch performs when specific conditions are met. A watch supports actions such as email, webhook, index, and logging. For more information, see Actions.
    Note Alibaba Cloud Elasticsearch does not support an email action because an endpoint restriction is imposed on this type of action. We recommend that you use a webhook action.

Prerequisites

  • A single-zone Alibaba Cloud Elasticsearch cluster is created.
    For more information, see Create an Alibaba Cloud Elasticsearch cluster.
    Note In the original network architecture, X-Pack Watcher is available only for single-zone Elasticsearch clusters. In the new network architecture, X-Pack Watcher is available for both single-zone Elasticsearch clusters and multi-zone Elasticsearch clusters.
  • X-Pack Watcher is enabled for the Elasticsearch cluster. By default, X-Pack Watcher is disabled.

    For more information, see Configure the YML file.

  • An Elastic Compute Service (ECS) instance is created in your virtual private cloud (VPC), and the required applications are deployed on the ECS instance.

    For more information, see Create an instance by using the wizard.

    Note
    • When you use PrivateLink to establish private connections between VPCs, the ECS instance is used as a backend server to receive requests that are forwarded by a Server Load Balancer (SLB) instance. The ECS instance can be deployed in a zone that is different from the SLB instance but must be deployed in the same VPC and region as the SLB instance.
    • X-Pack Watcher cannot directly access the Internet. X-Pack Watcher must use the internal endpoint of your Elasticsearch cluster to access the Internet. In this case, you can enable source network address translation (SNAT) for or associate an elastic IP address (EIP) with an ECS instance that is deployed in a VPC. This way, you can use the ECS instance as a proxy to forward requests.

Precautions

The network architecture of Alibaba Cloud Elasticsearch in different regions has been adjusted since October 2020. The adjustment has the following impacts on clusters:
  • Clusters that are created before October 2020 are deployed in the original network architecture. In this architecture, clusters are deployed in VPCs that are created by users. If you want a cluster that is deployed in this architecture to access the Internet, you can use an ECS instance for which SNAT is enabled or use an NGINX proxy to forward requests.
  • Clusters that are created in October 2020 or later are deployed in the new network architecture. In this architecture, clusters are deployed in the VPC within the service account of Alibaba Cloud Elasticsearch. If you want to use X-Pack Watcher for an Elasticsearch cluster that is created in October 2020 or later, you must first use the PrivateLink service to establish a private connection between the VPC within the service account of Alibaba Cloud Elasticsearch and your VPC. For more information, see Configure a private connection for an Elasticsearch cluster. If you want to send alert notifications to the Internet, you must also configure an NGINX proxy or enable SNAT for an ECS instance that is used as a backend server to receive requests forwarded by an SLB instance.
    Notice If your cluster is deployed in the new network architecture and you want to use the X-Pack Watcher, reindex, LDAP authentication, or AD user authentication feature, you must configure a private connection for your cluster. To ensure the availability of the features, you must strictly follow the instructions in this topic.

Procedure

  1. Step 1: Configure a DingTalk chatbot
  2. Step 2: (Optional) Configure a private connection for the Elasticsearch cluster
  3. Step 3: Configure a security group rule for the ECS instance and configure an NGINX proxy
  4. Step 4: Configure a watch for alerting
  5. Step 5: View the alert notifications

Step 1: Configure a DingTalk chatbot

  1. Create a DingTalk group that is used to receive alert notifications.
  2. Click Group Settings in the upper-right corner of the chat window of the DingTalk group. In the Group Settings panel, click Group Assistant. In the Group Assistant panel, click Add Robot. In the ChatBot dialog box, click the icon next to Add Robot. In the ChatBot dialog box, click Custom. Configure basic information and security settings for the DingTalk chatbot. Then, copy the webhook URL that is displayed.

Step 2: (Optional) Configure a private connection for the Elasticsearch cluster

This step is required only for a cluster that is deployed in the new network architecture.

  1. Log on to the Elasticsearch console.
  2. Configure a private connection for the Elasticsearch cluster and obtain the domain name of the endpoint that is used to access the endpoint service.

Step 3: Configure a security group rule for the ECS instance and configure an NGINX proxy

  1. Configure a security group rule for the ECS instance.
    1. Log on to the ECS console.
    2. In the left-side navigation pane, choose Instances & Images > Instances.
    3. On the Instances page, find the ECS instance for which you want to configure a security group rule and choose More > Network and Security Group > Configure Security Group in the Actions column.
    4. On the Security Groups tab of the page that appears, find the desired security group and click Add Rules in the Actions column.
    5. On the Inbound tab, click Add Rule.
    6. Configure the parameters.
      Configure a security group rule
      Parameter Description
      Action Select Allow.
      Priority Retain the default value.
      Protocol Type Select Custom TCP.
      Port Range Set this parameter to the port that you frequently use. If you want to configure an NGINX proxy, you must configure this parameter. In this example, port 8080 is used.
      Authorization Object Enter the IP addresses of all nodes in the Elasticsearch cluster.
      Note For more information about how to obtain the IP addresses of the nodes, see View the basic information of nodes.
      Description The description of the rule.
    7. Click Save in the Actions column.
  2. Configure an NGINX proxy.
    1. Install NGINX on the ECS instance.
    2. Configure the nginx.conf file.
      Replace the server configuration in the nginx.conf file with the following code. Server configuration in the nginx.conf file
      server
        {
          listen 8080;# Listening port
          server_name localhost;# Domain name
          index index.html index.htm index.php;
          root /usr/local/webserver/nginx/html;# Website directory
            location ~ .*\.(php|php5)?$
          {
            #fastcgi_pass unix:/tmp/php-cgi.sock;
            fastcgi_pass 127.0.0.1:9000;
            fastcgi_index index.php;
            include fastcgi.conf;
          }
          location ~ .*\.(gif|jpg|jpeg|png|bmp|swf|ico)$
          {
            expires 30d;
            # access_log off;
          }
          location / {
            proxy_pass <Webhook URL of the DingTalk chatbot>;
          }
          location ~ .*\.(js|css)?$
          {
            expires 15d;
            # access_log off;
          }
          access_log off;
        }

      Replace <Webhook URL of the DingTalk chatbot> with the webhook URL of the DingTalk chatbot that you configured to receive alert notifications. For more information about how to obtain the webhook URL of a DingTalk chatbot, see Step 1: Configure a DingTalk chatbot.

    3. Reload the NGINX configuration file and restart NGINX.
      /usr/local/webserver/nginx/sbin/nginx -s reload            # Reload the NGINX configuration file.
      /usr/local/webserver/nginx/sbin/nginx -s reopen            # Restart NGINX.

Step 4: Configure a watch for alerting

  1. Log on to the Kibana console of your Elasticsearch cluster and go to the homepage of the Kibana console as prompted.
    For more information about how to log on to the Kibana console, see Log on to the Kibana console.
    Note In this example, an Elasticsearch V6.7.0 cluster is used. Operations on clusters of other versions may differ. The actual operations in the console prevail.
  2. In the left-side navigation pane of the page that appears, click Dev Tools.
  3. On the Console tab of the page that appears, run the following command to create a watch.
    In this example, a watch named log_error_watch is created to search the logs index for errors every 10 seconds. If more than 0 errors are found, an alert is triggered.
    PUT _xpack/watcher/watch/log_error_watch
    {
      "trigger": {
        "schedule": {
          "interval": "10s"
        }
      },
      "input": {
        "search": {
          "request": {
            "indices": ["logs"],
            "body": {
              "query": {
                "match": {
                  "message": "error"
                }
              }
            }
          }
        }
      },
      "condition": {
        "compare": {
          "ctx.payload.hits.total": {
            "gt": 0
          }
        }
      },
      "actions" : {
      "test_issue" : {
        "webhook" : {
          "method" : "POST",
          "url" : "http://<yourAddress>:8080",
          "body" : "{\"msgtype\": \"text\", \"text\": { \"content\": \"An error is found. Handle the issue immediately.\"}}"
        }
      }
    }
    }
    Table 1. Parameters
    Parameter Network architecture Value Description
    <yourAddress> New network architecture Domain name of the endpoint In the new network architecture, private connections must be established between VPCs, and the domain name of the related endpoint is used to forward requests.
    Notice You must set this parameter to the domain name of the related endpoint instead of the domain name of the related endpoint service. For more information about how to obtain the domain name of an endpoint, see View the domain name of an endpoint.
    Original network architecture IP address of the NGINX proxy The NGINX proxy in the same VPC as the Elasticsearch cluster is used to forward requests over the Internet.
    Webhook URL of the DingTalk chatbot (for the url parameter) The SNAT feature must be enabled for the ECS instance. This feature enables an ECS instance in a VPC to access the Internet when no public IP address is associated with the ECS instance.
    Notice
    • If the error No handler found for uri [/_xpack/watcher/watch/log_error_watch_2] and method [PUT] is returned after you run the preceding command, X-Pack Watcher is disabled for the Elasticsearch cluster. In this case, enable X-Pack Watcher and run the command again. For more information, see Configure the YML file.
    • You must configure the body parameter in the preceding command based on the security settings of the DingTalk chatbot. For more information about how to obtain the security settings of a DingTalk chatbot, see Step 1: Configure a DingTalk chatbot. In this example, Security Settings is set to Custom Keywords and the error keyword is specified. In this case, the DingTalk chatbot sends alert notifications only if the content field in the body parameter contains error.

Step 5: View the alert notifications

In normal cases, if the conditions specified in Step 4: Configure a watch for alerting are met, an alert notification similar to An error is found. Handle the issue immediately is sent to the DingTalk group.
Note If you no longer require this watch, you can run the following command to delete the watch:
DELETE _xpack/watcher/watch/log_error_watch