This topic describes how to configure X-Pack Watcher for Elasticsearch. X-Pack Watcher allows you to trigger specific actions when specified conditions are met. For example, you can create a watch for Elasticsearch to search the logs index for errors and send alerts through emails or DingTalk messages. X-Pack Watcher is a monitoring and alerting service based on Elasticsearch.

Prerequisites

  • An Elasticsearch cluster that is deployed in a single zone is created.
    For more information, see Create an Elasticsearch cluster.
    Note X-Pack Watcher is available only for an Elasticsearch cluster that is deployed in a single zone.
  • X-Pack Watcher is enabled for an Elasticsearch cluster. It is disabled by default.

    For more information, see Modify the YML configuration.

  • An Elastic Compute Service (ECS) instance is created.
    The ECS instance must be accessible over the Internet and located in the same region and Virtual Private Cloud (VPC) as the Elasticsearch cluster. For more information, see Step 1: Create an ECS instance.
    Note The X-Pack Watcher feature of Elasticsearch cannot directly access the Internet. You must use the internal endpoint of an Elasticsearch cluster in a VPC to access the Internet. Therefore, you must create an ECS instance that can access both the Internet and the Elasticsearch cluster and use it as a proxy to perform actions.

Background information

X-Pack Watcher allows you to create watches. A watch consists of a trigger, input, condition, and actions.
  • Trigger

    Determines when the watch is executed. You must configure a trigger for each watch. X-Pack Watcher allows you to create various types of triggers. For more information, see Schedule Trigger.

  • Input

    Loads data into the payload of a watch. Inputs are used as filters to match the specified type of index data. For more information, see Inputs.

  • Condition

    Controls whether the actions of a watch are performed.

  • Actions

    Determines the actions to be performed when the specified condition is met. This topic uses the webhook action as an example.

Procedure

  1. Configure a security group rule for the ECS instance.
    1. Log on to the Alibaba Cloud Elastic Compute Service console. In the left-side navigation pane, click Instances & Images. Then, click Instances.
    2. On the Instances page, find the target instance, click More, and then choose Network and Security Group > Configure Security Group in the Actions column.
    3. On the Security Groups tab, find the target security group and click Add Rules in the Actions column.
    4. On the Security Group Rules page, click Add Security Group Rule.
    5. Specify the required parameters.
      Add Security Group Rule
      Parameter Description
      Rule Direction Select Inbound.
      Action Select Allow.
      Protocol Type Select Custom TCP.
      Priority Retain the default value.
      Port Range Set the port to your frequently used port. This parameter is required for NGINX configurations. In this example, port 8080 is used.
      Authorization Type Select IPv4 CIDR Block.
      Authorization Object Add the IP addresses of all nodes in your Elasticsearch cluster.
      Note To query the IP addresses of the nodes, log on to the Kibana console of your Elasticsearch cluster based on Log on to the Kibana console, click Monitoring in the left-side navigation pane, and then click Nodes.
    6. Click OK.
  2. Configure an NGINX proxy.
    1. Install NGINX on the ECS instance.
    2. Configure the nginx.conf file.
      Replace the server configuration in the nginx.conf file with the following content:
      server
        {
          listen 8080;# Listening port
          server_name localhost;# Domain name
          index index.html index.htm index.php;
          root /usr/local/webserver/nginx/html;# Website directory
            location ~ . *\.(php|php5)? $
          {
            #fastcgi_pass unix:/tmp/php-cgi.sock;
            fastcgi_pass 127.0.0.1:9000;
            fastcgi_index index.php;
            include fastcgi.conf;
          }
          location ~ . *\.(gif|jpg|jpeg|png|bmp|swf|ico)$
          {
            expires 30d;
            # access_log off;
          }
          location / {
            proxy_pass <Webhook address of the DingTalk Chatbot>;
          }
          location ~ . *\.(js|css)? $
          {
            expires 15d;
            # access_log off;
          }
          access_log off;
        }
      <Webhook address of the DingTalk Chatbot>: Replace it with the webhook address of the DingTalk Chatbot that is used to receive alert notifications.
      Note To query the webhook address of the DingTalk Chatbot, create an alert group in DingTalk. Then, in the DingTalk group, click the More icon in the upper-right corner, click ChatBot, and then select Custom to add a ChatBot that is connected by using webhooks. You can then view the webhook address of the DingTalk Chatbot.
    3. Reload the NGINX configuration file and restart NGINX.
      /usr/local/webserver/nginx/sbin/nginx -s reload            # Reload the NGINX configuration file.
      /usr/local/webserver/nginx/sbin/nginx -s reopen            # Restart NGINX.
  3. Create a watch.
    1. Log on to the Kibana console of your Elasticsearch cluster.
      Note For more information, see Log on to the Kibana console.
    2. In the left-side navigation pane, click Dev Tools.
    3. On the Console tab, run the following command to create a watch:
      The following example shows how to create a watch named log_error_watch to search the logs index for errors every 10 seconds. If more than 0 errors are found, an alert is triggered.
      PUT _xpack/watcher/watch/log_error_watch
      {
        "trigger": {
          "schedule": {
            "interval": "10s"
          }
        },
        "input": {
          "search": {
            "request": {
              "indices": ["logs"],
              "body": {
                "query": {
                  "match": {
                    "message": "error"
                  }
                }
              }
            }
          }
        },
        "condition": {
          "compare": {
            "ctx.payload.hits.total": {
              "gt": 0
            }
          }
        },
        "actions" : {
        "test_issue" : {
          "webhook" : {
            "method" : "POST",
            "url" : "http://<Private IP address of your ECS instance>:8080",
            "body" : "{\"msgtype\": \"text\", \"text\": { \"content\": \"An error has been found. Handle the issue immediately.\"}}"
          }
        }
      }
      }
      Notice
      • url specified in actions must contain the private IP address of your ECS instance that is deployed in the same region and VPC as your Elasticsearch cluster. You must also create a security group rule for the ECS instance by following the preceding procedure. Otherwise, the instance cannot connect to X-Pack Watcher.
      • If error No handler found for uri [/_xpack/watcher/watch/log_error_watch_2] and method [PUT] is returned when you run the preceding command, X-Pack Watcher is disabled for your Elasticsearch cluster. In this case, enable X-Pack Watcher and then run the command. For more information, see Modify the YML configuration.
      If you no longer need this watch, run the following command to delete the watch:
      DELETE _xpack/watcher/watch/log_error_watch