X-Pack Watcher is a monitoring and alerting service developed for Elasticsearch. If
you configure X-Pack Watcher for your cluster, X-Pack Watcher can trigger actions
when specific conditions are met. For example, if the logs index contains errors,
X-Pack Watcher triggers the system to send alert notifications by using emails or
DingTalk messages. X-Pack Watcher is an Elasticsearch-based monitoring and alerting
service. This topic describes how to configure a DingTalk chatbot to receive alert
notifications from X-Pack Watcher.
Background information
X-Pack Watcher allows you to create watches. A watch consists of a trigger, an input,
a condition, and actions.
- Trigger
Determines when a watch starts to run. You must configure a trigger for each watch.
X-Pack Watcher allows you to create various types of triggers. For more information,
see Schedule Trigger.
- Input
Loads data to the payload of a watch. Inputs are used as filters to match the specified
type of index data. For more information, see Inputs.
- Condition
Controls whether a watch performs actions.
- Actions
Determines the actions that a watch performs when the specified condition is met.
In this example, the webhook action is used.
Prerequisites
- A single-zone Alibaba Cloud Elasticsearch cluster is created.
For more information, see
Create an Alibaba Cloud Elasticsearch cluster.
Note In the original network architecture, X-Pack Watcher is available only for single-zone
Elasticsearch clusters. In the new network architecture, X-Pack Watcher is available
for both single-zone Elasticsearch clusters and multi-zone Elasticsearch clusters.
- X-Pack Watcher is enabled for the Elasticsearch cluster. By default, X-Pack Watcher
is disabled.
For more information, see Configure the YML file.
- An Elastic Compute Service (ECS) instance is created in your virtual private cloud
(VPC), and the required applications are deployed on the ECS instance.
For more information, see Create an instance by using the wizard.
Note
- When you use PrivateLink to establish private connections between VPCs, the ECS instance
is used as a backend server to receive requests that are forwarded by a Server Load
Balancer (SLB) instance. The ECS instance can be deployed in a zone that is different
from the SLB instance but must be deployed in the same VPC and region as the SLB instance.
- X-Pack Watcher cannot directly access the Internet. It must use the internal endpoint
of your Elasticsearch cluster to access the Internet. In this case, you can enable
source network address translation (SNAT) for or associate an elastic IP address (EIP)
with an ECS instance that is deployed in a VPC. This way, you can use the ECS instance
as a proxy to forward requests.
Precautions
The network architecture of Alibaba Cloud Elasticsearch in different regions has been
adjusted since October 2020. The adjustment has the following impacts on clusters:
- Clusters that are created before October 2020 are deployed in the original network
architecture. In this architecture, clusters are deployed in the VPCs that are created
by users. If you want a cluster that is deployed in this architecture to access the
Internet, you can use an ECS instance for which SNAT is enabled or use an NGINX proxy to forward requests.
- Clusters that are created in October 2020 or later are deployed in the new network
architecture. If you want to use X-Pack Watcher for an Elasticsearch cluster that
is created in October 2020 or later, you must first use the PrivateLink service to
establish private connections between VPCs. For more information, see Configure a private connection for an Elasticsearch cluster. If you want a cluster that is deployed in the new network architecture to access
the Internet, you can configure an NGINX proxy to forward requests.
Procedure
- Configure a private connection to the Elasticsearch cluster and obtain the domain
name of the related endpoint. The domain name is used to access external services.
- Configure a security group rule for the ECS instance.
- Log on to the ECS console.
- In the left-side navigation pane, click Instances.
- On the Instances page, find the ECS instance and choose in the Actions column.
- On the Security Groups tab, find your security group and click Add Rules in the Actions column.
- On the Inbound tab, click Add Rule.
- Configure parameters.

Parameter |
Description |
Action |
Select Allow.
|
Priority |
Retain the default value. |
Protocol Type |
Select Custom TCP.
|
Port Range |
Set this parameter to the port that you frequently use. If you want to configure an
NGINX proxy, you must configure this parameter. In this example, port 8080 is used.
|
Authorization Object |
Enter the IP addresses of all the nodes in the Elasticsearch cluster.
|
Description |
The description of the rule. |
- Click Save.
- Configure an NGINX proxy.
- Install NGINX on the ECS instance.
- Configure the nginx.conf file.
Replace the
server
configuration in the nginx.conf file with the following code.

server
{
listen 8080;# Listening port
server_name localhost;# Domain name
index index.html index.htm index.php;
root /usr/local/webserver/nginx/html;# Website directory
location ~ .*\.(php|php5)?$
{
#fastcgi_pass unix:/tmp/php-cgi.sock;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
include fastcgi.conf;
}
location ~ .*\.(gif|jpg|jpeg|png|bmp|swf|ico)$
{
expires 30d;
# access_log off;
}
location / {
proxy_pass <Webhook URL of the DingTalk chatbot>;
}
location ~ .*\.(js|css)?$
{
expires 15d;
# access_log off;
}
access_log off;
}
Replace
<Webhook URL of the DingTalk chatbot> with the webhook URL of the DingTalk chatbot that you configured to receive alert
notifications.
Note To obtain the webhook URL of the DingTalk chatbot, create an alert group in DingTalk.
In the upper-right corner of the DingTalk group, click the Group Settings icon. In
the Group Settings panel, click Group Assistant. In the Group Assistant panel, click
Add Robot. In the ChatBot dialog box, click the Add icon on the right side of Add
Robot to add a chatbot that you can access by using a webhook. Then, you can view
the webhook URL of the DingTalk chatbot.
- Reload the NGINX configuration file and restart NGINX.
/usr/local/webserver/nginx/sbin/nginx -s reload # Reload the NGINX configuration file.
/usr/local/webserver/nginx/sbin/nginx -s reopen # Restart NGINX.
- Create a watch for alerting.
- Log on to the Kibana console of the Elasticsearch cluster.
- In the left-side navigation pane, click Dev Tools.
- On the Console tab of the page that appears, run the following command to create a watch.
In this example, a watch named
log_error_watch
is created to search the
logs
index for
errors
every
10 seconds
. If more than
0
errors are found, an alert is triggered.
PUT _xpack/watcher/watch/log_error_watch
{
"trigger": {
"schedule": {
"interval": "10s"
}
},
"input": {
"search": {
"request": {
"indices": ["logs"],
"body": {
"query": {
"match": {
"message": "error"
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 0
}
}
},
"actions" : {
"test_issue" : {
"webhook" : {
"method" : "POST",
"url" : "http://<yourAddress>:8080",
"body" : "{\"msgtype\": \"text\", \"text\": { \"content\": \"An error is found. Handle the issue immediately.\"}}"
}
}
}
}
Table 1. Parameters
Parameter |
Network architecture type |
Value |
Description |
<yourAddress> |
New network architecture |
Domain name of the endpoint |
In the new network architecture, private connections need to be established between
VPCs, and the domain name of the related endpoint is used to forward requests.
Notice You must set the parameter to the domain name of the related endpoint rather than
the domain name of the related endpoint service. For more information about how to
obtain the domain name of an endpoint, see View the domain name of an endpoint.
|
Original network architecture |
IP address of the NGINX proxy |
The NGINX proxy in the same VPC as the Elasticsearch cluster is used to forward requests
over the Internet.
|
Webhook URL of the DingTalk chatbot |
The SNAT feature must be enabled. This feature enables an ECS instance in a VPC to
access the Internet if no public IP address is associated with the ECS instance.
|
Notice
- If the error
No handler found for uri [/_xpack/watcher/watch/log_error_watch_2] and method [PUT]
is returned after you run the preceding command, X-Pack Watcher is disabled for the
Elasticsearch cluster. In this case, enable X-Pack Watcher and run the command again.
For more information, see Configure the YML file.
- When you create a DingTalk chatbot, you must configure security settings. This is
because the body parameter in the preceding code must be specified based on the security settings.
For more information, see Configure security settings. In this example, Security Settings is set to Custom Keywords and the error keyword is specified. In this case, the DingTalk chatbot sends alert notifications
only if the content field in the body parameter contains error.
If you no longer require this watch, you can run the following command to delete the
watch:
DELETE _xpack/watcher/watch/log_error_watch