You can use Telegraf to collect Elasticsearch metrics, use Logtail to send the metric data to a MetricStore, and then create a dashboard to visualize the Elasticsearch metrics. This topic describes how to use Log Service to collect metrics from an Elasticsearch server and visualize the Elasticsearch metrics.
- Linux Logtail 0.16.48 or later is installed on the server. For more information, see Install Logtail on a Linux server.
- Telegraf is installed on a server that is connected to the Elasticsearch server over a private network.
- Log on to the Log Service console.
- In the Import Data section, click the Elasticsearch Monitoring Data card.
- In the Specify Logstore step, select the project and the Metricstore that you want to use. Then, click Next.
- In the Create Machine Group step, create a machine group.
- If a machine group is available, click Using Existing Machine Groups.
- If no machine groups are available, perform the following steps to create a machine
group. In this example, an Elastic Compute Service (ECS) instance is used.
- Select the ECS instance on which you want to install Logtail. For more information,
see Install Logtail on ECS instances.
If Logtail is installed on the ECS instance, click Complete Installation.Note If you want to collect data from servers in self-managed clusters or servers on third-party clouds, you must manually install Logtail. For more information, see Install Logtail on a Linux server.
- After Logtail is installed, click Complete Installation.
- Create a machine group.
- Select the ECS instance on which you want to install Logtail. For more information, see Install Logtail on ECS instances.
- Select the newly created machine group and move it from the Source Server Groups section to the Applied Server Groups section. Then, click Next. Notice If you apply a machine group immediately after it is created, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Log Service. In this case, you can click Automatic Retry. If the issue persists, see What do I do if no heartbeat connections are detected on Logtail?
- In the Specify Data Source step, set the required parameters. The following table describes the parameters.
Parameter Description Configuration Name The name of the Logtail configuration file. Cluster Name The name of the Elasticsearch cluster. After you set this parameter, Log Service adds the cluster=<cluster name> tag to your data.Note Make sure that the cluster name is unique. Otherwise, data conflicts may occur. Server List Click the + icon to add the Elasticsearch server and set the required parameters:
- Address: the endpoint of the Elasticsearch server.
- Port: the port number of the Elasticsearch server. The default value is 9200. You do not need to modify the value.
You can add multiple Elasticsearch servers based on your business requirements.
Index Names Click the + icon to add an Elasticsearch index name to monitor Elasticsearch Index metrics. If you set this parameter to _all, all index metrics are collected.
You can add multiple index names based on your business requirements.
Custom Tags In a MetricStore, you can create multiple Logtail configuration files. You can add custom tags that are specified in the Custom Tags field to data that is collected based on these Logtail configuration files.
Click the + icon to add a custom tag. You can add multiple custom tags. These custom tags are added to each data entry.
How can I check whether Telegraf collects data as expected?
You can check the logs of the /etc/ilogtail/telegraf/telegraf.log file on your server. You can also collect this log file to Log Service and search for the required information.
What to do next
- Query and analyze metrics
After the configuration, Telegraf uses Logtail to upload collected metrics to the MetricStore of Log Service. You can query the data on the Search & Analysis page of the MetricStore. For more information, see Query and analyze time series data.
- Visualize metrics
After the configuration, Log Service automatically creates a dashboard named Elasticsearch Monitoring Data_<cluster name> in the related project. In the dashboard, you can perform the required operations. For example, you can view query results or configure alert rules.